What's the deal with scraping right now?

sad_muso

I'm trying to scrape with Skyscraper using Screenscraper and it's failing every time. I have setup an account and added the credentials to the config file to no avail. Do I need to pay? I want to get everything for all systems, including videos and I don't think many other scrapers support video

windg

Maybe the service is too busy or you have reach the daily scrap requests. Check again the credentials and try also to login on the screenscraper site before start to scrape.

mitu

Doesn't seem like their servers are exceptionally busy right now. Are you sure you've entered the correct configuration for user/password ? They're commented by default, with a # sign, did you un-comment them ?

shavecat

So what i have learned that u can get until 2000 scrapes A day or 24 hours after that u need to pay them if u wanna scrapes more..
its 2000 scrapes for each day or 24 hours .
Also having this wanted to scrape like 20000 but its always keep getting stuck after like 100 scrapes :\ dont know whats that all about but in the site it goes up to 2000 :\

u can check your info in https://www.screenscraper.fr/
just login there and u will see

sad_muso

Thanks guys. I have this in my config.ini:

[screenscraper] userCreds="username:password"

quicksilver

@sad_muso not sure if it matters but it should be like this:

[screenscraper] 
userCreds="username:password"

mitu

@shavecat said in What's the deal with scraping right now?:

So what i have learned that u can get until 2000 scrapes A day or 24 hours after that u need to pay them if u wanna scrapes more..
its 2000 scrapes for each day or 24 hours .
Also having this wanted to scrape like 20000 but its always keep getting stuck after like 100 scrapes :\ dont know whats that all about but in the site it goes up to 2000 :\

You don't 'pay' them, they're using the donation for the infrastructure (~500eur/month). There's a limit of 20000 requests/24h (with ~4 requests for a game) for an unregistered account and even if you donate more, you're still limited to 50.000 requests per day.

I successfully scraped more than a few hundreds games per day, but it all depends on the load they have. Skyscraper does caching, so you can resume scraping any time.

sad_muso

@quicksilver said in What's the deal with scraping right now?:

@sad_muso not sure if it matters but it should be like this:
[screenscraper] 
userCreds="username:password"

It is but I forgot the line break in the code snippet 😕

sad_muso

So scraping from one of the API services is still better than trying to find a dump somewhere online I'm guessing, due to ROM compatibility...

sad_muso

This post is deleted!