TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........

nc4080

I'm pretty new to the whole scraping thing (thanks again herb_fargus) and today I've been trying to scrape a few psx titles, through the two scrapers on emulation station and the two on sselphs scraper.
Unfortunately 'TheGamesDB' is down as herb_fargus has told me, which is not a good start to scraping but I'll be patient and wait for it to come back online.

stevetb

@nc4080 I've found its goes down all the time. Scraping is frustrating because their site is not dependable. Hopefully the Retropie developers find a way to use a more reliable source.

lilbud

@stevetb Console Grid seems like a good alternative, it is used for emulated games on steam.

Console Grid

nc4080

@lilbud Can you run Console Grid from the RetroPie Setup?

schmulkfuster

To be fair... the fact that we're scraping at all (no matter the service), is probably not a great solution. I suspect they probably didn't build the website with the intent of it being scraped and certainly not by hundreds of people with thousands of games that never even visited the website or supported the service in any way and want their images right now, rather than scraping slowly over a 24 hour period.

I think unless they have very large, professional scale infrastructure behind the scenes then they're going to struggle to support this kind of thing unless they add some rate limiting or some kind of CDN and/or proper API for scraping.

Perhaps it would be good if we could all just take our feet off the accelerator and stop pounding them until they get stable again :)

nc4080

@schmulkfuster I agree, although it would be great if there was a community that would contribute or create a DB. Not sure if this is against forum rules but I would be willing to help create and maintain a DB if there is anyone interested.

lilbud

@nc4080 Sadly you cannot

urgrue

@nc4080 said in TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........:

@schmulkfuster I agree, although it would be great if there was a community that would contribute or create a DB. Not sure if this is against forum rules but I would be willing to help create and maintain a DB if there is anyone interested.

Openvgdb could be a good starting point. Basically when retropie fails to find the exact hash, it should allow searching games based on keywords and manually selecting+editing the right match, and then automatically allow to submit this hash to the online game database.
I know the second part (manual search) seems to exist in retropie but it has not worked for me ever.

It'd be really cool if this all worked smoothly straight from the interface. As it stands I barely have half my roms correctly scraped and manually updating them from the command line is basically an impossible effort.

zerojay

@urgrue OpenVGDB is already supported by sselph's scraper and by Retropie Setup.

urgrue

@zerojay said in TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........:

@urgrue OpenVGDB is already supported by sselph's scraper and by Retropie Setup.

I wasn't saying it wasn't - I was suggesting using openvgdb as a starting point to create a online DB that is better integrated into retropie and also automatically takes submissions.
Like I said, today there is no way to manually fix the game metadata short of a hugely painful one-by-one effort that benefits no one else but yourself. Considering the hit rate of the scraper is quite low, it's a shame.