TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........

stevetb

Hey All,

Our primary source to use as a scraper is of course TheGamesDB. This site currently has been down and is the second outage this week. Since I have been using the scraper for Selph or EmulationStation within RetroPie it has consistently failed. This is at least the fifth failure I can recall over the past month of use.

LaunchBox got fed up and has announced they are moving away from TheGamesDB because it is not stable enough.

What is the take from the RetroPie Community on moving away from TheGamesDB and moving to our own server? Is it something which is being looked at or are we just kinda stuck with TheGamesDB?

I've tried the other listed DB in EmulationStation on the RetroPie but it doesn't seem to work at all.

Thanks for listening,
Steve

herb_fargus

@stevetb did you try the other one listed under sselphs scraper?

stevetb

I went back in and switched this over to OpenVGDB. I'll report back in after this has all ran. I was hesitant to run this because I thought it would overwrite all of the previous scraping I have done through TheGamesDB.

Thank you herb_fargus.

nc4080

I'm pretty new to the whole scraping thing (thanks again herb_fargus) and today I've been trying to scrape a few psx titles, through the two scrapers on emulation station and the two on sselphs scraper.
Unfortunately 'TheGamesDB' is down as herb_fargus has told me, which is not a good start to scraping but I'll be patient and wait for it to come back online.

stevetb

@nc4080 I've found its goes down all the time. Scraping is frustrating because their site is not dependable. Hopefully the Retropie developers find a way to use a more reliable source.

lilbud

@stevetb Console Grid seems like a good alternative, it is used for emulated games on steam.

Console Grid

nc4080

@lilbud Can you run Console Grid from the RetroPie Setup?

schmulkfuster

To be fair... the fact that we're scraping at all (no matter the service), is probably not a great solution. I suspect they probably didn't build the website with the intent of it being scraped and certainly not by hundreds of people with thousands of games that never even visited the website or supported the service in any way and want their images right now, rather than scraping slowly over a 24 hour period.

I think unless they have very large, professional scale infrastructure behind the scenes then they're going to struggle to support this kind of thing unless they add some rate limiting or some kind of CDN and/or proper API for scraping.

Perhaps it would be good if we could all just take our feet off the accelerator and stop pounding them until they get stable again :)

nc4080

@schmulkfuster I agree, although it would be great if there was a community that would contribute or create a DB. Not sure if this is against forum rules but I would be willing to help create and maintain a DB if there is anyone interested.

lilbud

@nc4080 Sadly you cannot

urgrue

@nc4080 said in TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........:

@schmulkfuster I agree, although it would be great if there was a community that would contribute or create a DB. Not sure if this is against forum rules but I would be willing to help create and maintain a DB if there is anyone interested.

Openvgdb could be a good starting point. Basically when retropie fails to find the exact hash, it should allow searching games based on keywords and manually selecting+editing the right match, and then automatically allow to submit this hash to the online game database.
I know the second part (manual search) seems to exist in retropie but it has not worked for me ever.

It'd be really cool if this all worked smoothly straight from the interface. As it stands I barely have half my roms correctly scraped and manually updating them from the command line is basically an impossible effort.

zerojay

@urgrue OpenVGDB is already supported by sselph's scraper and by Retropie Setup.

urgrue

@zerojay said in TheGamesDB: It's Up.......It's Down......It's Up.......It's Down........:

@urgrue OpenVGDB is already supported by sselph's scraper and by Retropie Setup.

I wasn't saying it wasn't - I was suggesting using openvgdb as a starting point to create a online DB that is better integrated into retropie and also automatically takes submissions.
Like I said, today there is no way to manually fix the game metadata short of a hugely painful one-by-one effort that benefits no one else but yourself. Considering the hit rate of the scraper is quite low, it's a shame.