Can't scrape from thegamesdb.net - known problem?

jasonmicron

I just attempted to scrape using the following command:

sudo /opt/retropie/supplementary/scraper/scraper -thumb_only -workers 4
2018/07/06 23:44:59 arcade srcs:
2018/07/06 23:44:59 console srcs: gdb
2018/07/06 23:44:59 INFO: Checking for new hash.csv.
2018/07/06 23:44:59 INFO: hash.csv "d9eedf6e622a8ab6a1a14486bb90ffb8" up to date.
It appears that thegamesdb.net isn't up. If you are sure it is use -skip_check to bypass this error.
No sources available

Running the same command with -skip_check results in a repeated message of the following per game:

sudo /opt/retropie/supplementary/scraper/scraper -thumb_only -workers 4 -skip_check
2018/07/06 23:53:20 arcade srcs:
2018/07/06 23:53:20 console srcs: gdb
2018/07/06 23:53:20 INFO: Checking for new hash.csv.
2018/07/06 23:53:20 INFO: hash.csv "d9eedf6e622a8ab6a1a14486bb90ffb8" up to date.
2018/07/06 23:53:21 INFO: Starting: 3 Ninjas Kick Back.smc
2018/07/06 23:53:21 INFO: Starting: 7th Saga, The.smc
2018/07/06 23:53:21 INFO: Starting: 2020 Super Baseball.smc
2018/07/06 23:53:21 INFO: Starting: A.S.P. Air Strike Patrol.smc
2018/07/06 23:53:22 ERR: error processing 3 Ninjas Kick Back.smc: hash not found
2018/07/06 23:53:22 ERR: error processing 7th Saga, The.smc: hash not found
2018/07/06 23:53:22 ERR: error processing A.S.P. Air Strike Patrol.smc: hash not found
2018/07/06 23:53:22 ERR: error processing 2020 Super Baseball.smc: hash not found

A quick google search pointed me here, from a post as of today. Above it, another post from today indicates that indeed, scraping has been somewhat soft-blocked until the scrapers use the new API.

This seems to indicate scraping is, as of right now, completely broken. Thegamesdb seems to have implemented a new API, so as far as I can ascertain, all scraping is dead in the water right now until all scrapers are updated.

Am I correct in that assessment, or is there something else afoot?

I'm fine with using sselph, but I don't fancy going through a menu to do it, I'd like to run this through the CLI. Is that possible? Or is that what my command above is doing?

https://retropie.org.uk/docs/Scraper/#steven-selphs-scraper

mitu

@jasonmicron said in Can't scrape from thegamesdb.net - known problem?:

This seems to indicate scraping is, as of right now, completely broken. Thegamesdb seems to have implemented a new API, so as far as I can ascertain, all scraping is dead in the water right now until all scrapers are updated.
Am I correct in that assessment, or is there something else afoot?

The posts you referenced are about the 2 different scrapers. The Emulationstation built-in scraper has been updated when the 1st change on thegamesdb.net has been implemented, but the final change on the 3rd of July has broken the legacy.thegamesdb.net API site and triggered the errors in Emulationstation's scraper.

You can choose another source for scraping in Steven Selph's scraper, so you don't get this error and scraping will work. You can certainly run the scraper from the CLI (though you'd still have to stop ES from running) - see https://github.com/sselph/scraper/wiki.

Clyde

@mitu said in Can't scrape from thegamesdb.net - known problem?:

You can certainly run the scraper from the CLI (though you'd still have to stop ES from running) - see https://github.com/sselph/scraper/wiki.

I'm not sure I understand you correctly. Do you mean that it's still possible to scrape from TheGamesDB with Selph's scraper on the command line? If so, do you know the options required?

mitu

@clyde said in Can't scrape from thegamesdb.net - known problem?:

Do you mean that it's still possible to scrape from TheGamesDB with Selph's scraper on the command line?

No, I don't think so.

jasonmicron

@mitu said in Can't scrape from thegamesdb.net - known problem?:

@jasonmicron said in Can't scrape from thegamesdb.net - known problem?:

This seems to indicate scraping is, as of right now, completely broken. Thegamesdb seems to have implemented a new API, so as far as I can ascertain, all scraping is dead in the water right now until all scrapers are updated.
Am I correct in that assessment, or is there something else afoot?

The posts you referenced are about the 2 different scrapers. The Emulationstation built-in scraper has been updated when the 1st change on thegamesdb.net has been implemented, but the final change on the 3rd of July has broken the legacy.thegamesdb.net API site and triggered the errors in Emulationstation's scraper.

You can choose another source for scraping in Steven Selph's scraper, so you don't get this error and scraping will work. You can certainly run the scraper from the CLI (though you'd still have to stop ES from running) - see https://github.com/sselph/scraper/wiki.

Ahh cheers, thanks! I only dabble in RetroPie when I'm demanded by my family to add/remove games, and the last time I dabbled was late 2016. So I took the opportunity to update to Stretch from Jessie, among other outstanding updates. I saw a few blank entries for some games, which included, of all things, Super Metroid, so I wanted to re-scrape, which to got me here.

Thanks for the links :) If all goes well, you won't see a reply!

akafox

For a week now I have not been able to scrape for the games data base..that is from Steven Selph's Scraper. the Emulation Station gui, even from Attract Mode or the universal xml scraper. As far as I can tell they are doing some updating might have to wait till they get it fixed and their end..nothing can be done from a users ends right now..unless you want to build it from scratch by hand..the gamesdb.net site is up and mostly working

Arno

Any news on this?

mitu

@arno Yes, update your Emulationstation package and it should work.

Arno

Indeed it does. Thanks!

nestor1924

@mitu my PI is a version before 4.4 i didnt update cause i didnt need too since all the games i run are 8 and 16 bit, is there anyways to fix this without updating it?... im sure it will screw thing up right?

edmaul69

@nestor1924 updating retropie-setup then updating emulationstation isnt going to screw anything up. Why do you think it will?

mitu

@nestor1924 It's enough to update just Emulationstation if you want the fix for the scraping error. Updating the rest of the emulators won't hurt either, as @edmaul69 said.

AndersHP

Is it possible to manually scrape, e.g. by logging into screenscraper.fr or something? Never really found out how, but my scraper refuses to scrape a few games, and I'm just getting fed up now :)

mitu

@andershp Which scraper are you talking about ? The built-in ES scraper uses thegamesdb.net, not screenscraper.fr.

nestor1924

@edmaul69 because i thought that it would update it to 4.4 aswell and when the update came out people where sugested to do a fresh install of yhe firmware, which is the option thaf would update it to 4.4 ? just so that i can avoid it

AndersHP

@mitu Sorry, getting OT here, it's Universal XML Scraper I'm using. But nevertheless - any place to scrape manually online, if your scraper refuses to scrape a few games?

mitu

@andershp I guess you could always get the media from the site you wish, save if on your Pi and them manually edit the gamelist.xml and point to it.

nestor1924

@mitu i updated the retropie setup script and still not working? What did i do wromg?

lostless

@nestor1924 you also need to update emulation station under core packages. The newest version of ES fixes the scraper

nestor1924

@lostless but like i said before i have not update my retropie, mine its 4.3, if i do will there be an issue with my system?