Problem with Skyscraper, no data found or not showing up

m2306

I’m on a RPi3B+ with Retropie 4.4. I updated the setup script and deleted and reinstalled Skyscraper. I’m launching Skyscraper from the command line, after quitting ES. I’m using default settings. I also know I have to choose the option to generate the gamelists for the data to show up in ES. I got good results even before updating with many systems including one with 1500 roms. The problem started with scraping for GBA. When the scraper got to the letter P, about 700 roms down it started getting a not found error and also any system scraped in the same session after GBA was left without data. If I scrape another system separately I get mixed results. I got the data for Vectrex which has just a few roms but no luck with other systems. Rescraping GBA results in the same problem. I know there are some logs that could possibly help but don’t know how access them. After updating I don’t get a error message anymore, the scraper finds info, I generate gamelists but when I go into ES there’s no data. If I go to the location pointed to in the error message before updating I found several .log.gz files that are not readable with Nano. I can SSH from OSX if I get some pointers on which log to choose and how to open it.

mitu

You seem to have 2 problems

scraping not getting all the data, but based on your story it looks like you've been able to finish the scraping
the scraped info not showing up in Emulationstation - this most likely is caused by duplicate gamelists. If you're running from the command line, where is the gamelist saved ? Skyscraper - in the generating step - should tell you where it saved the gamelist.xml file populated with the scraped info.

m2306

@mitu Thanks for the input. Sorry if I was not clear. I start Skyscraper by launching the setupscript from command line and then launch Skyscraper from the manage packages menu. There I use the graphic user interface with default options, just choosing which system to scrape.

mitu

@m2306 You said you're using Skyscraper from the command line, which for me means you're running it directly (Skyscraper [... options..]).
If you're running it from the RetroPie-Setup front-end then some options are predefined - including where the gamelist.xml is stored. Take a look at the Docs - especially the section with Where are my scraped media and metadata saved? - and see where the gamelist.xml is stored - if it's not in the ROMs folder, then check your ROM folder for a gamelist.xml and remove it, then re-generate again the gamelist from the interface (without gathering).

m2306

@mitu Thank you very much. It worked for one of the systems that Skyscraper gathered data for but didn’t show up in ES. As for GBA I will have to scrape again to see if it works as it actually didn’t download any data for half the roms.

Clyde

@m2306 A quick way to disable all existing gamelists in your rom folders is this one-line shell script that I wrote a while ago: https://retropie.org.uk/forum/topic/23235/script-one-liner-to-backup-or-disable-the-gamelists-in-all-rom-directories

This way (and/or checking the Skyscraper options regarding the gamelist path in the Retropie Setup like @mitu suggested) you can be sure that ES won't load old gamelists instead of the ones you created anew with Skyscraper.

m2306

@mitu So, while the scraping worked for some systems and regenerating the gamelists worked for some of the affected ones I get the same error again with some systems. I removed gamelist, purged and scraped again for GBA but the same thing happened again at around the letter T (I clenaed up the romset so it's also a few roms shorter now, around 1000). I also got the same problem on PSX that only has around 70 roms, the last few are left without any data. Any ideas on how to proceed?

mitu

Assuming it's a throttling issue, you should try re-scraping. But the ScreenScraper servers are usually busy during the week-end, so I'd wait until monday.

Clyde

Also, registering at www.screenscraper.fr for free will get you still access if SS closes its API for non-registered users because of too much traffic.

Furthermore, a monthly donation of at least 1€ will raise you to "contributor" status with access if they close the API for non-registered and registered users, and 50.000 scraping requests instead of 20.000.

edit: To let Skyscraper use your Screenscraper user credentials, you'll have to add theses lines to your ~/.skyscaper/config.ini:

[screenscraper]
userCreds="username:password"

Beware that .skyscraper is a hidden directory, because its name begins with a dot. So, you might not see it, but nevertheless you can edit it by entering nano ~/.skyscraper/config.ini.

m2306

@Clyde I made a user account. The whole config.ini file in that directory is commented and the same as the config.ini.example file. Since it’s all commented I can’t see how it could be the correct config file.. I tried to uncomment the line with the username and password (and saved changes) but it wasn’t recognised by Skyscraper.

muldjord

@m2306 You also need to have the line [screenscraper] above the userCreds line for it to work. Like this:

[screenscraper)
userCreds="USER:PASSWORD"

Also be aware that ScreenScraper has several limitations in general. If it stopped getting data at some point, you probably ran out of requests for the day. So as others point out, just try again the next day. But yes, always create a user and use your credentials. It's much more stable as the API is very often shut down for unregistered users due to high server loads.

m2306

@muldjord ok. Thank you for a great application!