Versatile C++ game scraper: Skyscraper

muldjord

@ByteThis 3.0.4 out now. Please update and test.

mitu

@muldjord That's great ! Just discovered that the new ES version doesn't install on Jessie (due to a missing dependency), so the updated ES scraper can't be used.

ByteThis

@muldjord Works perfect now man! Thanks for your hard work and your Scraper !

ByteThis

@mitu Yes..the error is "Could not install package(s): rapidjson.dev". I updated ES some time ago to install the Screenscraper scraper within Retropie and tried to re-update to include TheGAMESDB but got that error.

mitu

@ByteThis I posted a workaround in the sticky topic with the ES scraper problems in the Help sections. If you could give it a try and see if it works, that would be great.

ByteThis

@mitu Sure. I will try it. BRB.

ByteThis

@mitu The dependency couldn't be installed : https://pastebin.com/zknu3NVV

mitu

@ByteThis The workaround I posted involves installing the rapidjson-dev package manually before updating

wget http://ftp.us.debian.org/debian/pool/main/r/rapidjson/rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb
sudo dpkg -i rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb

ByteThis

@mitu That's the one I tried to install. I tried both WinSCP (through the terminal) and Putty.

ByteThis

@mitu I tried it again a couple of times and ES is building from source..stay tuned.

ByteThis

@mitu It works.Not sure what initially happened but the dependency installed and ES updated with both Screenscraper and THEGAMESDB are both there. Sorry for any confusion. Seems like some kind of hiccup installing rapidjson-dev.

mitu

@ByteThis You're supposed to execute those commands from a SSH session (Putty). Glad it works, thank you for the confirmation.

ByteThis

@mitu Yea..as I mentioned above I tried both Putty and the terminal in WinSCP. Thanks for your hard work ! Again apologies for the hiccup. Am glad to be of some help...lol.

semakelh50

@muldjord The documentation contains a misspelling in SCRAPINGMODULES.md. In the OpenRetro section -> platform support, "aswell" should be "as well." It's two words, not one.

muldjord

Just a quick update: The screenscraper module is currently having issues with their internal server cache (not related to the Skyscraper cache, just to be clear) when you scrape without user credentials. It will sometimes (quite often) return faulty error messages even though Skyscraper uses the API correctly. I've contacted them and they verified that this is an issue. They've been working on fixing it for the past couple of days, but their cache (again, not the Skyscraper cache, their internal server cache) needs to catch up to the fixes. So hopefully this issue will resolve itself over the coming week.

muldjord

The culprit has been found for the screenscraper issues. This stems from their server CPU's usage being too high. When it reaches a certain limit, it starts rejecting requests from unregistered users (this is by design, it's not a bug).

3.0.5 has just been released which adds a note about this when the error occurs. It also fixes a bug in the metadata preservation code, so please update.

Once again, please consider supporting screenscraper when you go to register a user. And also consider supporting RetroPie while you are at it.

ByteThis

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

mitu

@ByteThis That thought crossed my mind, but that can only be answered by the ScreenScraper admins.

muldjord

@ByteThis said in Versatile C++ game scraper: Skyscraper:

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

All I know is that this is only an issue for unregistered users. But in general it is slightly worrying that their servers are reaching these usage levels. Skyscraper does a lot to alleviate it by caching the data. But I'm not sure other scrapers do this.

ByteThis

@muldjord They definitely have a lot on their plate.

EDIT: This is posted on their site : This month: 4382 euros collected. / 265 euros for a dedicated server.

Maybe they can afford another server ;-)