Versatile C++ game scraper: Skyscraper

ByteThis

@mitu The dependency couldn't be installed : https://pastebin.com/zknu3NVV

mitu

@ByteThis The workaround I posted involves installing the rapidjson-dev package manually before updating

wget http://ftp.us.debian.org/debian/pool/main/r/rapidjson/rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb
sudo dpkg -i rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb

ByteThis

@mitu That's the one I tried to install. I tried both WinSCP (through the terminal) and Putty.

ByteThis

@mitu I tried it again a couple of times and ES is building from source..stay tuned.

ByteThis

@mitu It works.Not sure what initially happened but the dependency installed and ES updated with both Screenscraper and THEGAMESDB are both there. Sorry for any confusion. Seems like some kind of hiccup installing rapidjson-dev.

mitu

@ByteThis You're supposed to execute those commands from a SSH session (Putty). Glad it works, thank you for the confirmation.

ByteThis

@mitu Yea..as I mentioned above I tried both Putty and the terminal in WinSCP. Thanks for your hard work ! Again apologies for the hiccup. Am glad to be of some help...lol.

semakelh50

@muldjord The documentation contains a misspelling in SCRAPINGMODULES.md. In the OpenRetro section -> platform support, "aswell" should be "as well." It's two words, not one.

muldjord

Just a quick update: The screenscraper module is currently having issues with their internal server cache (not related to the Skyscraper cache, just to be clear) when you scrape without user credentials. It will sometimes (quite often) return faulty error messages even though Skyscraper uses the API correctly. I've contacted them and they verified that this is an issue. They've been working on fixing it for the past couple of days, but their cache (again, not the Skyscraper cache, their internal server cache) needs to catch up to the fixes. So hopefully this issue will resolve itself over the coming week.

muldjord

The culprit has been found for the screenscraper issues. This stems from their server CPU's usage being too high. When it reaches a certain limit, it starts rejecting requests from unregistered users (this is by design, it's not a bug).

3.0.5 has just been released which adds a note about this when the error occurs. It also fixes a bug in the metadata preservation code, so please update.

Once again, please consider supporting screenscraper when you go to register a user. And also consider supporting RetroPie while you are at it.

ByteThis

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

mitu

@ByteThis That thought crossed my mind, but that can only be answered by the ScreenScraper admins.

muldjord

@ByteThis said in Versatile C++ game scraper: Skyscraper:

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

All I know is that this is only an issue for unregistered users. But in general it is slightly worrying that their servers are reaching these usage levels. Skyscraper does a lot to alleviate it by caching the data. But I'm not sure other scrapers do this.

ByteThis

@muldjord They definitely have a lot on their plate.

EDIT: This is posted on their site : This month: 4382 euros collected. / 265 euros for a dedicated server.

Maybe they can afford another server ;-)

chubsta

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

muldjord

@chubsta said in Versatile C++ game scraper: Skyscraper:

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

quicksilver

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

chubsta

@muldjord said in Versatile C++ game scraper: Skyscraper:

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

Thanks for the reply - I was hoping to keep them separate, I seem to be having the same issue with the pc engine and tg16 cd sets, which I hoped to keep in separate folders from their rom sets, looks like I may have to reconsider...

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

Yes, this is due to you using an unregistered account. Their servers are quite pressured at the moment, and it will reject some requests from unregistered users because of this. My best advice is to create an account with them and use your credentials with Skyscraper when using the 'screenscraper' module.

muldjord

A small teaser demonstrating the upcoming resource cache editing features I am working on: