Versatile C++ game scraper: Skyscraper

ByteThis

@mitu It works.Not sure what initially happened but the dependency installed and ES updated with both Screenscraper and THEGAMESDB are both there. Sorry for any confusion. Seems like some kind of hiccup installing rapidjson-dev.

mitu

@ByteThis You're supposed to execute those commands from a SSH session (Putty). Glad it works, thank you for the confirmation.

ByteThis

@mitu Yea..as I mentioned above I tried both Putty and the terminal in WinSCP. Thanks for your hard work ! Again apologies for the hiccup. Am glad to be of some help...lol.

semakelh50

@muldjord The documentation contains a misspelling in SCRAPINGMODULES.md. In the OpenRetro section -> platform support, "aswell" should be "as well." It's two words, not one.

muldjord

Just a quick update: The screenscraper module is currently having issues with their internal server cache (not related to the Skyscraper cache, just to be clear) when you scrape without user credentials. It will sometimes (quite often) return faulty error messages even though Skyscraper uses the API correctly. I've contacted them and they verified that this is an issue. They've been working on fixing it for the past couple of days, but their cache (again, not the Skyscraper cache, their internal server cache) needs to catch up to the fixes. So hopefully this issue will resolve itself over the coming week.

muldjord

The culprit has been found for the screenscraper issues. This stems from their server CPU's usage being too high. When it reaches a certain limit, it starts rejecting requests from unregistered users (this is by design, it's not a bug).

3.0.5 has just been released which adds a note about this when the error occurs. It also fixes a bug in the metadata preservation code, so please update.

Once again, please consider supporting screenscraper when you go to register a user. And also consider supporting RetroPie while you are at it.

ByteThis

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

mitu

@ByteThis That thought crossed my mind, but that can only be answered by the ScreenScraper admins.

muldjord

@ByteThis said in Versatile C++ game scraper: Skyscraper:

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

All I know is that this is only an issue for unregistered users. But in general it is slightly worrying that their servers are reaching these usage levels. Skyscraper does a lot to alleviate it by caching the data. But I'm not sure other scrapers do this.

ByteThis

@muldjord They definitely have a lot on their plate.

EDIT: This is posted on their site : This month: 4382 euros collected. / 265 euros for a dedicated server.

Maybe they can afford another server ;-)

chubsta

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

muldjord

@chubsta said in Versatile C++ game scraper: Skyscraper:

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

quicksilver

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

chubsta

@muldjord said in Versatile C++ game scraper: Skyscraper:

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

Thanks for the reply - I was hoping to keep them separate, I seem to be having the same issue with the pc engine and tg16 cd sets, which I hoped to keep in separate folders from their rom sets, looks like I may have to reconsider...

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

Yes, this is due to you using an unregistered account. Their servers are quite pressured at the moment, and it will reject some requests from unregistered users because of this. My best advice is to create an account with them and use your credentials with Skyscraper when using the 'screenscraper' module.

muldjord

A small teaser demonstrating the upcoming resource cache editing features I am working on:

quicksilver

@muldjord thanks for the reply, I am using a registered account (though it took me a little bit to figure out how to have it accept my credentials). Ill run it again to be sure and report back if the issue continues.

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord thanks for the reply, I am using a registered account (though it took me a little bit to figure out how to have it accept my credentials). Ill run it again to be sure and report back if the issue continues.

My guess is that if you keep repeating it (test it by adding a single file name on command line) then it will eventually work, which means it's a screenscraper issue unfortunately.

muldjord

Skyscraper 3.1.0 released: https://github.com/muldjord/skyscraper

MAJOR: Added '--cache edit' command line option which allows viewing, editing and deleting cached resources for the roms in the queue. Narrow the queue down by providing file names on command line or by using '--startat <FILENAME>' and '--endat <FILENAME>'
Added 'zx81' platform. Note! The only module that supports it is the 'screenscraper' module

I finally found the time to work on the planned resource cache editing features of Skyscraper. This allows you to easily add and remove resources to / from any given rom simply by using the --cache edit option. With that option you can edit cached resources for a few roms simply by providing their filenames on command line. Or you can use the --startat and --endat option(s). If no filenames are provided on command line or both --startat and --endat are left out, Skyscraper will edit all files in your input folder one by one.

Any resources added by the user will be marked as user and will be prioritized above all other types. The edit feature will also allow you to get a look at what data will currently be used for any given rom when generating the game list. You can also list all resources for any given rom. And it also enables you to remove resources as you see fit, both single resources or all resources of specific types or from specific modules.

Have fun! And let me know if this is useful to you. I've already used it myself quite a bit and find it to be very handy indeed.

quicksilver

@muldjord must be a screenscraper issue then. My credentials are definitely being accepted but for whatever reason some roms (I have found others now) that will scrape only partial data. It will get the screenshot but not the box art or vice-versa even though all the data is there on the screenscraper website. Hopefully they continue to improve the website. Thanks again for your help.