Versatile C++ game scraper: Skyscraper

muldjord

Just a quick update: The screenscraper module is currently having issues with their internal server cache (not related to the Skyscraper cache, just to be clear) when you scrape without user credentials. It will sometimes (quite often) return faulty error messages even though Skyscraper uses the API correctly. I've contacted them and they verified that this is an issue. They've been working on fixing it for the past couple of days, but their cache (again, not the Skyscraper cache, their internal server cache) needs to catch up to the fixes. So hopefully this issue will resolve itself over the coming week.

muldjord

The culprit has been found for the screenscraper issues. This stems from their server CPU's usage being too high. When it reaches a certain limit, it starts rejecting requests from unregistered users (this is by design, it's not a bug).

3.0.5 has just been released which adds a note about this when the error occurs. It also fixes a bug in the metadata preservation code, so please update.

Once again, please consider supporting screenscraper when you go to register a user. And also consider supporting RetroPie while you are at it.

ByteThis

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

mitu

@ByteThis That thought crossed my mind, but that can only be answered by the ScreenScraper admins.

muldjord

@ByteThis said in Versatile C++ game scraper: Skyscraper:

@muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

All I know is that this is only an issue for unregistered users. But in general it is slightly worrying that their servers are reaching these usage levels. Skyscraper does a lot to alleviate it by caching the data. But I'm not sure other scrapers do this.

ByteThis

@muldjord They definitely have a lot on their plate.

EDIT: This is posted on their site : This month: 4382 euros collected. / 265 euros for a dedicated server.

Maybe they can afford another server ;-)

chubsta

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

muldjord

@chubsta said in Versatile C++ game scraper: Skyscraper:

I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

quicksilver

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

chubsta

@muldjord said in Versatile C++ game scraper: Skyscraper:

Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

Thanks for the reply - I was hoping to keep them separate, I seem to be having the same issue with the pc engine and tg16 cd sets, which I hoped to keep in separate folders from their rom sets, looks like I may have to reconsider...

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

Yes, this is due to you using an unregistered account. Their servers are quite pressured at the moment, and it will reject some requests from unregistered users because of this. My best advice is to create an account with them and use your credentials with Skyscraper when using the 'screenscraper' module.

muldjord

A small teaser demonstrating the upcoming resource cache editing features I am working on:

quicksilver

@muldjord thanks for the reply, I am using a registered account (though it took me a little bit to figure out how to have it accept my credentials). Ill run it again to be sure and report back if the issue continues.

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord thanks for the reply, I am using a registered account (though it took me a little bit to figure out how to have it accept my credentials). Ill run it again to be sure and report back if the issue continues.

My guess is that if you keep repeating it (test it by adding a single file name on command line) then it will eventually work, which means it's a screenscraper issue unfortunately.

muldjord

Skyscraper 3.1.0 released: https://github.com/muldjord/skyscraper

MAJOR: Added '--cache edit' command line option which allows viewing, editing and deleting cached resources for the roms in the queue. Narrow the queue down by providing file names on command line or by using '--startat <FILENAME>' and '--endat <FILENAME>'
Added 'zx81' platform. Note! The only module that supports it is the 'screenscraper' module

I finally found the time to work on the planned resource cache editing features of Skyscraper. This allows you to easily add and remove resources to / from any given rom simply by using the --cache edit option. With that option you can edit cached resources for a few roms simply by providing their filenames on command line. Or you can use the --startat and --endat option(s). If no filenames are provided on command line or both --startat and --endat are left out, Skyscraper will edit all files in your input folder one by one.

Any resources added by the user will be marked as user and will be prioritized above all other types. The edit feature will also allow you to get a look at what data will currently be used for any given rom when generating the game list. You can also list all resources for any given rom. And it also enables you to remove resources as you see fit, both single resources or all resources of specific types or from specific modules.

Have fun! And let me know if this is useful to you. I've already used it myself quite a bit and find it to be very handy indeed.

quicksilver

@muldjord must be a screenscraper issue then. My credentials are definitely being accepted but for whatever reason some roms (I have found others now) that will scrape only partial data. It will get the screenshot but not the box art or vice-versa even though all the data is there on the screenscraper website. Hopefully they continue to improve the website. Thanks again for your help.

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord must be a screenscraper issue then. My credentials are definitely being accepted but for whatever reason some roms (I have found others now) that will scrape only partial data. It will get the screenshot but not the box art or vice-versa even though all the data is there on the screenscraper website. Hopefully they continue to improve the website. Thanks again for your help.

Please try rescraping those roms with the --refresh option enabled or by providing the filenames on command line to scrape just those few roms that are giving you difficulties. That will most likely fix your issue. I'm very interested in knowing if this works for you. If not, then there's something else going on that I am not aware of.

quicksilver

@muldjord the refresh option took care of my issue. Once again thank you for your help!

ZXDunny

@muldjord - a quick question; can the "dbs" folder be relocated anywhere else? My uSD card is full and nothing can be scraped any further despite all my ROMs and skyscraper cache being located on the external 2TB HDD. The dbs folder contains 2.3GB of data, by all accounts.

Can it be relocated? I'd empty it but I'm not sure what effect that would have, and it will be a PITA to have to do that every time I want to scrape a collection.

RetroPie is running from a 32GB uSD card that I cannot access physically, hence I've attached the HDD to the board so that ~/pi/RetroPie is now accessed from there. I edited config to move the cache there also, but the dbs folder doesn't seem to be configurable?

Edit: Even though my config has been edited to move the cache to ~/pi/RetroPie/skyscraper/cache, Skyscraper is still using the original location on my uSD card :(

mitu

@ZXDunny Can you post your config.ini file and your Skyscraper version ? Since 3.x the dbs folder is named cache, so you might be using an older version.