Versatile C++ game scraper: Skyscraper

muldjord

I've been working behind-the-scenes with the good people of igdb. This has resulted in Skyscraper becoming part of the partner program which ups the request limit for the module to 50000 per month. And they also let me know that they are working on improving the artwork situation for their database during 2019. This basically means that we at some point will be able to distinguish platforms for their screenshots and other artwork, which it can't at the moment (which is why all artwork is disabled for the module). So this is some pretty awesome news on all accounts.

Silent

Great! With 50k requests per month, it should probably be fairly safe to expose IGDB in Retropie-Setup script?

No biggie if not, just making an observation.

muldjord

@Silent I'm thinking about it, but the problem here being just "letting loose". If someone decides to scrape 5000 roms on it we wouldn't be able to stop them and... Well, you can guess the rest. A couple of days and the requests would be gone.

It's a little easier to manage with thegamesdb, where the requests are per IP.

EDIT: But I do believe it would be worth trying it out and see how it goes. It will have to wait until the next release though, as I need to remove the 5 roms per run limit.

EDIT2: I'll up the limit to 250 roms per scraping run. This should be adequate for most users for a platform. If users have more, it will just quit on them.

robertvb83

@muldjord lets say i have 100 roms that i want to scrape. Now i already have video for 95 of those put in the input folder. Gathering the local videos will fill the cache with those 95 videos
How can i now search videos for the remaining 5. If i run the scrape again via gathering from screenscraper or arcadedb, it will again download all the videos for 100 roms. This is not what i want, it would be nice to be able to "only fill the gaps"

Especially for videos there is often no need for redundancy. But also other artwork i would prefer a "fill the gaps" mode.

muldjord

@robertvb83

If i run the scrape again via gathering from screenscraper or arcadedb, it will again download all the videos for 100 roms.

No it won't. If the videos are already scraped, they are detected from the cache and not fetched again. You will notice that it is a lot faster to scrape a platform the second time around, since it will detect that all data is already in the cache and just show what data is has on each file (check the output that says "From cache: YES". If it says "NO" maybe you have enabled --refresh, I do not recommend that as it force all data to be redownloaded which is the opposite of what you are asking). Except for the ones that aren't cached, where it will retry from the source.

And if you want to just fetch data for the remaining 5 roms, just use the --startat and --endat options or add the file names to the command line. This will refetch the data for just those files.

EDIT: But I see room for improvement here. Clearly you are confused about the output as it is right now. So I will ponder on this and see if I can make it clearer in a future version of Skyscraper. I'll add it to the roadmap.

EDIT2: I realize that I misunderstand you. You are correct, if the 95 videos came from the import module, then it will download videos for all roms if you then use screenscraper or arcadedb. But as mentioned, if you are just missing 5, just use command line file names or --startat and --endat

robertvb83

@muldjord hmm but what is the point in redownloading videos with each scraping module... i am not using this via command line because i like the lazy gui way... so i end up with a lot of wasted disk space (at least for videos, it doesnt really matter for images)
Dont get me wrong... i am not really complaining. I know such things are not just coded in 5 minutes and i am very happy with your tool and could not thank you enough. I will just switch of the video download in any second scraping module run.

muldjord

@robertvb83 Skyscraper's cache is the basis on which you would generate any game list. This means that you can, at any time, change anything in the general config, artwork config, the resource priorities, the frontend you wish to generate game lists for. Basically anything can be changed and you can regenerate the game lists as if you just re-downloaded everything. But this only works if the cache is consistent with the data from the sources. This is the basic principle of Skyscraper.

You could consider simply generating the ES gamelist.xml with videos. Then remove all videos from the cache (just the videos! Not everything as the ES gamelists don't support all data) and then use the esgamelist module to reimport the data back in. Then all videos will be in the cache from the esgamelist module and from no other modules. I would strongly suggest making a backup of the cache before you try this though.

Silent

@muldjord said in Versatile C++ game scraper: Skyscraper:

and then use the esgamelist module to reimport the data back in.

That made me realize, is it possible to specify only specific things to import? Sounds like this case would only need videos imported, and importing everything may introduce a lot of unnecessary data.

muldjord

@Silent Just remove it afterwards. Or you can use the 'cacheCovers="false"' and so on options. I'll add those to the scraping modules section in config.ini aswell. That will provide what you are looking for.

EDIT: There, this will now be in 3.0.2.

fartsparkles

Does anyone have any idea why I can't scrape two ROMs specifically? I'm using screenscraper and I can see in their database both the game and exact file I have (filename, crc, md5, and sha1 are exact matches)? This only happens on two specific ROMs that also happen to have period/. chars in their names (not sure if this is relevant). ROMs are in binary form - not zipped - as per the screenscraper.fr DB entry.

Running Skyscraper 3.0.1 via the RetroPie Setup script.

mitu

@fartsparkles Maybe it would help to give the names/crcs for those ROMs and what system they're in. They might exist on SS, but have a different system.

fartsparkles

I tested with Skraper and had no issue scraping the ROMs. There are the specific ROMs that won't scrape with Skyscraper:
https://www.screenscraper.fr/rominfos.php?romid=315352
https://www.screenscraper.fr/rominfos.php?romid=314736
I'm not a C++ person so I haven't stepped through the code however I find it interesting that both these ROMs have periods in the name.

muldjord

This post is deleted!

muldjord

@fartsparkles I've been looking into this a bit, and I can rule out that it has to do with the "." in the file name (at least not on the Skyscraper side of things). I tested this by simply giving Screenscraper a custom query with the --query option where I specifically ask for the md5 or sha1 checksums of the rom you mention. And even by doing so, I still get a direct "Rom not found" back from Screenscraper. So this almost has to be a Screescraper bug, and not a Skyscraper bug. If Screenscraper does not provide a result when using a direct md5 or sha1 checksum query, I simply can't do anything about it.

EDIT: If you want to mess around with this yourself, here's the command line needed to try with specific md5 sums. You can also change that to sha1 if you want:

Skyscraper -p atari2600 -s screenscraper --verbosity 3 --query "md5=<CHECKSUM>" "<FILENAME>"

Verbosity 3 simply gives you a bit more details in the output.

EDIT2: Something is seriously wrong with Screenscraper right now. I can't get it to find any rom based entirely on checksum right now. I just tried with Super Mario World for Snes, and that only returns a reply on file name, not when I try just with the checksums. This usually works just fine.

My best advice is to give it a week and see if it resolves itself. I'm guessing they are having some issues with their database that needs resolving before this will work.

Brunnis

@muldjord Maybe I suffer from the same issue as @fartsparkles ? The strange thing is that Steven Selph's scraper, using Screenscraper, finds these 40 games that Skyscraper can't identify. I'll re-post what I just wrote on Reddit:

"I've been setting up a new RetroPie setup and thought I'd use Skyscraper. However, I'm having a strange issue with it. I have a curated list of NES roms which are all verified against No-Intro. Both RetroArch's own DB (which checks against No-Intro hashes) and Steven Selph's scraper (using Screenscraper) correctly identify these ROMs. However, Skyscraper fails on 40 of them, with "No returned matches".

Here's an example: Batman (U) [!].nes (MD5: 2e9f52556273aa735d0e75649541d812)

This ROM can be found at screenscraper.fr with the exact same hash: https://www.screenscraper.fr/rominfos.php?romid=135447

I've tried both versions 2.9.5 and 3.0.1 of Skyscraper. Same issue. Anyone have any ideas? Maybe /u/muldjord

I feel like I must be missing something obvious here... Still, the other methods mentioned above work out of the box with these ROMs."

muldjord

I'm currently following some threads on their forums that might be related to this. For the time being Skyscraper seems to be blocked - no idea why. I'm awaiting reply from them.

EDIT: Skyscraper is currently banned from using the Screenscraper database due to the 4 passes I do with varying checksums and file name. I'm working on resolving this. It will most likely mean that Skyscraper will get a new API key and all users will have to update to a new version of Skyscraper when the new key has been issued. Stay tuned.

Brunnis

@muldjord said in Versatile C++ game scraper: Skyscraper:

I'm currently following some threads on their forums that might be related to this. For the time being Skyscraper seems to be blocked - no idea why. I'm awaiting reply from them.

EDIT: Skyscraper is currently banned from using the Screenscraper database due to the 4 passes I do with varying checksums and file name. I'm working on resolving this. It will most likely mean that Skyscraper will get a new API key and all users will have to update to a new version of Skyscraper when the new key has been issued. Stay tuned.

Ahh, okay! Do you think this ties into the issue I was seeing regarding not finding ROMs by using checksums?

muldjord

@Brunnis Sortof. :) I can't test anything right now as Skyscraper is blocked, so we'll see when the situation clears up.

mo418

@muldjord I was about to ask why I get « no game found » for all my roms using screenscraper.

Thanks for the heads up. Keep us posted (and keep up the good work!)

Regards :)

muldjord

The good people of Screenscraper has been very helpful in figuring out how to move forward with this and the issue is now resolved. Stay tuned for more info soon...