Versatile C++ game scraper: Skyscraper

muldjord

This post is deleted!

muldjord

Skyscraper 2.0.1 released: https://github.com/muldjord/skyscraper

Slightly changed help text for scraping modules
'thegamesdb' now properly uses Qt's XML parser
'screenscraper' now properly uses Qt's XML parser
Started implementing region and lang support for 'screenscraper', but still not enabled

Not the most interesting release from a user perspective. I am basically rewriting some stuff to make it work more standardized. But hey, it's still quite important work, so here it is! :D

screech

Hi @muldjord,

I'm thinking about something. not sure I tell it to you :

Do you know there is an API V2 on Screenscraper ?
It's not "the official" one for now, but soon ;)
It work exactly the same as the API V1, but the XML generated is much much easier to use ;)

to use it simply put a "2" just after "www.screenscraper.fr/api" in the URL like this :

https://www.screenscraper.fr/api2/jeuInfos.php?devid=xxx&devpassword=yyy&softname=zzz&output=xml&ssid=test&sspassword=test&crc=50ABC90A&systemeid=1&romtype=rom&romnom=Sonic The Hedgehog 2 (World).zip&romtaille=749652

not so much documentation, but if you are working on the country/language data, you must see this ;) maybe it will be easier for you to use the new API ;)

muldjord

@screech Haha, interesting! No, I didn't know it was online anywhere. I knew they were working on it, but not that it was already available. Seems like a bunch of the stuff I had gripes about with v1 has been fixed. Really cool!

Thanks for the heads-up! Do you have any idea how stable the format is? I mean, would it make sense to implement it now, or should I wait until xml format freeze?

EDIT: Especially I like that they now make use of attributes properly. It makes much more sense than before.

screech

Normally it's pretty stable. In the worst case more metada/média will be added. But on the same format. And Things already in it won't change ;)

BladeHunter

@screech, I just noticed that V2 doesn't list all the roms in a game:

https://www.screenscraper.fr/api/jeuInfos.php?devid=<removed>&devpassword=<removed>&devsoftname=RMS01output=xml&ssid=<removed>&sspassword=<removed>&md5=45e12558e1b9eda1b66b86f68f995f2a&sha1=533c497761cc802624708108bc9ef6e7c4f40296

Look at the bottom, there are 3 roms listed, 2 zip and 1 j64

https://www.screenscraper.fr/api2/jeuInfos.php?devid=<removed>&devpassword=<removed>&devsoftname=RMS01output=xml&ssid=<removed>&sspassword=<removed>&md5=45e12558e1b9eda1b66b86f68f995f2a&sha1=533c497761cc802624708108bc9ef6e7c4f40296

Only the j64 is listed.

Whilst this is the ROM I have specified in the url from the MD5 and SHA-1 shouldn't it still list the whole collection because it's the gameinfo part of the API not a ROMinfo part.

AnalogHero

Soooooo...... i fried my 2 month old 64 gb sd card (stupid), which gave me the "opportunity" to rescrape my systems with the latest version.

And i must say that the old versions were better due to more scrape-sources. I know what the issues were, but i wish that i had a backup of either the old Skyscraper or the localdb.

Other than that it works flawless. :)

One thing what could be added is that if it doesnt find a match in the current module, try another one, as an option.

Edit: Scraping psx with screenscraper aborts with std::bad_alloc

muldjord

Skyscraper 2.0.2 released: https://github.com/muldjord/skyscraper

Updated 'arcadedb' result parsing to fit new format
Now scrapes 'msx' platform families correctly with the 'screenscraper' module
Changed limit for iso checksumming to 20 megs to avoid running out of memory.

Mostly minor stuff and a bug fix. :)

muldjord

@AnalogHero, The crash comes from running out of memory when trying to do a sha1 checksum of the iso files. I've changed the limit in 2.0.2 so please try that and see if it works. Either way, the psx platform won't work with the screenscraper module, simply because the raspberry pi doesn't have enough memory to checksum the iso files. So you'll have to use thegamesdb for those. :)

Concerning trying another module while scraping if it doesn't find one with the current one: This won't be implemented, simply because it's already possible by scraping with another scraper afterwards. Just choose to "skip existing". :) And always remember to finish off by scraping with 'localdb' to get the best result from all cached data. :) Have fun!

muldjord

Skyscraper 2.0.3 released: https://github.com/muldjord/skyscraper

Added '--lang' and '--region' command line options. Defaults to 'us' and 'wor' if none is provided.
Language and region support now implemented for 'screenscraper' module.
Removed 'stats' option as it didn't make much sense anymore.

You can now set language and region which will be used currently by the 'screenscraper' module only. In a later release I will implement the option to get a list of supported languages and regions. But for now I just wanted to enable it. It defaults to language 'us' and region 'wor' if none is provided.

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

Either way, the psx platform won't work with the screenscraper module, simply because the raspberry pi doesn't have enough memory to checksum the iso files.

That's not necessarily true - the md5sum standard utility can operate in a constant memory space, no matter how large is the input stream/file. The checksumming does not require the whole file contents to be in memory, MD5 requires only reading sequentially 512 bits from the input.

muldjord

@mitu Thanks for pointing this out. I actually started doing sha1 and md5 summing differently in 2.0.2. Instead of feeding the functions the entire file data I feed it bits of 1 Meg until it's done. So that would actually mean that it has already been fixed. :)

AnalogHero

@muldjord Works with psx now. A little slow, but no problem if you dont have a big psx collection like me.

Thanks for the update. :)

muldjord

@analoghero Glad it works :)

I'm gonna start work on a gui now. This will be a whopper so it'll be a while before the next release.

BenMcLean

EmulationStation needs a way to quickly delete incorrect scraped info on a game in the GUI from the gamepad without deleting the game. That's more important than the actual scraper being improved IMO. But this is still great though. Thanks for all your hard work.

A Former User

@benmclean Yr barking up the wrong tree as @muldjord doesn't develop for ES.

@muldjord Is there a feature, or are you planning on a feature to scrape a single element and append the gamelist? Say, I wanted the title and genre from GamesDB, published and developed info from Screenscraper, etc.

muldjord

@livefastcyyoung There's not a feature to just scrape one title per se. But you can always just copy just 1 rom to a different folder, set the input folder to that folder with '-i'. Then scrape that rom using the different modules 1 by 1, then prioritize the resource type in '~/.skyscraper/dbs/[platform]/priorities.xml' and finish by scraping using '-s localdb'. That would give you exactly the result you are asking for, except it would have created a new gamelist.xml file with just that entry in it, so you'd have to copy the files over manually afterwards.

If you are in doubt about the features I mention, please check the readme's on github. It's all very well explained in there (although I am aware that it can seem confusing since Skyscraper can do so many things! Hence why I am working on a gui).

I do not have plans to enable scraping just 1 rom from a folder in a simple way. It doesn't work well with the current design of the code and how it creates the gamelist.xml, sorry.

A Former User

@muldjord I think I was misunderstood. I wasn’t referring to a single rom. I was referring to scraping a set of roms as we can now but setting a flag to just scrape the genre, release date, etc.

I’ve found no one database has all elements in great shape so putting together a gamelist by scraping the elements that database does best seems like the next best alternative.

Hopefully that made sense now.

AnalogHero

@LiveFastCyYoung Im not sure that this is what youre looking for but you can set priorities in the priorites.xml, but its only for a whole system. You cant say use info A on rom A and info B on rom B afaik.

A gui will really help here imo.

muldjord

@livefastcyyoung Makes perfect sense now. That is not currently supported by Skyscraper but it can do just what you want in a similar way. Have you read up on the '-s localdb' scraping module contained in Skyscraper? You can basically do what you are referring to simply prioritizing the resources correctly. It's all in the readme's. :)