Versatile C++ game scraper: Skyscraper

muldjord

@SammyBoy
'*.scummvm' added to scummvm platform. Will be in 2.3.0.

I am currently testing all of the new composite effects. I'm getting real close to being done with all the features I want in 2.3.0, and yes, it is indeed the biggest Skyscraper release to date. And also the toughest to code so far. :D But it's turning out much better than I had expected, so I really look forward to getting it out to you guys. It's a matter of days now unless I run into something unexpected.

AnalogHero

@muldjord I know youre busy with 2.3.0 which is released any moment, but i tried something which doesnt work for me.

Problem: Wings.uae is scraped as Wings of Fury.
Tried Screenscraper Wings.uae -p amiga but it just scrapes everything.

muldjord

@analoghero Are you in the correct folder? I mean, are you in the folder where "Wings.uae" is located? Also, I assume you meant "Skyscraper" not "Screenscraper".

Openretro doesn't seem to have "Wings" on their website which is really odd... So try scraping with '-s thegamesdb' instead. That might help.

AnalogHero

@muldjord Lol ofcourse Skyscraper, just a typo.

Yes im in the right folder, but even with -s thegamesdb added it asks me if i want to overwrite gamelist.xml and if i want to skip existing entries as usual.

muldjord

@analoghero Ah, I know why. You need to put the full path on the filename. I will look into fixing this for 2.3.0 aswell. Thanks for letting me know.

muldjord

Quick update: All of the effects are now done and working. There's one hiccup though. When I scrape a bunch of files with screenscraper I get a sporadic segfault (crash) after about 20 minutes in. Bad news is that it's not the same place each time which means this could take me a while to track down... I'll keep you updated on the progress.

AnalogHero

@muldjord Yes, with full path it works. I wonder why i didnt try that. In fact i tried Skyscraper ./Wings.uae -p amiga -s thegamesdb but that didnt work, too.

EDIT: Im confused now. When i see that the scraper has a name mismatch, as in my examples Wings gets confused with Wings of death or Wings of Fury, id like to rescrape that only game with another scraper (-s thegamesdb for example).

But the correct result doesnt get written to the gameslist.xml. Rescraping with -s localdb doesnt help either, as it sure gets mismatched again.
Tried scraping with -m 100 but this is a 100% match.

Another thing related: Suddenly all my Shadow of the beast games have no number in the name, dont have the right cover orscreenshot. Ill try deleting the dbs and see if it helps.

muldjord

@analoghero Yes, it's because I change directory to "~/.skyscraper" immediately after Skyscraper starts, to make it easier to get to the files I need. But I have a workaround in mind that will allow you to do both "Wings.uae" and "./Wings.uae".

muldjord

@analoghero You need to look at "~/.skyscraper/dbs/[platform]/priorities.xml". That file defines source priorities for the different resources. Currently you probably have openretro above thegamesdb for a bunch of the resource types which makes it prioritize the faulty entries from Openretro. So just move the "<source>thegamesdb</source> up above the "<source>openretro</source>" node for all resources and it should work as you expect.

EDIT: And yes, the gamelist.xml is NOT being written when scraping filenames on command line. It merely adds the data to the local cache so you can use it with localdb as you did.

EDIT2: I will advice against deleting the entire dbs folder. Not that it breaks anything, but if you have scraped all platforms, most of the dbs inside the folder are absolutely fine and will be useful for rescrapings later. Instead, consider deleting just the one folder inside dbs matching the platform you are having trouble with. But, best of all, consider using the prioritize feature as described in the documentation.

AnalogHero

@muldjord I just deleted the amiga folder insided the dbs folder. Its not that big loss.

I was thinking about priorities.xml but its a bit hard to decide, as sometimes it isnt clear which source is better or its depends on the game.

Another thing bugs me for long time: Can the result in number of players somehow streamlined? Most Emulationstation themes cant handle any text. 1 - 2 (2 simultaneous) would only display garbage.

hi_im_daft

Question: I know there's a way to do it, and it's probably posted somewhere but I haven't found it yet.

Can you change the priority for databases when the scraper runs? I don't like the current "primary" database, it throws in a lot of incorrect game names (or maybe just defaults to EU region titles instead of US), so I want to change it to a different one. I'm sure it's in a configuration somewhere to do it.

muldjord

@hi_im_daft Please read the documentation: https://github.com/muldjord/skyscraper
You set scraper with '-s'. Check all options with '--help'. Please keep in mind that if you only scrape a platform with one scraper you are not using Skyscraper to the best of it's ability. Check here for more information on the local database cache: https://github.com/muldjord/skyscraper#local-database-features

bleuge

@muldjord I have some little question. I run Skyscraper in all my folders, some roms are not identified and so no images or metadata.
But, then, if i run sselph scraper, and end populating the lists, because it can find some things that Skycraper can't.

If i run again Skyscraper, the local db is updated with the new content downloaded with sselph? So it can be more complete and useful in the future?

Thanks!

AnalogHero

@bleuge Try running skyscraper more than one time per system with different scrapers each time.

You can specify the scraper with -s So if you try to scrape snes do your second run like this : Skyscraper -p snes -s screenscraper . Maybe after this you can do: Skyscraper -p snes -s openretro

After it finished do as final step Skyscraper -p snes -s localdb so that all scraped resources are used.

bleuge

@analoghero I am using skyscript.sh generated script, for every platform. So i guess this is right.

hi_im_daft

If it uses an amalgam of information from different databases though, how does it know what info is "correct"? Or, at least, more accurate to use? That's the problem I fear I'm going to wind up with.

muldjord

@hi_im_daft You can prioritize sources for each type of information (check the documentation). But you will undoubtedly end up with some faulty entries, that can't be avoided with an automated scraper like this. If you only want correct results, just only ever scrape with 'screenscraper' and 'arcadedb'. Those are based on mame names and file checksums. But then you'll give up the option of scraping platforms that have scripts to start the games. Such as svm for scummvm and cue files for psx.

Bottom line, you can't avoid faulty results with sources that are based on file name searches.

muldjord

In release related news: I think I tracked down the segfault that made Skyscraper crash sporadically. And I might even have fixed the rare problem some have been reporting where image and or resources would have been mixed up.

So, once again, I will be doing testing over the next days. And if I don't run into more trouble, I can start documenting the new compositor and, finally, release 2.3.0... Phew. I have literally been coding on this for 6-10 hours a day for the past week. I hope you guys appreciate this! It's starting to take it's toll here :D

hi_im_daft

@muldjord: I'm sure I can say for many of us here that we definitely appreciate the effort. The system itself is pretty great.

One last silly question, is it possible to set a different UI theme to use different Metadata? Some themes seems to look a lot nicer with the layout you use (screenshot with boxart in the corner), while others have a smaller area for images and squishing yours looks worse.

For example, I think the Skyscraper images look great with Retrorama, but for the NES Mini theme the boxart by itself would look better.

I don't think there's a way to do this one, but I am curious.

AnalogHero

@bleuge Ok if youre using simplemode the script is correct. Sometimes roms doesnt get scraped because the match wasnt close enough. What im doing then is lowering the -m value. For example Skyscraper -p snes -m 50. Answer Y to overwrite and Y to skip existing entries . You can lower the value if you like in a second run.

@hi_im_daft If the information online isnt correct, your scraped data will be so. If you have like 700 snes games for example, i would never know if a few games have incorrect descriptions.