Versatile C++ game scraper: Skyscraper

muldjord · 6 Jan 2019, 14:24

I've been talking to the good people at IGDB and gotten some things cleared up. The "user-key" provided to me for the API is only meant for developers. As such, the 10k monthly limit should be applied to all Skyscraper users in total, not 1 key per user. So this will be changed to use a hidden key instead. The good thing is that people will then no longer need to supply their own to use it. The caveat is the limit obviously. But to keep the databases stable, we need to adhere to these things. And I certainly will with Skyscraper.

Silent · 6 Jan 2019, 16:50

That's great news! From what I have seen IGDB has very good resources, so it may improve quality of some scrapes (especially for newer games) significantly.

Halvhjearne · 7 Jan 2019, 18:49

really nice tool
skyscraper+screenscraper was able to scrape 98% of all my nintendo and sega roms. :D

however i do have a problem when scraping fba and mame roms.
as example:
1941 when scraped with sselph scraper returns the name "1941: Counter Attack (World 900227)" from arcadeitalia and 1941u returns the name "1941: Counter Attack (USA 900227)", but when i scrape it with skyscraper, all 1941 variants have the name "1941: Counter Attack" and its impossible to tell them apart.
is there a way to force it to return the name in the same way as sselph's scraper?
i noticed that the nintendo and sega games show version and region in the name, but my guess is thats maybe from the filenames?
idk

anyway, it would also be nice to see a detailed description of the options in config.ini, its rather hard to figure out how or what some of them does.

otherwise, keep up the good work :)

muldjord · 1 Jul 2019, 20:40

@Halvhjearne It already does so :) It's only in the output it doesn't show the bracket notes (if not you might have disabled it with the --nobrackets option or brackets="false" in config.ini. Please reload your ES and check it. It shows up as "1941: Counter Attack (World 900227)" here on my system.

Halvhjearne · 7 Jan 2019, 20:55

@muldjord
i tried it a few times and every time i scraped with skyscraper, all variants have the same name.

its running now with brackets="true" as i thought maybe i got it wrong and it takes a while to scrape, when only using 1 thread, however its almost done now and i will see what came out and if its still the same i will try with brackets="false", but isnt that supposed to be default behaviour?

im pretty sure i tried that too with same result or maybe im just confused by now ...

muldjord · 1 Jul 2019, 21:05

@Halvhjearne I think you are missing out a bit. If you scrape with the same scraping module twice, it will be really fast, unless you have enabled "--refresh". If refresh is enabled it will rescrape all of the files from the source again. No need for that. Skyscraper has a cache that is much faster. With refresh disabled, it will simply use the already cached data. I recommend skimming the documentation if you haven't already done so. Understanding how the cache works is pretty important if you want to use Skyscraper to its full potential: https://github.com/muldjord/skyscraper . It's a very powerful tool beyond just scraping from a single source.

Either way, I am not sure what you have done to your setup, but by default, if nothing in the config has been changed, it will have the USA and WORLD designations for fba and mame roms. I just tested it here and it works perfectly.

For a quick use case example, check here: https://github.com/muldjord/skyscraper/blob/master/USECASE.md

Halvhjearne · 7 Jan 2019, 21:10

@muldjord
so it only cache the pictures and not the text?

muldjord · 7 Jan 2019, 21:13

@Halvhjearne No, it caches everything. Including the USA and WORLD designations. But it's a bit much to explain here, try reading through the documentation. I think you'll understand once you have.

Halvhjearne · 7 Jan 2019, 21:29

@muldjord
i think i got it now, but i am using the "interface" from retropie setup and im pretty sure there is a bug with it, if its supposed to work as you explained.
i just scraped with brackets="true" in config.ini and now it shows up as i wanted it.

in the interface it seems not to care what setting its set to, it just removes brackets with fba and mame games, no matter what, but after setting brackets in config.ini, it now works.

i hope it still scrapes nintendo and sega games as expected tho ... lol

muldjord · 1 Jul 2019, 21:38

@Halvhjearne Have you updated the RetroPie script lately? There was a bug with the bracket option at one point I think. If that doesn't fix it, please report it to @mitu here (he's the one who made the RetroPie script for Skyscraper): https://retropie.org.uk/forum/topic/19588/skyscraper-now-officially-part-of-retropie-please-test

Halvhjearne · 1 Jul 2019, 21:46

@muldjord
it was updated within the last 3 days

edit:
i like that it does not sort "the" under t, although i almost got used to that by now, so probably wont be able to find the games i play that starts with "the" :D

muldjord · 7 Jan 2019, 21:53

@Halvhjearne Yeah, that's one of the earliest features I added back a year ago or so. :)

Halvhjearne · 9 Jan 2019, 02:06

i got an api key from igdb and i added userCreds="APIKEY" in config.ini, but for some reason it keeps saying the key dosnt work even if i run it at command line with -u APIKEY, am i doing something wrong?

also i noticed that when i set maxFails in config.ini over 200 it will always end when it reaches 42 and that it for some reason always refreshes when scraping mame-mame4all ... is this intended behavior?

sglavach · 9 Jan 2019, 05:53

@Halvhjearne There are responses to this issue below but in a nutshell IGDB scraping won't work until the next release and you won't need your personal api key as it will use an api key assigned to skyscraper. There will also be a limit to the number of skyscraper api calls per day/month I believe.

Baz · 9 Jan 2019, 09:09

I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

muldjord · 9 Jan 2019, 09:16

@Baz said in Versatile C++ game scraper: Skyscraper:

I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

That would indicate an error at screenscraper, yes. :)

Slow1 · 9 Jan 2019, 11:59

Firstly thanks for such an awesome scraper!

Got this running from a VM processing and storing all the data on my NAS and it's working great!! :)

Secondly, I have a question about matching ROMs that can't be found. Is there a nice easy way to do this via Skyscraper?

An example would be Bust-A-Move '99.cue with a MD5 of cd1fac631f6a45ae4ae2b28738ba4f6a which doesn't match anything on screenscraper.

However I expected it to match on name? I can see on the info pane on the left it's listed by name as Bust-a-move 3 Dx. Would my ROM need to be named to same (or rather added to the alias csv) in order to match? Or am I totally misunderstanding here :D

Edited: Added more info

muldjord · 9 Jan 2019, 12:52

@Slow1 It needs to match exactly one of the filenames at screenscraper and there needs to exist only one in total in the entirety of screenscraper before it works. If two platforms has a file with the same name, it won't return anything on filename search.

Instead I recommend checking the --query option as described here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

It allows you to overwrite the md5 sum for instance.

Slow1 · 9 Jan 2019, 13:19

@muldjord fantastic! Thank you - down I go..! :D

Baz · 9 Jan 2019, 20:27

@muldjord I could do with using this actually because I can't get Pokemon Blaze Black (a romhack version of Pokémon: Black Version) to scrape for love nor money.

I'm a complete linux noob but would it be $ Skyscraper -p nds -s screenscraper --query md5=37BFF1431EDA9B3A525737C7F59A432D PokemonBlazeBlack.nds , presuming I rename my rom to PokemonBlazeBlack.nds?