Versatile C++ game scraper: Skyscraper

Halvhjearne · 9 Jan 2019, 02:06

i got an api key from igdb and i added userCreds="APIKEY" in config.ini, but for some reason it keeps saying the key dosnt work even if i run it at command line with -u APIKEY, am i doing something wrong?

also i noticed that when i set maxFails in config.ini over 200 it will always end when it reaches 42 and that it for some reason always refreshes when scraping mame-mame4all ... is this intended behavior?

sglavach · 9 Jan 2019, 05:53

@Halvhjearne There are responses to this issue below but in a nutshell IGDB scraping won't work until the next release and you won't need your personal api key as it will use an api key assigned to skyscraper. There will also be a limit to the number of skyscraper api calls per day/month I believe.

Baz · 9 Jan 2019, 09:09

I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

muldjord · 9 Jan 2019, 09:16

@Baz said in Versatile C++ game scraper: Skyscraper:

I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

That would indicate an error at screenscraper, yes. :)

Slow1 · 9 Jan 2019, 11:59

Firstly thanks for such an awesome scraper!

Got this running from a VM processing and storing all the data on my NAS and it's working great!! :)

Secondly, I have a question about matching ROMs that can't be found. Is there a nice easy way to do this via Skyscraper?

An example would be Bust-A-Move '99.cue with a MD5 of cd1fac631f6a45ae4ae2b28738ba4f6a which doesn't match anything on screenscraper.

However I expected it to match on name? I can see on the info pane on the left it's listed by name as Bust-a-move 3 Dx. Would my ROM need to be named to same (or rather added to the alias csv) in order to match? Or am I totally misunderstanding here :D

Edited: Added more info

muldjord · 9 Jan 2019, 12:52

@Slow1 It needs to match exactly one of the filenames at screenscraper and there needs to exist only one in total in the entirety of screenscraper before it works. If two platforms has a file with the same name, it won't return anything on filename search.

Instead I recommend checking the --query option as described here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

It allows you to overwrite the md5 sum for instance.

Slow1 · 9 Jan 2019, 13:19

@muldjord fantastic! Thank you - down I go..! :D

Baz · 9 Jan 2019, 20:27

@muldjord I could do with using this actually because I can't get Pokemon Blaze Black (a romhack version of Pokémon: Black Version) to scrape for love nor money.

I'm a complete linux noob but would it be $ Skyscraper -p nds -s screenscraper --query md5=37BFF1431EDA9B3A525737C7F59A432D PokemonBlazeBlack.nds , presuming I rename my rom to PokemonBlazeBlack.nds?

muldjord · 9 Jan 2019, 20:44

@Baz Looks correct yes :)

megaJekan · 11 Jan 2019, 07:54

Hi there. I am used to emulationstation internal scrapper and just switched to skyscraper few days ago. In general, I like the way it works, also it could easily scrap my TurboGrafx-CD images. But there are few things I wanted to specify.

It seems like after scrapping (using retropie menu), the games are removed from the favourites list. Could it be fixed somehow or is there a parameter in config.ini to keep games in the list?

Maybe you can also advice how to search only for newly added roms/images without updating and overwriting the whole database. I have some manual changes in Mega CD names (like disc 1, disc 2), which I would like to keep.

mitu · 1 Nov 2019, 08:08

@megaJekan said in Versatile C++ game scraper: Skyscraper:

It seems like after scrapping (using retropie menu), the games are removed from the favourites list. Could it be fixed somehow or is there a parameter in config.ini to keep games in the list?

What version if Skyscraper are you using ? This was fixed in 2.8.0.

megaJekan · 11 Jan 2019, 08:14

@mitu I use v2.9.5. I've done it only through "scrap chosen systems". It is 100% reproducible for PC Engine, at least.

mitu · 11 Jan 2019, 08:16

@megaJekan I'll check on my install - thanks for reporting it.

megaJekan · 1 Nov 2019, 08:22

@megaJekan said in Versatile C++ game scraper: Skyscraper:

Maybe you can also advice how to search only for newly added roms/images without updating and overwriting the whole database. I have some manual changes in Mega CD names (like disc 1, disc 2), which I would like to keep.

@mitu Thank you. Could you please also advice any workaround for this?

mitu · 11 Jan 2019, 08:25

@megaJekan Hm, why would you need separate entries for Disc1/Disc2 ? Shouldn't they be collapsed in a single entry (using a .cue or an .m3u file) in the gamelist ?

megaJekan · 11 Jan 2019, 08:46

@mitu said in Versatile C++ game scraper: Skyscraper:

@megaJekan Hm, why would you need separate entries for Disc1/Disc2 ? Shouldn't they be collapsed in a single entry (using a .cue or an .m3u file) in the gamelist ?

I have *.m3u files for PSX, they work fine. But for Sega CD they are not in the list of parsed file formats.
https://github.com/retropie/retropie-setup/wiki/Sega-CD

megaJekan · 11 Jan 2019, 09:01

@mitu Anyway, this is just an example. Sometimes you have really long titles scrapped (which include names for all regions + system name), which you don't like. And you just wanna keep your manual edits unchanged. Like in the case of Metal Slug or Samurai Shodown for Neo Geo.

muldjord · 1 Nov 2019, 10:13

I just tested the favorite preservation, and it works just fine. I'd need to test it with the GUI. Maybe there's something odd in that implementation.

muldjord · 11 Jan 2019, 10:16

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Silent · 11 Jan 2019, 11:18

@muldjord said in Versatile C++ game scraper: Skyscraper:

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Using it on an entire gamelist sounds like a disaster though, so may be best to scrape single ROMs with this only.