Versatile C++ game scraper: Skyscraper

Slow1

@muldjord fantastic! Thank you - down I go..! :D

Baz

@muldjord I could do with using this actually because I can't get Pokemon Blaze Black (a romhack version of Pokémon: Black Version) to scrape for love nor money.

I'm a complete linux noob but would it be $ Skyscraper -p nds -s screenscraper --query md5=37BFF1431EDA9B3A525737C7F59A432D PokemonBlazeBlack.nds , presuming I rename my rom to PokemonBlazeBlack.nds?

muldjord

@Baz Looks correct yes :)

megaJekan

Hi there. I am used to emulationstation internal scrapper and just switched to skyscraper few days ago. In general, I like the way it works, also it could easily scrap my TurboGrafx-CD images. But there are few things I wanted to specify.

It seems like after scrapping (using retropie menu), the games are removed from the favourites list. Could it be fixed somehow or is there a parameter in config.ini to keep games in the list?

Maybe you can also advice how to search only for newly added roms/images without updating and overwriting the whole database. I have some manual changes in Mega CD names (like disc 1, disc 2), which I would like to keep.

mitu

@megaJekan said in Versatile C++ game scraper: Skyscraper:

It seems like after scrapping (using retropie menu), the games are removed from the favourites list. Could it be fixed somehow or is there a parameter in config.ini to keep games in the list?

What version if Skyscraper are you using ? This was fixed in 2.8.0.

megaJekan

@mitu I use v2.9.5. I've done it only through "scrap chosen systems". It is 100% reproducible for PC Engine, at least.

mitu

@megaJekan I'll check on my install - thanks for reporting it.

megaJekan

@megaJekan said in Versatile C++ game scraper: Skyscraper:

Maybe you can also advice how to search only for newly added roms/images without updating and overwriting the whole database. I have some manual changes in Mega CD names (like disc 1, disc 2), which I would like to keep.

@mitu Thank you. Could you please also advice any workaround for this?

mitu

@megaJekan Hm, why would you need separate entries for Disc1/Disc2 ? Shouldn't they be collapsed in a single entry (using a .cue or an .m3u file) in the gamelist ?

megaJekan

@mitu said in Versatile C++ game scraper: Skyscraper:

@megaJekan Hm, why would you need separate entries for Disc1/Disc2 ? Shouldn't they be collapsed in a single entry (using a .cue or an .m3u file) in the gamelist ?

I have *.m3u files for PSX, they work fine. But for Sega CD they are not in the list of parsed file formats.
https://github.com/retropie/retropie-setup/wiki/Sega-CD

megaJekan

@mitu Anyway, this is just an example. Sometimes you have really long titles scrapped (which include names for all regions + system name), which you don't like. And you just wanna keep your manual edits unchanged. Like in the case of Metal Slug or Samurai Shodown for Neo Geo.

muldjord

I just tested the favorite preservation, and it works just fine. I'd need to test it with the GUI. Maybe there's something odd in that implementation.

muldjord

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Silent

@muldjord said in Versatile C++ game scraper: Skyscraper:

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Using it on an entire gamelist sounds like a disaster though, so may be best to scrape single ROMs with this only.

muldjord

@Silent said in Versatile C++ game scraper: Skyscraper:

@muldjord said in Versatile C++ game scraper: Skyscraper:

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Using it on an entire gamelist sounds like a disaster though, so may be best to scrape single ROMs with this only.

Yes, I agree.

megaJekan

@muldjord said in Versatile C++ game scraper: Skyscraper:

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Thanks, I will try this out. Though it would be nice anyway to have GUI for scrapping single ROMs from the checklist as well as single platforms. Searching for newly added games (like emulationstation scrapper did) is of course easier, but the ability just to select what you scrap is good enough.

muldjord

@megaJekan If you venture into so specific requirements, I'd simply recommend using it on command line. It's really easy once you get the hang of it.

muldjord

Hi guys, I just spend the last 4 hours documenting all of the command line options down to the last detail. Check it out here.

I will do the same for all config.ini options and all of the scraping modules when I get the time.

I've also moved pretty much all documentation to the /docs subfolder on github. Get a nice overview here: https://github.com/muldjord/skyscraper/tree/master/docs

@mitu You might want to include a link to the command line option doc in the wiki.

mitu

@muldjord Thanks, I'll take a look when adding the new module additions.

Are you still including the 'simple mode' with the new version ? I was thinking of adding 2 new things related to the new release

a Simple Mode switch: toggling it to on will automatically do a scrape with localdb after an online scrape. Since we intend to separate scraping from localdb from other scraping modules, it somehow makes sense and would resemble the the simple mode you have.
if the user runs a scraping operation, but doesn't scrape with localdb, to give a warning if it tries to exit the GUI to say it needs to scrape with localdb because there were no gamelist.xml generated.

muldjord

@mitu I think it should be like this:

There should be a main menu item simply called "Gather resources" inside which all of the non-localdb modules are listed with a menu item (pretty much as they are now). All these do when selecting them is scrape data, nothing else. It shouldn't auto-run localdb afterwards as it would simply muddy up the process. The "--refresh" bool option should be moved inside this "Gather resources" submenu. There needs to be a help text on this submenu clearly stating that running any of those options will cache resources, but not generate the actual game list.

The "localdb" module menu item should moved out to be a main menu item simply called "Generate game list(s)". No mention of "localdb" anymore as I wish to do away with it as much as possible (it is the default scraping module after all, so the user doesn't need to know about it). There needs to be a nice help text for this option stating that it will generate a game list from all previously cached resources.

So they'd gather a bunch of resources from the "Gather" submenu, and lastly go back to the main menu and choose "Generate game list". And voila, it's all nice and split up to avoid confusion.