Versatile C++ game scraper: Skyscraper

megaJekan

@muldjord said in Versatile C++ game scraper: Skyscraper:

@megaJekan Also be sure to check out the module esgamelist which will scrape data from an existing game list. So if you've changed a bunch of data for certain roms, you can add it to the Skyscraper cache using this module, which will then use it when scraping with the localdb module afterwards. That's a great way to preserve information you've changed yourself in the gamelist.xml files. I can't remember if that module is available from the GUI though, so you might need to get familiar with the command line version.

Thanks, I will try this out. Though it would be nice anyway to have GUI for scrapping single ROMs from the checklist as well as single platforms. Searching for newly added games (like emulationstation scrapper did) is of course easier, but the ability just to select what you scrap is good enough.

muldjord

@megaJekan If you venture into so specific requirements, I'd simply recommend using it on command line. It's really easy once you get the hang of it.

muldjord

Hi guys, I just spend the last 4 hours documenting all of the command line options down to the last detail. Check it out here.

I will do the same for all config.ini options and all of the scraping modules when I get the time.

I've also moved pretty much all documentation to the /docs subfolder on github. Get a nice overview here: https://github.com/muldjord/skyscraper/tree/master/docs

@mitu You might want to include a link to the command line option doc in the wiki.

mitu

@muldjord Thanks, I'll take a look when adding the new module additions.

Are you still including the 'simple mode' with the new version ? I was thinking of adding 2 new things related to the new release

a Simple Mode switch: toggling it to on will automatically do a scrape with localdb after an online scrape. Since we intend to separate scraping from localdb from other scraping modules, it somehow makes sense and would resemble the the simple mode you have.
if the user runs a scraping operation, but doesn't scrape with localdb, to give a warning if it tries to exit the GUI to say it needs to scrape with localdb because there were no gamelist.xml generated.

muldjord

@mitu I think it should be like this:

There should be a main menu item simply called "Gather resources" inside which all of the non-localdb modules are listed with a menu item (pretty much as they are now). All these do when selecting them is scrape data, nothing else. It shouldn't auto-run localdb afterwards as it would simply muddy up the process. The "--refresh" bool option should be moved inside this "Gather resources" submenu. There needs to be a help text on this submenu clearly stating that running any of those options will cache resources, but not generate the actual game list.

The "localdb" module menu item should moved out to be a main menu item simply called "Generate game list(s)". No mention of "localdb" anymore as I wish to do away with it as much as possible (it is the default scraping module after all, so the user doesn't need to know about it). There needs to be a nice help text for this option stating that it will generate a game list from all previously cached resources.

So they'd gather a bunch of resources from the "Gather" submenu, and lastly go back to the main menu and choose "Generate game list". And voila, it's all nice and split up to avoid confusion.

megaJekan

Besides that scraping PSX is a total nightmare (it does it so bad with CUE/BIN images), the only game, which I couldn't really scrap was Q*bert (fba or mame-libretro). I even tried to run the command manually with query option, it didn't help:
https://www.screenscraper.fr/gameinfos.php?gameid=172110&action=onglet&zone=gameinfosinfos

There was also an issue in Github:
https://github.com/muldjord/skyscraper/issues/62

For me it seems like the 'asterisk' sign in the name is a problem.

Silent

@megaJekan said in Versatile C++ game scraper: Skyscraper:

Besides that scraping PSX is a total nightmare

Manually hashing .bin files and providing that as a query parameter for screenscraper module provides godlike results. I spent some time doing that and it was very worth it.

megaJekan

@Silent said in Versatile C++ game scraper: Skyscraper:

@megaJekan said in Versatile C++ game scraper: Skyscraper:

Besides that scraping PSX is a total nightmare

Manually hashing .bin files and providing that as a query parameter for screenscraper module provides godlike results. I spent some time doing that and it was very worth it.

Sorry, I think I didn't get the point. What if my md5 hash doesn't match the one from screenscraper site? Because most of the images were converted manually by my own from CCD/IMG/SUB to CUE/BIN. Also I can just take hashes directly from screenscraper and use them as an argument. Then I don't see the logic at all, why name comparison works so bad. So I'm a bit lost.

mitu

@megaJekan Can you give some examples ?

RDBowers

My RetroPie setup uses the Daphne and MAME emulators exclusively. I have used Skyscraper to collect "resources" into the LocalDB and created custom images by editing the artwork.xml. This is all working great!

However, I would like to include a cabinet image as part of the custom image for each arcade game. I think this would be more appropriate than including the "box" image.

Can Skyscraper pull cabinet images from the online sources making them available as a resource for the LocalDB scrape and creation of custom images? I have read and re-read the online documentation and I don't believe this is currently possible. Maybe this could be an addition to a future release of Skyscraper.

Any suggestions for a possible workaround?

mitu

@RDBowers You can add your own image (a cabinet instead of a box) in the artwork.xml file and have it part of the final image. Is that what you're trying to do ?

RDBowers

@mitu That is what I am trying to do.

My question is, can Skyscraper pull cabinet images from online sources, save them as part of the LocalDB, making them available as a resource (just like a screenshot, cover, wheel or marquee) to be referenced in the artwork.xml?

mitu

@RDBowers I don't think it's supported right now. Can you give more details about where these images can be downloaded (site/source), maybe an example of such image associated with an arcade game ?

RDBowers

@mitu One source already built into Skyscraper is adb.arcadeitalia.net. For most arcade ROMs they include a cabinet image. On their Pac-man page you will find a link to this image.

Pac-man cabinet

Another potential source of cabinet images not built into Skyscraper is arcade-museum.com.

They are also a great source for additional information, images and flyers. The Arcade Flyer Archive

Halvhjearne

@muldjord

this is awsome, it was pretty scattered before and this info clarifies things a lot.
only thing i can think of to make it any better would be to maybe add the equivalent option for config.ini to next to the command line option.

sglavach

I have Skyscraper (SS) installed and running on my local machine, I have a number of SBC's (RPi, Odroid, etc..) running and connected to network as well. I can point to the Rom path on each SBC using the -i parameter and it will use the localdb on my local machine and it will scan the corresponding SBC Rom path and create the media correctly on each SBC. My challenge is with Videos since they take up a lot of space. I would rather not copy my videos (already on the SBC's) to the import directory on my local machine and then import them prior to scraping, I would like to point to a non-default import path for my videos (same way I point to a non-default Rom path with -i)... import them from there and then scrape. Is this possible? am I missing a way to accomplish this? In fact having a way to point/configure a path for any non-local import resource would be helpful. Thanks!

raspy+

@muldjord

Wow so much reading and I still can't find the answer to a simple question unless I missed it and fell asleep at the keyboard LOL...

Q: What does each color represent when Skyscraper is collecting data?

(green number) / (yellow number) / (blue number)

Green number = how many roms it checked?
Yellow number = how many roms it skipped?
Blue number = how many confirmed roms matched?

Just a guess on my part or did I just answer my own question?

Q2: If Skyscraper can't find a game while scraping, does that mean it can't find a match from "screenscraper.fr"?

Silent

Just a thought I had due to my invalid bug report (which I have now closed):

A lot of games have region tags, like (E), [U] in their filenames. Could Skyscraper optionally infer region to use when scraping a ROM from this data? This way it would try to collect data matching what you actually see in the game (example: Star Fox vs Star Wing) without the user having to tailor that manually on per-ROM basis.

muldjord

@Silent said in Versatile C++ game scraper: Skyscraper:

Just a thought I had due to my invalid bug report (which I have now closed):

A lot of games have region tags, like (E), [U] in their filenames. Could Skyscraper optionally infer region to use when scraping a ROM from this data? This way it would try to collect data matching what you actually see in the game (example: Star Fox vs Star Wing) without the user having to tailor that manually on per-ROM basis.

I already do this for (Europe) and others. I will include the ones you suggest aswell.

muldjord

@raspy

Q1: found/skipped/total

Q2: Depends on what scraping module you are scraping with. But yes, if it's not found, that means it's not found at the source using the rom information available to Skyscraper.