Versatile C++ game scraper: Skyscraper

Halvhjearne

this is awsome, it was pretty scattered before and this info clarifies things a lot.
only thing i can think of to make it any better would be to maybe add the equivalent option for config.ini to next to the command line option.

sglavach

I have Skyscraper (SS) installed and running on my local machine, I have a number of SBC's (RPi, Odroid, etc..) running and connected to network as well. I can point to the Rom path on each SBC using the -i parameter and it will use the localdb on my local machine and it will scan the corresponding SBC Rom path and create the media correctly on each SBC. My challenge is with Videos since they take up a lot of space. I would rather not copy my videos (already on the SBC's) to the import directory on my local machine and then import them prior to scraping, I would like to point to a non-default import path for my videos (same way I point to a non-default Rom path with -i)... import them from there and then scrape. Is this possible? am I missing a way to accomplish this? In fact having a way to point/configure a path for any non-local import resource would be helpful. Thanks!

raspy+

@muldjord

Wow so much reading and I still can't find the answer to a simple question unless I missed it and fell asleep at the keyboard LOL...

Q: What does each color represent when Skyscraper is collecting data?

(green number) / (yellow number) / (blue number)

Green number = how many roms it checked?
Yellow number = how many roms it skipped?
Blue number = how many confirmed roms matched?

Just a guess on my part or did I just answer my own question?

Q2: If Skyscraper can't find a game while scraping, does that mean it can't find a match from "screenscraper.fr"?

Silent

Just a thought I had due to my invalid bug report (which I have now closed):

A lot of games have region tags, like (E), [U] in their filenames. Could Skyscraper optionally infer region to use when scraping a ROM from this data? This way it would try to collect data matching what you actually see in the game (example: Star Fox vs Star Wing) without the user having to tailor that manually on per-ROM basis.

muldjord

@Silent said in Versatile C++ game scraper: Skyscraper:

Just a thought I had due to my invalid bug report (which I have now closed):

A lot of games have region tags, like (E), [U] in their filenames. Could Skyscraper optionally infer region to use when scraping a ROM from this data? This way it would try to collect data matching what you actually see in the game (example: Star Fox vs Star Wing) without the user having to tailor that manually on per-ROM basis.

I already do this for (Europe) and others. I will include the ones you suggest aswell.

muldjord

@raspy

Q1: found/skipped/total

Q2: Depends on what scraping module you are scraping with. But yes, if it's not found, that means it's not found at the source using the rom information available to Skyscraper.

muldjord

@sglavach You can set a non-default import folder in config.ini with importFolder="[folder]"

muldjord

@RDBowers Skyscraper currently supports cover, screenshot, wheel and marquee resources. If you wish to use other more niche resources, I suggest using the import functionality to get them. Then you can import them as "marquee" resources for instance, and use them in the artwork.xml for the arcade platforms.

Halvhjearne

@muldjord

lets say i scrape fba once with screenscraper without videos and then later decides that i actually also wanted the videos, is there a way to make it download the videos without refreshing everything?

and does this means it does not check already scraped games for missing info/images either?

muldjord

@Halvhjearne You need to refresh everything, there is no way to only refresh one resource from a source. So just enable videos and use --refresh.

I don't understand your second question.

Halvhjearne

@muldjord

you basicly answered both.

would be nice to be able to refresh only one resource like covers or videos tho tho or maby an option to refresh missing resources.

muldjord

@Halvhjearne It's been requested before, but I don't really see why it would be useful (and it's quite a huge task to code). The data retrieved from the source is the same no matter how much of it Skyscraper decides to use. So it's not relevant for optimization reasons.

Halvhjearne

@muldjord

ahh i see, i thought maybe it would put less strain on the servers.

raspy+

Thanks for the reply to my questions muldjord!

Slow1

@muldjord thanks a bunch for doing that! :D

Halvhjearne

@muldjord

i noticed that it does not scrape scummvm games correctly, perhaps course of the wierd filenames?
the actual name of the game is actually listet inside the .svm files if that helps any.
i also noticed the alias.csv (or whatever its called) file to change the output name, but is there a way to tell skyscraper what name it should search for instead of the filename?

muldjord

@Halvhjearne

It already looks up the name in the scummvm.ini file. I tested this on my own setup only a couple weeks ago. What is the location of your scummvm.ini file?
Yes, check the --query option here :)

hhromic

@muldjord for the (recent) lr-scummvm core things work a bit different:

Games do not need to be added to the scummvm.ini file, which is located in $biosdir/scummvm. Users can add them manually but for "plug-and-play" operation it is not necessary so games might not appear there.
The gameid (that is in scummvm.ini file) is instead always in the .svm files inside the game folders. For example you can have atlantis.svm with the string atlantis inside. That is where you should be looking for the game ids.

So I would scrape the svm files indeed by reading their content for the game ids. See more information on how the games are setup in this topic. I plan to write official documentation about the core as soon as I have more time. Cheers!

Halvhjearne

@muldjord

if thats the case, i think it fails sometimes, for me it only writes correct names when i write them in aliasMap.csv.
i also think there is something funky going on with the brackets removal ... its a bit hit and miss for each console, but seems to work well with most nintendo and sega games or fba and mame with arcadeitalia only.

is there a way to use aliasMap.csv for only one/each platform?

thanks, i think i missed that option ... :)

muldjord

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

is there a way to use aliasMap.csv for only one/each platform?

No, it's global