Versatile C++ game scraper: Skyscraper
-
Q1: found/skipped/total
Q2: Depends on what scraping module you are scraping with. But yes, if it's not found, that means it's not found at the source using the rom information available to Skyscraper.
-
@sglavach You can set a non-default import folder in config.ini with
importFolder="[folder]"
-
@RDBowers Skyscraper currently supports cover, screenshot, wheel and marquee resources. If you wish to use other more niche resources, I suggest using the import functionality to get them. Then you can import them as "marquee" resources for instance, and use them in the artwork.xml for the arcade platforms.
-
lets say i scrape fba once with screenscraper without videos and then later decides that i actually also wanted the videos, is there a way to make it download the videos without refreshing everything?
and does this means it does not check already scraped games for missing info/images either?
-
@Halvhjearne You need to refresh everything, there is no way to only refresh one resource from a source. So just enable videos and use
--refresh
.I don't understand your second question.
-
you basicly answered both.
would be nice to be able to refresh only one resource like covers or videos tho tho or maby an option to refresh missing resources.
-
@Halvhjearne It's been requested before, but I don't really see why it would be useful (and it's quite a huge task to code). The data retrieved from the source is the same no matter how much of it Skyscraper decides to use. So it's not relevant for optimization reasons.
-
ahh i see, i thought maybe it would put less strain on the servers.
-
Thanks for the reply to my questions muldjord!
-
@muldjord thanks a bunch for doing that! :D
-
i noticed that it does not scrape scummvm games correctly, perhaps course of the wierd filenames?
the actual name of the game is actually listet inside the .svm files if that helps any.
i also noticed the alias.csv (or whatever its called) file to change the output name, but is there a way to tell skyscraper what name it should search for instead of the filename? -
- It already looks up the name in the scummvm.ini file. I tested this on my own setup only a couple weeks ago. What is the location of your scummvm.ini file?
- Yes, check the
--query
option here :)
-
@muldjord for the (recent)
lr-scummvm
core things work a bit different:- Games do not need to be added to the
scummvm.ini
file, which is located in$biosdir/scummvm
. Users can add them manually but for "plug-and-play" operation it is not necessary so games might not appear there. - The gameid (that is in
scummvm.ini
file) is instead always in the.svm
files inside the game folders. For example you can haveatlantis.svm
with the stringatlantis
inside. That is where you should be looking for the game ids.
So I would scrape the svm files indeed by reading their content for the game ids. See more information on how the games are setup in this topic. I plan to write official documentation about the core as soon as I have more time. Cheers!
- Games do not need to be added to the
-
if thats the case, i think it fails sometimes, for me it only writes correct names when i write them in aliasMap.csv.
i also think there is something funky going on with the brackets removal ... its a bit hit and miss for each console, but seems to work well with most nintendo and sega games or fba and mame with arcadeitalia only.is there a way to use aliasMap.csv for only one/each platform?
thanks, i think i missed that option ... :)
-
@Halvhjearne said in Versatile C++ game scraper: Skyscraper:
is there a way to use aliasMap.csv for only one/each platform?
No, it's global
-
@Halvhjearne said in Versatile C++ game scraper: Skyscraper:
i also think there is something funky going on with the brackets removal ... its a bit hit and miss for each console, but seems to work well with most nintendo and sega games or fba and mame with arcadeitalia only.
I'd need a better description than that to look into it. :D "Funky" can be pretty much anything. Are they missing where you expect them to be? Are they there where you expect them not to be. My guess is that it is because of how some sources return their titles and how other don't. So if that is the case, it's not a Skyscraper issues but rather what the sources return. I can't change that.
The one important thing I'd like to know is if brackets show up in some cases where you have them disabled. If that's the case, then it's a Skyscraper issue and I'd like to know about away to reproduce it so I can fix it.
-
i only noticed times where i wanted the brackets and didnt get them, as an example i have in aliasmap.csv "loom-fm;Loom (FM-TOWNS/English)", but it still insists on removing the brackets no mater if its set to true or false in config.ini.
if i scrape fba with arcadeitalia i get brackets, if set to true.
scraping fba with screenscraper will have no brackets, but i guess thats cause of what is returned to skyscraper? -
@Halvhjearne said in Versatile C++ game scraper: Skyscraper:
if i scrape fba with arcadeitalia i get brackets, if set to true.
scraping fba with screenscraper will have no brackets, but i guess thats cause of what is returned to skyscraper?Yes, exactly. arcadedb has a lot of brackets in their titles. screenscraper doesn't.
I'll consider the rest, I can see the issue with aliasMap ignoring the brackets. It's working as intended, but it's confusing, I can see that.
-
i still dont understand how there are so little matches for scummvm games?
it seems to me that it is searching for the filename, which will obviusly not give many matches cause of the wierd filenames and if it does its most likely the wrong game.i tried a few times with the query option, but i cant seem to get a hit with that either.
i think i managed to scrape 3 of 60 and atleast 1 is the wrong game.
-
@Halvhjearne No, it scrapes scummvm games by looking them up in scummvm.ini. But if that isn't used by the lr-scummvm core, then that's why that stopped working for you. I will of course look into updating that implementation once it solidifies.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.