Versatile C++ game scraper: Skyscraper

raspy+

Thanks for the reply to my questions muldjord!

Slow1

@muldjord thanks a bunch for doing that! :D

Halvhjearne

i noticed that it does not scrape scummvm games correctly, perhaps course of the wierd filenames?
the actual name of the game is actually listet inside the .svm files if that helps any.
i also noticed the alias.csv (or whatever its called) file to change the output name, but is there a way to tell skyscraper what name it should search for instead of the filename?

muldjord

@Halvhjearne

It already looks up the name in the scummvm.ini file. I tested this on my own setup only a couple weeks ago. What is the location of your scummvm.ini file?
Yes, check the --query option here :)

hhromic

@muldjord for the (recent) lr-scummvm core things work a bit different:

Games do not need to be added to the scummvm.ini file, which is located in $biosdir/scummvm. Users can add them manually but for "plug-and-play" operation it is not necessary so games might not appear there.
The gameid (that is in scummvm.ini file) is instead always in the .svm files inside the game folders. For example you can have atlantis.svm with the string atlantis inside. That is where you should be looking for the game ids.

So I would scrape the svm files indeed by reading their content for the game ids. See more information on how the games are setup in this topic. I plan to write official documentation about the core as soon as I have more time. Cheers!

Halvhjearne

@muldjord

if thats the case, i think it fails sometimes, for me it only writes correct names when i write them in aliasMap.csv.
i also think there is something funky going on with the brackets removal ... its a bit hit and miss for each console, but seems to work well with most nintendo and sega games or fba and mame with arcadeitalia only.

is there a way to use aliasMap.csv for only one/each platform?

thanks, i think i missed that option ... :)

muldjord

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

is there a way to use aliasMap.csv for only one/each platform?

No, it's global

muldjord

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

i also think there is something funky going on with the brackets removal ... its a bit hit and miss for each console, but seems to work well with most nintendo and sega games or fba and mame with arcadeitalia only.

I'd need a better description than that to look into it. :D "Funky" can be pretty much anything. Are they missing where you expect them to be? Are they there where you expect them not to be. My guess is that it is because of how some sources return their titles and how other don't. So if that is the case, it's not a Skyscraper issues but rather what the sources return. I can't change that.

The one important thing I'd like to know is if brackets show up in some cases where you have them disabled. If that's the case, then it's a Skyscraper issue and I'd like to know about away to reproduce it so I can fix it.

Halvhjearne

@muldjord

i only noticed times where i wanted the brackets and didnt get them, as an example i have in aliasmap.csv "loom-fm;Loom (FM-TOWNS/English)", but it still insists on removing the brackets no mater if its set to true or false in config.ini.

if i scrape fba with arcadeitalia i get brackets, if set to true.
scraping fba with screenscraper will have no brackets, but i guess thats cause of what is returned to skyscraper?

muldjord

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

if i scrape fba with arcadeitalia i get brackets, if set to true.
scraping fba with screenscraper will have no brackets, but i guess thats cause of what is returned to skyscraper?

Yes, exactly. arcadedb has a lot of brackets in their titles. screenscraper doesn't.

I'll consider the rest, I can see the issue with aliasMap ignoring the brackets. It's working as intended, but it's confusing, I can see that.

Halvhjearne

@muldjord

i still dont understand how there are so little matches for scummvm games?
it seems to me that it is searching for the filename, which will obviusly not give many matches cause of the wierd filenames and if it does its most likely the wrong game.

i tried a few times with the query option, but i cant seem to get a hit with that either.

i think i managed to scrape 3 of 60 and atleast 1 is the wrong game.

muldjord

@Halvhjearne No, it scrapes scummvm games by looking them up in scummvm.ini. But if that isn't used by the lr-scummvm core, then that's why that stopped working for you. I will of course look into updating that implementation once it solidifies.

hhromic

@muldjord said in Versatile C++ game scraper: Skyscraper:

I will of course look into updating that implementation once it solidifies.

You mean the lr-scummvm scriptmodule? I would say it is pretty much stable now. Let me know if you have questions, but I think there is not much to add to what I already explained some posts ago.

hhromic

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

@muldjord

i still dont understand how there are so little matches for scummvm games?
it seems to me that it is searching for the filename, which will obviusly not give many matches cause of the wierd filenames and if it does its most likely the wrong game.

i tried a few times with the query option, but i cant seem to get a hit with that either.

i think i managed to scrape 3 of 60 and atleast 1 is the wrong game.

The integrated ES scraper, that used TheGamesDB worked pretty well with ScummVM games as long as the .svm file is named properly or in the case of lr-scummvm the folder itself is named properly. I understand Skyscraper also supports TheGamesDB, so maybe you should prefer that source for your ScummVM games?

Halvhjearne

@hhromic

yes, but unfortunately i already used up my requests for thegamesdb.
however i looked up the games manually on screenscraper and most of them are actually there, they are just not found correctly for some reason.

hhromic

@Halvhjearne can you confirm if you are using the standalone or the libretro version of scummvm?

If you are using the libretro version, I can recommend you to name the .svm files with the full name of the game too. For the libretro core it doesn't actually matter as long as the content of the .svm file remains with the proper gameid. For example You can have:

/home/RetroPie/roms/scummvm/Indiana Jones and The Fate of Atlantis.svm/   <- note the "extension" in the name of the folder
        ATLANTIS.001
        ATLANTIS.002
        Indiana Jones and The Fate of Atlantis.svm      <- with "atlantis" inside
           :

If skyscrapper not only looks on the scummvm.ini that should help it get the correct name for scraping from the .svm files. On the other hand if skyscraper only looks in the config file for the names, then it won't work I guess in the current version. @muldjord should advise better.

muldjord

@hhromic Skyscraper look up the filename in the scummvm.ini file and uses the long name in the corresponding section for lookups for modules that are search based. Screenscraper is not, so it won't work with that either way.

hhromic

@muldjord so just to confirm I understand:

For the scummvm platform, Skyscraper only tries to get the game title (long name) from the scummvm.ini file and always ignores the actual .svm file name?

muldjord

@hhromic For the search based scraping modules, yes. For screenscraper it uses the filename directly as that's what they support for scummvm games. Checksumming the svm files clearly wouldn't make much sense.

Halvhjearne

@muldjord

i just tried searching manually on screenscraper for some of the filenames and then after with skyscraper using the query option, but it dosnt seem to get any hits even tho there is only one game returned from the search on the website.

a god example would be atlantis, as the only game that shows up on a screenscraper search is " Indiana Jones and the Fate of Atlantis" but skyscraper returns nothing for this game.

another would be gob1, which only returns "Gobliiins" on screenscraper, but skyscraper returns nothing.

@hhromic

iirc the lr-scummvm is still in experimental, so i am probably using the regular version, but i will check it out.