Versatile C++ game scraper: Skyscraper

hhromic

@Halvhjearne said in Versatile C++ game scraper: Skyscraper:

@muldjord

i still dont understand how there are so little matches for scummvm games?
it seems to me that it is searching for the filename, which will obviusly not give many matches cause of the wierd filenames and if it does its most likely the wrong game.

i tried a few times with the query option, but i cant seem to get a hit with that either.

i think i managed to scrape 3 of 60 and atleast 1 is the wrong game.

The integrated ES scraper, that used TheGamesDB worked pretty well with ScummVM games as long as the .svm file is named properly or in the case of lr-scummvm the folder itself is named properly. I understand Skyscraper also supports TheGamesDB, so maybe you should prefer that source for your ScummVM games?

Halvhjearne

@hhromic

yes, but unfortunately i already used up my requests for thegamesdb.
however i looked up the games manually on screenscraper and most of them are actually there, they are just not found correctly for some reason.

hhromic

@Halvhjearne can you confirm if you are using the standalone or the libretro version of scummvm?

If you are using the libretro version, I can recommend you to name the .svm files with the full name of the game too. For the libretro core it doesn't actually matter as long as the content of the .svm file remains with the proper gameid. For example You can have:

/home/RetroPie/roms/scummvm/Indiana Jones and The Fate of Atlantis.svm/   <- note the "extension" in the name of the folder
        ATLANTIS.001
        ATLANTIS.002
        Indiana Jones and The Fate of Atlantis.svm      <- with "atlantis" inside
           :

If skyscrapper not only looks on the scummvm.ini that should help it get the correct name for scraping from the .svm files. On the other hand if skyscraper only looks in the config file for the names, then it won't work I guess in the current version. @muldjord should advise better.

muldjord

@hhromic Skyscraper look up the filename in the scummvm.ini file and uses the long name in the corresponding section for lookups for modules that are search based. Screenscraper is not, so it won't work with that either way.

hhromic

@muldjord so just to confirm I understand:

For the scummvm platform, Skyscraper only tries to get the game title (long name) from the scummvm.ini file and always ignores the actual .svm file name?

muldjord

@hhromic For the search based scraping modules, yes. For screenscraper it uses the filename directly as that's what they support for scummvm games. Checksumming the svm files clearly wouldn't make much sense.

Halvhjearne

@muldjord

i just tried searching manually on screenscraper for some of the filenames and then after with skyscraper using the query option, but it dosnt seem to get any hits even tho there is only one game returned from the search on the website.

a god example would be atlantis, as the only game that shows up on a screenscraper search is " Indiana Jones and the Fate of Atlantis" but skyscraper returns nothing for this game.

another would be gob1, which only returns "Gobliiins" on screenscraper, but skyscraper returns nothing.

@hhromic

iirc the lr-scummvm is still in experimental, so i am probably using the regular version, but i will check it out.

muldjord

@Halvhjearne Some examples command lines would be helpful. I know it works, so I can't really help you without specific command that won't work for you. Then I can tell you what's going wrong.

Halvhjearne

@muldjord

/opt/retropie/supplementary/skyscraper/Skyscraper --unattend -g /home/pi/RetroPie/roms/scummvm -o /home/pi/RetroPie/roms/scummvm/media --relative -p scummvm -s screenscraper --refresh --query "atlantis" /home/pi/RetroPie/roms/scummvm/atlantis.svm

i also tried:

/opt/retropie/supplementary/skyscraper/Skyscraper --unattend -g /home/pi/RetroPie/roms/scummvm -o /home/pi/RetroPie/roms/scummvm/media --relative -p scummvm -s screenscraper --refresh --query "Indiana+Jones+Atlantis" /home/pi/RetroPie/roms/scummvm/atlantis.svm

or

/opt/retropie/supplementary/skyscraper/Skyscraper --unattend -g /home/pi/RetroPie/roms/scummvm -o /home/pi/RetroPie/roms/scummvm/media --relative -p scummvm -s screenscraper --refresh --query "gob1" /home/pi/RetroPie/roms/scummvm/gob1cd-us.svm

if i go to the screenscraper website and search for atlantis under scummvm games, the only one that comes up is the correct one (Indiana Jones And The Fate Of Atlantis) if i search for gob1, the only one that comes up is Gobliiins.

im pretty sure if you search for the shortname of any of the scummvm games on screenscraper, it will find the correct one every time.

muldjord

@Halvhjearne It's because your query isn't complete. As per the documentation, screenscraper is not search name based in the same way as some of the other modules. You'd instead need --query "romnom=atlantis.svm" if that's one of the files that's on screenscraper (I haven't checked). But it's in the documentation.

Halvhjearne

@muldjord

that worked ... but without the .svm part

edit:
for Gobliiins, but not for atlantis

its a bit hit and miss, but it does finds a lot of the games this way

edit:
if i add .scummvm to the shortname, i belive it will find all my games ... except when the svm filename differs from "the real shortname", as an example Gobliiins shortname is "gob1" but the svm file is for some reason called gob1cd-us (probably cause its a later cd release).

muldjord

This post is deleted!

muldjord

Just a heads up to all Skyscraper users. From 3.0.0 "localdb" will change name entirely to "cache". It's another of the big changes in 3.0.0, which means that a lot of config variables will change name and the old ones will be removed. I realize this might seem inconvenient for current users, but this really is a change that needed to happen as the "localdb" name was rather confusing when I often used it in conjunction with "cache" since it's the same thing.

@mitu Is the localdb name used in the script? I can't remember. Please change it to "cache" instead if it is.

EDIT: --purgedb vacuumwill become --cache vacuum & --purgedb all will become --cache purge:all. Also, the ~/.skyscraper/dbs folder will become ~/.skyscraper/cache. This will automatically be renamed for users when they run Skyscraper the first time after updating to 3.0.0.

Halvhjearne

@muldjord

i mananged to scrape all my games like this just using the shortnames, which in most cases is the same as the svm filename, except when its called something like kq6-cd or gob1cd-us.
in those cases i just used the real shortname from scummvm, alltho a game like the sci remake of larry i had to search for lsl1sci and for vga remake of quest for glory i searched for qfg1vga, which is ofcourse also the "real" shortnames for those games.

mitu

@muldjord I think there's one place in the help section referring to LocalDB. Does this mean also -s localdb is now -s cache ?

I'll have to fix the purge commands then, in the new version I used the purgedb commands instead of manually removing the cache data.

But this last change will create an incompatibility with the old installations - if the script is updated without updating Skycraper script or vice-versa (script updated, but not Skyscraper).

muldjord

@mitu Yes, it is now technically -s cache but just leave it out entirely (which is then compatible with both 2.x and 3.x) as it's not necessary. If -s cache is left out it will default to create a game list. So creating a game list should always just be Skyscraper -p <platform> without -s.

And yes, there will be incompatibility with the purge options, so we'd need to coordinate the release precisely somehow.

EDIT: I've added auto-migration from old config.ini format to new 3.0.0 format. I also auto-migrate the "dbs" folder to "cache" automatically. So you don't need to worry about this.

EDIT: --refresh is now --cache refresh btw, so that should be changed aswell when we "flip the switch". I will still support the --refresh option for a while, so we can just use that until we feel like most users updated to 3.0.0.

megaJekan

@mitu said in Versatile C++ game scraper: Skyscraper:

@megaJekan Can you give some examples ?

I tried to rename ROM filenames (to match the internal ScreenScraper name), that solved the most of the issues. Sometimes underscores are the source of problem.

For SNES I can report that I still have problems with scrapping these ROMs:

Power Lode Runner
https://www.screenscraper.fr/gameinfos.php?plateforme=110&gameid=64853
(probably because Nintendo Power is written as platform instead of Super Nintendo)

Super Tetris 2 + Bombliss
https://www.screenscraper.fr/gameinfos.php?plateforme=4&gameid=65158
(maybe '+' sign in the name is an issue)

Radical Dreamers
https://www.screenscraper.fr/gameinfos.php?plateforme=107&gameid=108733
(Satellaview is written as platform instead of SNES)

My ROMs have totally different md5, than in the ScreenScraper database. Then I have one more question. Can I add my own filenames and md5 checksums to screenscrapper.fr?

muldjord

@megaJekan You can use the --query option to force md5 sums for any rom: https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md#--query-string

Silent

@muldjord said in Versatile C++ game scraper: Skyscraper:

And yes, there will be incompatibility, so we'd need to coordinate the release precisely somehow.

Why not leave those as undocumented aliases for compatibility? Yes, it sucks, but sometimes you gotta do what you gotta do.

Either that or the RetroPie-Setup script should detect the exact Skyscraper version and adapt. Is this possible?

muldjord

@Silent I think you answered that yourself. Because it sucks (and I can't "ghost" them, they will be visible and cause confusion). ;) And I do migrate old config.ini and folders. And it's to be expected that there are changes for major versions. Users should know that. I do keep the --refreshfor a while though, so for those where it makes sense, I do actually keep them a couple months. And the change from localdb to cache is compatible with both versions, as it should be simply left out for both which gives the same result. I think, in the end, this will turn out to be a non-issue when it comes to the script as we can work around it.