Versatile C++ game scraper: Skyscraper

LineOf7s

@muldjord said in Versatile C++ game scraper: Skyscraper:

@LineOf7s Best approach is simply to run the following commands:
$ Skyscraper -p amiga -s openretro --refresh
$ Skyscraper -p amiga
The openretro module has been improved a bit, so you might have luck with that. With that said, I would strongly recommend using the RetroPlay lha files with Amiberry. It turns the Amiga into almost-a-console in that it uses the WHDLoad files. And Skyscraper scrapes these really well with OpenRetro since I've worked behind-the-scenes with some really cool guys to make use of OpenRetro's uuid's for these games.

After many attempts and much frustration (mostly, I think, due to an old version of Skyscraper hanging around in the background), I managed to get this happening as hoped. Hooray! (Also, I think I already have those LHA files you recommended, so all good there)

A curiosity, however: I happened to notice that at least one game - The Sentinel - had been found and correctly identified on OpenRetro.org, but Skyscraper was reporting that it had no covershot and no screenshots (ie Cover: NO Screenshot: NO) despite the fact it does (checked on the website). From what you understand of Skyscraper and the work you've done with the OpenRetro.org people, where do you think the fault may lie? Short of importing my own pictures, is there anything I can do as an end-user to get it to work? Again, it finds it and identifies it just fine (with a 66% search match) but doesn't think there are any assets to download.

bleuge

@muldjord thanks for impressive support! :D

Silent

I have somewhat of a request/code cleanup suggestion:

Platform::getDefaultScraper is overblown a lot, because looks like as of now all systems have "cache" as a default scraper. This just kind of bit me when migrating Moonlight support to 3.x.x (I initially overlooked "localdb" -> "cache" change here).

I personally think replacing this function with this would make it a lot better and less error prone:

// If user provides no scraping source with '-s' this sets the default for the platform
QString Platform::getDefaultScraper(const QString &platform)
{
  QString scraper = "cache";

  // If any platforms do NOT use cache as default, set them here

  return scraper;
}

Not to mention it'd syntactically be more correct, as default scraper is supposed to be cache, unless stated otherwise.

muldjord

@Silent I am well aware of these things, no need to report them to me. As I said earlier I am not interested in code contributions and I have plenty of code cleanup I want to do. I have closed the PR, I'd much rather code it myself when the Moonlight implementation is ready. Thanks.

muldjord

@LineOf7s That's a bit odd. I just tried scraping my own Sentinel_v1.01_3025.lha and it finds both cover and screenshot for it. What is your file called for it?

LineOf7s

@muldjord Sentinel_v1.0.lha

I don't mean to keep bringing it up, but the same file works fine (as far as scraping goes) in FS-UAE Arcade (which also scrapes from OpenRetro.org).

muldjord

@LineOf7s said in Versatile C++ game scraper: Skyscraper:

Sentinel_v1.0.lha

That's ok. It's the only way I can get it fixed if there's an error. So it seems like this particular file name doesn't have an uuid for searching, so it does an ordinary file name search for it. It finds 2 results where one has pretty much only the title, and the other has all data on it. And Skyscraper chooses the wrong one. I'll need to look into this.

LineOf7s

@muldjord I haven't looked yet into exactly how it works (perhaps I should before I ask), but is this something the use of --query might be able to deal with?

muldjord

@LineOf7s It's because I also include the "unpublished=1" entries when I search. FS-UAE probably doesn't. I include them since some of them are useful for demos and other unreleased / unfinished stuff (I had requests for including them). But I've just fixed this by implementing a fix that will distinguish it differently. I also found aminor bug in the whdLoadMap lookup, so this was pretty relevant. :) Anyways, it will be in 3.0.1 which I guess I could just release now.

LineOf7s

Hooray! Something tangible you could tweak! :) I love it when that happens. Thanks for your efforts.

muldjord

@LineOf7s

EDIT: Remember to use the --refresh or --cache refresh option when testing it. Otherwise it'll use the old cached "no screenshot" entry.

3.0.1 is out now. Please retest with the same file. It should work as expected now. Just keep in mind that this can happen for others. It all depends on whether they have two entries with the same name where one is unpublished. But as long as the unpublished one has a bracket note behind it, it should work for you (since it will choose the one without brackets which should be the complete result). Which is the case for the Sentinel release.

LineOf7s

@muldjord 3.0.1 worked a treat. There may or may not have been others affected too, but I can confirm that The Sentinel now has cover art and a screenshot. Huzzah!

robertvb83

@muldjord @mitu i have some custom Folders in my Rom structure but they are all using scrapable Rom categories…

for example i have a dedicated "adult" Folder with some of the fba adult Content. so in respect of scraping this is simply arcade. however i cannot scrape them because recognization is done via Rom Folder Name i guess… is there a way to tell Skyscraper that i want to scrape "adult" as "arcade"?

other examples are "satview" for seperated satella view roms. this would simply be scraped as "snes"

i am using the gui for scraping and i am on retropie 4

mitu

@robertvb83 said in Versatile C++ game scraper: Skyscraper:

is there a way to tell Skyscraper that i want to scrape "adult" as "arcade"?

I don't think so, right now. You can work-around by using a well-known folder/platform name (mame-libretro, mame-mame4all), but as a general feature, I don't think it works right now.
Maybe something can be added - similar with the existig aliasMap, but for platforms - i.e. nes-hacks, nes-homebrew for NES, cps2, cps3, cave for arcade, etc.

muldjord

This post is deleted!

muldjord

@robertvb83 Just add it to the fba folder instead. So it's inside of roms/fba/<FOLDER>

robertvb83

@muldjord thanks for the hint... but this will not lead to a seperate gamelist and also no separate media folders right? So i have to copy the total gamelist and change paths manually?

muldjord

@robertvb83 Correct. It will create one game list, but the subfolders can be entered and will show the game inside of them. You'll figure it out. Try playing around with it and see what you can come up with. I think that's easier than me trying to explain all the ways you could go about doing this.

muldjord

I've been working behind-the-scenes with the good people of igdb. This has resulted in Skyscraper becoming part of the partner program which ups the request limit for the module to 50000 per month. And they also let me know that they are working on improving the artwork situation for their database during 2019. This basically means that we at some point will be able to distinguish platforms for their screenshots and other artwork, which it can't at the moment (which is why all artwork is disabled for the module). So this is some pretty awesome news on all accounts.

Silent

Great! With 50k requests per month, it should probably be fairly safe to expose IGDB in Retropie-Setup script?

No biggie if not, just making an observation.