Versatile C++ game scraper: Skyscraper

robertvb83

@muldjord said in Versatile C++ game scraper: Skyscraper:

@robertvb83 Just set the login, it will automatically set it to the max number of threads that are allowed using your credentials. No need to set the number of threads yourself.

You did set it like so I assume:
[screenscraper]
userCreds="USERID:PASSWD"

this is what i did…
config.ini.JPG

how can i cancel the gathering?

muldjord

@robertvb83 Try ctrl+c or ctrl+d

robertvb83

i entered this to my config.ini (with my userID and PW obviously)

[screenscraper]
userCreds="USERID:PASSWD"

still gathering is with 1 thread only, i checked the Homepage and it says 7 threads

muldjord

@robertvb83 Please paste your entire config.ini file (but leave out the id and password again of course). Something must be wrong with how you are doing this. Also please reverify that you've typed your userid and password correctly. I am absolutely sure this works, I use it almost everyday, so it has to be something in your config or your setup in general.

EDIT: I assume you remember to choose the "screenscraper" module as the gathering source, right?

robertvb83

oh my god i had a typo i wrote

"[screenscaper]" instead of
"[screenscraper]"

now i corrected that and it is working fine using 7 threads making it really fast.

sorry for that :-) but many thanks for your quick and Kind help. REALLY cool tool!

muldjord

@robertvb83 Glad you got it working :)

bleuge

I just wanted to note that GUI still uses --refresh , as i see your deprecation message in each run, i know it still works :D

@muldjord what do you think, i don't even know if this has already been suggested, about somekind of interactive mode, where only files with no info (or definable missing fields) are searched in each scrape module, and if nothing is found you can manually type a search string?

something like this

Skyscraper -p amiga --interactive --missing=Wheel,Cover

Files with no wheel or (maybe and?) cover are searched, if nothing found ask for text to search.

If you think this is not viable, how are you guys dealing with missing info?, for example is difficult to find info about homebrew software, or some japanese games with hard to type names... ?

Also, is anyone using the priorities.xml feature?, any interesting one to share?
I'll read the source to see if i see the default priorities.

muldjord

@bleuge said in Versatile C++ game scraper: Skyscraper:

I just wanted to note that GUI still uses --refresh , as i see your deprecation message in each run, i know it still works :D

That is on purpose. We need to support both versions at this time. I might even end up keeping --refresh after all. We'll see.

somekind of interactive mode, where only files with no info (or definable missing fields) are searched in each scrape module, and if nothing is found you can manually type a search string?

It pretty much already does this. Any game that has already been scraped will be loaded from cache. So effectively, if you rescrape a platform, it only contacts the sources for the new roms you have added, or the ones it didn't find last time.

Skyscraper -p amiga --interactive --missing=Wheel,Cover

There won't ever be the option to only scrape certain pieces of information for a rom. The source delivers all data in one go, so there's no catch in doing this. So just rescrape the entire rom by adding the file name on command line.

If you think this is not viable, how are you guys dealing with missing info?

We're not really. This is up to the databases (and thereby the users) to add the missing data to the sources. :)

LineOf7s

@muldjord said in Versatile C++ game scraper: Skyscraper:

@LineOf7s Best approach is simply to run the following commands:
$ Skyscraper -p amiga -s openretro --refresh
$ Skyscraper -p amiga
The openretro module has been improved a bit, so you might have luck with that. With that said, I would strongly recommend using the RetroPlay lha files with Amiberry. It turns the Amiga into almost-a-console in that it uses the WHDLoad files. And Skyscraper scrapes these really well with OpenRetro since I've worked behind-the-scenes with some really cool guys to make use of OpenRetro's uuid's for these games.

After many attempts and much frustration (mostly, I think, due to an old version of Skyscraper hanging around in the background), I managed to get this happening as hoped. Hooray! (Also, I think I already have those LHA files you recommended, so all good there)

A curiosity, however: I happened to notice that at least one game - The Sentinel - had been found and correctly identified on OpenRetro.org, but Skyscraper was reporting that it had no covershot and no screenshots (ie Cover: NO Screenshot: NO) despite the fact it does (checked on the website). From what you understand of Skyscraper and the work you've done with the OpenRetro.org people, where do you think the fault may lie? Short of importing my own pictures, is there anything I can do as an end-user to get it to work? Again, it finds it and identifies it just fine (with a 66% search match) but doesn't think there are any assets to download.

bleuge

@muldjord thanks for impressive support! :D

Silent

I have somewhat of a request/code cleanup suggestion:

Platform::getDefaultScraper is overblown a lot, because looks like as of now all systems have "cache" as a default scraper. This just kind of bit me when migrating Moonlight support to 3.x.x (I initially overlooked "localdb" -> "cache" change here).

I personally think replacing this function with this would make it a lot better and less error prone:

// If user provides no scraping source with '-s' this sets the default for the platform
QString Platform::getDefaultScraper(const QString &platform)
{
  QString scraper = "cache";

  // If any platforms do NOT use cache as default, set them here

  return scraper;
}

Not to mention it'd syntactically be more correct, as default scraper is supposed to be cache, unless stated otherwise.

muldjord

@Silent I am well aware of these things, no need to report them to me. As I said earlier I am not interested in code contributions and I have plenty of code cleanup I want to do. I have closed the PR, I'd much rather code it myself when the Moonlight implementation is ready. Thanks.

muldjord

@LineOf7s That's a bit odd. I just tried scraping my own Sentinel_v1.01_3025.lha and it finds both cover and screenshot for it. What is your file called for it?

LineOf7s

@muldjord Sentinel_v1.0.lha

I don't mean to keep bringing it up, but the same file works fine (as far as scraping goes) in FS-UAE Arcade (which also scrapes from OpenRetro.org).

muldjord

@LineOf7s said in Versatile C++ game scraper: Skyscraper:

Sentinel_v1.0.lha

That's ok. It's the only way I can get it fixed if there's an error. So it seems like this particular file name doesn't have an uuid for searching, so it does an ordinary file name search for it. It finds 2 results where one has pretty much only the title, and the other has all data on it. And Skyscraper chooses the wrong one. I'll need to look into this.

LineOf7s

@muldjord I haven't looked yet into exactly how it works (perhaps I should before I ask), but is this something the use of --query might be able to deal with?

muldjord

@LineOf7s It's because I also include the "unpublished=1" entries when I search. FS-UAE probably doesn't. I include them since some of them are useful for demos and other unreleased / unfinished stuff (I had requests for including them). But I've just fixed this by implementing a fix that will distinguish it differently. I also found aminor bug in the whdLoadMap lookup, so this was pretty relevant. :) Anyways, it will be in 3.0.1 which I guess I could just release now.

LineOf7s

Hooray! Something tangible you could tweak! :) I love it when that happens. Thanks for your efforts.

muldjord

@LineOf7s

EDIT: Remember to use the --refresh or --cache refresh option when testing it. Otherwise it'll use the old cached "no screenshot" entry.

3.0.1 is out now. Please retest with the same file. It should work as expected now. Just keep in mind that this can happen for others. It all depends on whether they have two entries with the same name where one is unpublished. But as long as the unpublished one has a bracket note behind it, it should work for you (since it will choose the one without brackets which should be the complete result). Which is the case for the Sentinel release.

LineOf7s

@muldjord 3.0.1 worked a treat. There may or may not have been others affected too, but I can confirm that The Sentinel now has cover art and a screenshot. Huzzah!