Versatile C++ game scraper: Skyscraper

muldjord

@frezeen said in Versatile C++ game scraper: Skyscraper:

continue not work, anyways new 3.1.5 is not 3.1.5 but when launch it:
Running Skyscraper v3.1.1 by Lars Muldjord
.....
.....
i dont know if missmatch is normal or not.

That's because I forgot to "make clean" before recompiling it. It is 3.1.5 even if it doesn't say so.
I might be able to try it on a different Windows machine at work tomorrow. It might be a dll problem.

EDIT: Just updated the zip again. It will now read "3.1.5" as it's supposed to. But it's the same executable other than that.

Did you remember to follow the installation instructions in the README included in the zip file? There's some important stuff in there.

frezeen

readme warning are to copy 2 dirs in /user/myuser/ and dont launch it from powershell but from old cmd.
i try it with cmd (admin privile and not) and with cmder.exe (look great, its linux terminal emulator with colours too)
arcadedb module work with no problems and its really fast.
same romset (2850 mame roms) from arcadedb: 13 mins on windows. 28 min in virtual linux machine, about 40 mins on rpi. with 1000mbits connection
its normal arcadedb module work but not screenscraper?

anyways, really, im scraping my 16000 roms collection with virtual machine... i dont want to stolen more time from you with this windows questions. sorry

muldjord

@frezeen No idea then. Works fine with screenscraper when running it in cmd.exe here.

frezeen

ok, i found the best solution for me.
https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/
than install debian from windows 10 market.
debian linux bash shell work like a charm, its not emulated , its very fast with all my windows drive mounted already. i install with apt-get all request packages than compile skyscraper and wow... all work perfect , fast and great.... you are my hero :) im very happy. with this solution windows user dont need a windows version :)

edit: im surprised about this new windows 10 function. this shell work like a real debian machine... ill try to install retropie too... want to see if i can control skyscraper from retropie script.

mo418

Hi,

I installed Retropie on Lubuntu (latest version and up to date).

When I try to scrape, it says that it cannot create cache folders because of permission.

I’m not highly skilled in Linux. I’ve read about the chmod stuff but I would like to make sure I don’t mess permission for the whole computer haha

May I ask for some help to allow skyscraper to work. I have no issue on my RPi 3b+ so I’m sure it’s folder permission issues on my Linux machine, as the attached picture shows.!
B1D13B05-5773-4D8F-B77A-2FA800297EA4.jpeg

mitu

@mo418 Can you post your config.ini file, where you've put your user/pass for ScreenScraper ?

mo418

@mitu

Yes I will later today, but as you can see in the picture it says that my user is found and that I’m allowed 1 thread.

Edit : Hum, where’s the config.ini file on a Lubuntu install? Can’t seem to find it.

Thanks

mitu

@mo418 said in Versatile C++ game scraper: Skyscraper:

Edit : Hum, where’s the config.ini file on a Lubuntu install? Can’t seem to find it.

The same place where is on every install - $HOME/.skyscraper. How did you add your user/pass to the config.ini if you didn't edit it ?

mo418

@mitu
Using the skyscraper advanced option, from the Retropie setup script in terminal. No edit in Lubuntu directly.

Edit: Hidden folders were not showing. Now I see everything

mitu

@mo418 Run this command from a command line terminal:

sudo chown -R js $HOME/.skyscraper/*

then re-try the scraping, I suspect there's a permissions problem.

mo418

@mitu

Yup, success for creating folders! Scraping at the moment.

Thanks a lot.

victordg

First I'd like to thank you @muldjord for this amazing tool, I love the way collections look tidy and neat with the same artwork styling, and how customizable is everything. I take it must be hard to deal with improvements and bugs, so my sincere admiration to you.

I'm having problems scraping certain SNES roms from screenscraper.fr because they have added those as a different system (Nintendo Power). Same occurs with Satellaview for example, if one decides to have their collection as normal SNES games they won't be scraped. I've tried adding the --query "md5=..." or "sha1=..." but it doesn't work for those (NP or Satellaview), although it was useful for other difficult scrapes.
After reading the documentation I don't see a way to scrape those roms from Skyscraper (using the latest as of now, 3.1.4). Any workarounds or hacks to have those parsed?

Thanks very much in advance!

muldjord

@victordg As it turns out, there are actually quite a few of these aliases. I've added the following:

    aliases.append("nintendo power");
    aliases.append("satellaview");
    aliases.append("sufami turbo");
    aliases.append("snes - super mario world hacks");
    aliases.append("super nintendo msu-1");

which will be available in 3.1.5 when it's released (no eta currently).

If you want to add it in your current version you can just edit the platforms.cpp source code file and add the above lines to the aliases function. Just remember to compile it and install it afterwards, otherwise it'll just keep using your current version.

EDIT: Actually I've decided to remove the platform check altogether when scraping from ScreenScraper. There's no real need for it as ScreenScraper always returns correct results since it checksum based.

quicksilver

@muldjord how will removing the platform check affect games that are found through name match and have multiple platform entries? Games launched by .sh or .conf files for example.

victordg

@muldjord Nice to know! Both ideas are perfect to me, actually I was also kind of expecting that the platform would be ignored when expliciting the rom hash with --query, so I totally agree to your approach, at least using screenscraper.fr there is no real need to match the platform since hashes are included for every game :) I'm too impatient though so I'll recompile with those aliases to see the NPs and Satellaviews right now :D

Thumbs up for you @muldjord, Skyscraper is getting better and better.

EDIT: Just to confirm that now it works as expected. Thanks again @muldjord!

muldjord

@quicksilver said in Versatile C++ game scraper: Skyscraper:

@muldjord how will removing the platform check affect games that are found through name match and have multiple platform entries? Games launched by .sh or .conf files for example.

There is a potential chance that a search by romnom (file name) will return an entry for a different platform. And if this entry is "faulty" it will no longer be filtered out by the platform check. But I don't think it will anymore as I am complying exactly with the demands ScreenScraper have for their API calls these days. And that requires a parent system id to be part of the query. This is always prepended to the query, even when using --query. So any false positives should be avoided that way. Even with the platform check disabled.

muldjord

Skyscraper 3.1.5 coming up in a few days. I had some leftover changes that are actually pretty cool, some minor additions as requested by users and some platform additions. And I've also added a --cache report:missing=<RESOURCE> option.

I don't have much time for Skyscraper these days, so I want this stuff out there. I just need to do some testing over the coming days to make sure everything is still a-okay.

rkoster

@muldjord Thanks for adding those two I submitted the ticket about. :)

muldjord

@rkoster said in Versatile C++ game scraper: Skyscraper:

@muldjord Thanks for adding those two I submitted the ticket about. :)

You're welcome :)

muldjord

Skyscraper 3.2.0 released: https://github.com/muldjord/skyscraper

Improved output of edit mode to show queue size and initial prioritized resources
Now clearly shows missing resources when adding resources in editing mode
Made '--cache vacuum' more verbose at verbosity level 2 or higher
Removed platform check for screenscraper module as it wasn't really necessary
Added 'videoSizeLimit' config variable (Thank you to 'silverfr0st' for suggesting this)
Now also processes brackets for skipped games (Thanks to 'hexive' for pointing this out)
Added 'arcadia' and 'astrocde' MESS platforms (Thank you to 'raphkoster' for suggesting these)
Added '--cache report:missing=<GROUP or RESOURCE1,2,...>' option (Thank you to JaymesRS for suggesting this)
Added '--fromfile' option allowing you to work on a subset of files taken from a file. Compatible with files created using the '--cache report' option
MobyGames module now includes 'platform' id query when relevant. Should improve result accuracy
Added region support for MobyGames cover artwork
Added A LOT of regions to region autodetection in general
Now no longer creates media folders when in resource gathering mode
Moved help texts for '--cache' options to '--cache help'

Lots of interesting stuff in this release. The --cache edit function has been improved a bit to better show what resources are missing on each rom. I've also added a nifty --fromfile <FILENAME> option which allows you to add filenames, one per line, into a file and use it with this option. Skyscraper will then work on those files only. This brings me to the new --cache report:missing option which allows you to create file reports containing filenames of all files missing certain resources. When those are created you can then combine the edit option with the fromfile option and add the missing resources like so --cache edit --fromfile /home/pi/.skyscraper/reports/reportfile.txt.

I've also improved region autodetection quite a bit. It will now detect pretty much all relevant regions and / or countries from the rom filename. Also added region support to mobygames covers!

As for minor stuff, some new platforms were added and some requested features aswell. Check the list above for all of it.

Let me know if you run into problems.

Have fun!