Versatile C++ game scraper: Skyscraper

AnalogHero

@muldjord Could it be that version 2.2.5 is a lot slower than previous versions? Just added a few whdload games and rescraped my amiga folder. I thought it crashed because the first output took sol long. Scraping with localdb takes the same amount of time. Over 10 minutes for 61 .uae files...

muldjord

@analoghero It is slower yes, which is caused by the new compositor rendering the shadow. I've optimized it a bit, but it could probably be optimized even further in future releases. I've basically implemented my own custom shadow renderer based on gauss kernel, which takes a bit of time to render... If you want to speed it up, try disabling the drop shadow in artwork.xml (chech the documentation for that).

The first results sometimes (not always) take a long time, that's normal (also for pre-2.2.5 versions), there's nothing I can do to change that since it is caused by the Qt frameworks network manager.

EDIT: 10 minutes for 61 uae files means something is wrong on your end. Can't tell you what exactly. It's a lot faster on my Pi 3 (not overclocked).

AnalogHero

@muldjord Cant say whats wrong... just tested it with psx as platform (18 games here). Took 4:06 :( This was faster in the past for sure.

(ES is closed, nothing is overclocked or running too hot)

Shame that i dont have a older version, would test it.

Edit: When --pretend ist set, its fast. When it dont find any matches (using wrong scraper) its also fast.

AnalogHero

@muldjord Ok tested it. Compiled old version 2.2.1 and scraped my psx system with 18 games in it. Result: Took 12 seconds

---- Scraping run completed! YAY! ----
Writing 366 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/psx/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:12

Average search match: 93%
Average entry completeness: 83%

Total number of games: 18
Successfully scraped games: 17
Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')

With 2.2.5:

---- Scraping run completed! YAY! ----
Writing 366 resources to local database, please wait... Success!
Now writing '/home/pi/RetroPie/roms/psx/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:03:58

Average search match: 93%
Average entry completeness: 83%

Total number of games: 18
Successfully scraped games: 17
Skipped games: 1 (Filenames saved to 'skipped-localdb.txt')

muldjord

I will do some testing on this when I get the time. In the meantime, please post your artwork.xml file.

muldjord

Just scraped 11 files on scummvm platform. Took 18 seconds on 2.2.6 with localdb on my Pi3 model B with RetroPie 4.3.
Scraped 21 files on psx platform with 2.2.6. Took 30 seconds.

Here's my artwork.xml. Try using that:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640" height="400">
    <layer resource="cover" valign="bottom" y="-15" height="200">
      <shadow distance="10" softness="5" opacity="50"/>
    </layer>
    <layer resource="screenshot" align="right" width="520" height="390"/>
  </output>
</artwork>

EDIT: Just cleared out my localdb and rescraped which is the same as starting over. Still only takes 30 seconds for 21 psx games. I can't reproduce your problem, sorry.

AnalogHero

@muldjord Ok its much faster now. Here is my (slow) artwork.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- This is the default 'artwork.xml' file that exports a screenshot composited from the cover and screenshot artwork and the wheel file. Please check 'artwork.xml.example' for a more thorough example of the possibilites. Also be sure to check the full artwork documentation at https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md -->
<artwork>
  <output type="screenshot" width="640" height="400">
    <layer resource="cover" valign="bottom" y="-15" height="200">
      <shadow distance="10" softness="10" opacity="75"/>
    </layer>
    <layer resource="screenshot" align="right" valign="top" width="520" height="390"/>
  </output>
  <!-- delete the next line to disable 'wheel entirely -->
  <output type="wheel" width="350"/>
  <!-- delete the next line to disable 'cover' entirely -->
  <output type="cover" height="390"/>
  <!-- delete the next line to disable 'marquee' entirely -->
  <output type="marquee" width="350"/>
</artwork>

muldjord

@analoghero I'm pretty sure it's the "softness=10" that causes the huge slowdown, which is probably what I made the default value was for 2.2.5, sorry bout that.
Rendering a shadow using a gauss kernel is quite taxing on the cpu, and the higher the softness, the bigger the gauss kernel is, which results in a lot more work for the cpu. So that's the short explanation.

So, basically there isn't an issue, it's just because the shadow rendering is so taxing on the cpu and therefore it becomes slow.

EDIT: I will look into optimizing the shadow rendering in the future. If/when I do, it will be a bullet point on the release note.

AnalogHero

@muldjord Thanks for looking into it. I changed the xml a few times, so i wasnt sure if it was the default. No time to test atm, is there any difference visible between softness 5 and 10?

muldjord

Yes, it's the softness/blur of the shadow. But I personally prefer a softness of 5, so just keep it at that (I might put a max of 5 on it in future releases).

muldjord

I am currently looking into an optimized algorithm for the shadow renderer. If all goes well, it should provide the same result with a huuuuuge speed improvement. So "softness=10" would become usable.

EDIT: New algorithm is almost implemented and will be in 2.2.7. The speed improvement is insane! Thousands of times faster, literally.

muldjord

The new optimized gauss shadow rendering is now fully implemented and the results are in! For 21 roms, rendering 1 shadow with a softness of 10 for each:

Old unoptimized version 2.2.6: 3 minutes and 26 seconds
New optimized version 2.2.7(still in testing): 4 SECONDS!

Now, THAT's what I call a proper optimization! :D HAHA, I love this.

Keep in mind that this is for 1 shadow with a softness of 10. The more shadows you have, the more time is saved. And also, a softness of 10 is ridiculous, so most often you'd use a softness of 2-5, which is even faster!

2.2.7 will be released as soon as I've done some testing. :)

muldjord

Skyscraper 2.2.7 released: https://github.com/muldjord/skyscraper

Optimized shadow rendering A LOT! WROOOOOOOOOM!!!
'simple mode' 'Do you want to run the script' answer now defaults to Y
Fixed bug in 'simple mode' where answering 'y' to adding brackets would actually remove them

If you took note of the drastic slowdown in 2.2.6 when using shadows, this release will bring you to nirvana. Shadow rendering has been huuuugely optimized. Like, it went from minutes of rendering, to a few seconds. Please try it out and let me know what you think.

Happy scraping! :)

AnalogHero

@muldjord Tested it, and i have to say that its really fast now. Lol ... 6 seconds for my 18 psx games. Awesome work. Thanks for it.

Newellj79

Pardon my ignorance. Is there a way to upgrade to the newest version from my current version?

AnalogHero

@newellj79 Yes, just follow the instructions on github. It will install over the old version.

hi_im_daft

Dumb question -- after finishing the scrape, there were a few games that were skipped, and it said it was added to "skipped-localdb.txt". Where is that file placed? I tried checking through the Skyscraper directory and couldn't find it anywhere.

muldjord

@hi_im_daft All Skyscraper files are located in "/home/pi/.skyscraper"

SammyBoy

@muldjord I think that Skyscraper isn't properly scraping some platforms from screenscraper.fr because I was scraping roms from the famicom disk system and none of them had a marquee however on the website there was a marquee and no games get scraped on arcade, fba, ect but I think this is due to having multiple arcade categories. Do you think you can fix it? Because I find it really annoying having to scrape from import because it is quite time consuming. Thank you

muldjord

@sammyboy Just add '--region jp' to the command line. The famicom disk system was only released in japan, and I don't check for those by default.

EDIT: I'll add it as one of the regions it tries automatically by default to 2.3.0. But please use '--region jp' until then. Version 2.3.0 won't be released for another week or so.

EDIT2: You mention 'arcade' and 'fba' aswell. Please be more specific about what you are trying to do. Give me an example filename from the platform that isn't working, so I have a chance of reproducing it.