Versatile C++ game scraper: Skyscraper

bleuge

Users with experience scraping, do you have edited or used the priorities.xml file, general? per platform?
Any tips about best sources for each platform?

I am thinking to delete my localdb and rescrape everything, as i think i have some kind of mess after all the tests and learning about this in the last weeks.
Thanks

AnalogHero

@bleuge I didnt edit any of those except for amiga. Im also not sure about that. Im always using all sources.

muldjord

By popular request Skyscraper is now upgradeable. To get the updateable release go to https://github.com/muldjord/skyscraper and follow the installation instructions. After you've installed it you can update it using the new update_skyscraper.sh script. :)

bleuge

@muldjord I see sometimes this text in the output

'.known option 'pretend

Literal text. Is this right?

muldjord

@bleuge I don't understand. It looks like part of a message about the "pretend" option. Maybe a spelling error in your script somewhere? That's my best suggestion.

SammyBoy

@muldjord For the Skyscraper update script can I install it on top of the current script or do I need to delete everything again?

muldjord

@sammyboy Delete everything just to be sure. Then from the next release, you can just run "update_skyscraper.sh"

snock

When compiling it in the rpi3 it gives me error, in archlinux no.

muldjord

@snock I need a bit more information than that to help you. What error? Did you install Qt5 succesfully before installing as per the instruction on the github page? Are you using the official RetroPie 4.3 image?

snock

@muldjord yes . It is rare that running the script as the wiki says failed me when compiling. But doing qmake && make && sudo make install works. Thanks for your quick response.

muldjord

@snock I would very much like to know what problems you had specifically. If there's an error in the script I need to fix it. Did the script not work for you? What was the cause? Thank you.

muldjord

Skyscraper version 2.3.3 released: https://github.com/muldjord/skyscraper

Added 'hue' effect which changes the hue of the image
Added 'saturation' effect which changes the color saturation of the image
Added 'colorize' effect which renders the image using a single hue
Added 'rotate' effect that rotates the parent layer around the x, y or z axis
Added layer render 'mode' attribute ('overlay', 'colordodge', 'hardlight' and so on)
Added layer 'opacity' attribute (0-100)
Improved front and side textures for the 'gamebox' effect
Optimized search passes a bit
Re-added platform checking on 'screenscraper' and added all arcade platforms as aliases

This release is mostly a polishing release. I added a bunch more compositing effects and updated the gamebox textures to be a lot more cool looking.
If you have the "update_skyscraper.sh" script (most 2.3.2 users should have it) you can get this new release simply by running that. Otherwise go to the github page and follow the instructions on there.

muldjord

Skyscraper version 2.3.4 released: https://github.com/muldjord/skyscraper

'gamebox' effect no longer cuts top and bottom from source layer
'gamebox' effect now handles side artwork that is too tall
Optimized localdb mutexes
Fixed issue where searchName could be empty and cause Skyscraper to stall for 15 minutes
Now no longer runs screenscraper sha1 and md5 passes if filesize is 0

This is mostly a bugfix and polish release. Rare cases where Skyscraper would stall for 15 or more minutes should now be a thing of the past. I've also fixed it so screenscraper now no longer runs the sha1 and md5 passes if the filename in question is empty (such as the scummvm dummy files).

Happy scraping!

bleuge

Thanks for updates and for fixing the network stall i was getting!

muldjord

@bleuge You're welcome. I finally got a stall myself, so I could reproduce it, figure out what made it happen, and fix it! :)

muldjord

Heads up everyone. Skyscraper seems to currently be broken to some degree. I've been getting messages from people about artwork being mixed up for a while, but I never really was able to reproduce this myself unless it was for games that had close relation in their titles (which didn't surprise me as my scraper is very much filename based and duplicates would be expected from time to time).

But now, I am getting some really weird results. Screenshot from "Airborne Ranger" suddenly being saved in localdb for "Akira". This should NOT happen.

So while I investigate (this could take a loooooong time), I felt like I needed to inform you of this issue.

Also, I will be without internet from sunday and two weeks forward. So I won't be available for support or comments during that time.

muldjord

Quick update: I am hopeful I've FINALLY tracked down where the data gets mixed up between games. As I've mentioned earlier I was at a loss as to where this could happen since all of my game data structures are in-scope and get thrown out before moving on to the next game. So so far I thought the culprit was my local database, since I share it between threads. But, IT IS NOT!!! After much testing and COMPLETELY removing the local database from my loop, it still happened!

So what might be causing this? Well, there's a little thing called caching inside of Qt's own network request thingamabob. And as it turns out, that internal piece of data doesn't seem to always get cleared correctly before I make a new network request. And for some i-have-no-idea-how reason, it sometimes gets skewed and starts always returning the data from the previous game...

So am I 100% certain of this? No, I am not. But I just added code to clear the cache before each new network request, and after I compiled it I've been testing it 10 times (so far) and haven't been able to make it mix stuff up again. That's why I'm cautiously optimistic.

The longest I have been able to get a success streak before the cache clearing fix was 3 times in a row. So now being at 10 without flaws makes me hopeful. I'll keep testing for the remainder of the day.

If all goes well I'll release 2.3.5 with a fix before I go full caveman the next two weeks without internet.

bleuge

Thanks for being fast. I was thinking this could a very weird error that keep it broken for weeks... xD

Maybe this explain the strange broken pngs i was getting sometimes, if buffer didn't clear fully things could be mixed up.

Good work man!

muldjord

So I just got access to MobyGames' API aswell, that's pretty cool! They have pretty high restrictions on request usage (which is absolutely understandable), so when I include it it can mostly be used for command line scraping of just a few games that might not be found by the other modules. It won't be in 2.3.5 though, but just wanted to let you know of the good news.

2.3.5 is almost ready now btw. I've been testing it like crazy since yesterday and I can't get it to mix up game data anymore. So that's good!

muldjord

PLEASE UPDATE ASAP!!! Quick note: I urge you all to completely remove your "[homedir]/.skyscraper/dbs" folder as it contains faulty data. If you don't it will just reuse those data and it will appear as if the mixed entry bug hasn't been fixed at all!

Skyscraper version 2.3.5 released: https://github.com/muldjord/skyscraper

IMPORTANT: Fixed bug that caused resources to be mixed up between games because Qt's network cache wasn't cleared (Probably not a Qt bug, but a DAMN hard bug to spot either way). All previous Skyscraper releases have this bug, SO PLEASE UPDATE!!!
Made 'scummvm' parsing look for config in homedir aswell ('.scummvmrc')
Now always removes brackets from returned titles
Now always shows current scraping module in output
Optimized search matching even more
No longer asks user about skipping entries if filenames are provided on command line

This release should make mixed entries a thing of the past. I thought this bug was related to the local database cache (and I thought I already fixed it). But it was not! It was related to Qt's internal network requests caching its own data and sometimes returning the data of the previous request. After I included a full cache clear on it, I can no longer make it mix up entries. I've been testing it for well over 15 times on thousands of entries over the last two days. Before this fix I got the problem within 3 tries. I haven't been able to since.

I've also improved the search matching even more, which should now result in some positives that were filtered out before are now back in. Quite a lot actually. With that said, there will be the occasional false-positive still. It can't be prevented 100%. If you are very strict about this, please remember that you can set the "Minimum search percentage match" with the "-m" command line parameter.

If you experience problems with this release, please let me know. Also, I will be without internet for the next couple of weeks, so I won't be able to provide support or comment.

Please update asap! And happy scraping! :)