Versatile C++ game scraper: Skyscraper

bleuge

@muldjord profanitywordhere man, you work pretty fast. Thanks for doing the best scraper around.

muldjord

This post is deleted!

AnalogHero

@muldjord It was the thing that most Emulationstation themes can only handle integer numbers in the number of players field. Some scrapingsources return things like 1 - 2 simultaneous which will display garbage in the theme.

muldjord

Skyscraper 2.3.0 released: https://github.com/muldjord/skyscraper

The ARTWORK release. Check artwork documentation here

MAJOR: Completely rewrote the artwork compositing engine
- Now supports nested layers which anchors to the parent layer for easy placement
- Implemented 'balance' effect that adjusts the colors of the parent layer
- Implemented 'blur' effect that blurs the parent layer
- Implemented 'brightness' effect that adjusts the brightness of the parent layer
- Implemented 'contrast' effect that adjusts the contrast of the parent layer
- Implemented 'frame' effect that allows you to add a graphical frame to the parent layer
- Implemented 'gamebox' effect that turns the parent layer into a nice looking 3D game box
- Implemented 'mask' effect that allows you to mask out certain parts of the parent layer
- Implemented 'opacity' effect that adjusts the opacity of the parent layer
- Implemented 'rounded' effect that rounds the corners of the parent layer
- Implemented 'stroke' effect that outlines the parent layer
- Improved 'shadow' effect to adhere perfectly to softness as radius
Added '-a' command line option for setting custom artwork xml config file
Added 'artworkXml' config file options for setting custom artwork xml config file
Implemented resource system that allows user to place files in '[homedir]/.skyscraper/resources' and use them in the '[homedir]/.skyscraper/artwork.xml' layers and effects
Added 'From cache' boolean to output plus note about '--updatedb'
'simple mode' now also accepts "Y" as a yes answer instead of just "y"
Now also looks for 'jp' region if no english region media is found for 'screenscraper' module
Now always accepts 'screenscraper' results no matter if platform matches or not
Now sets 'minMatch' to 0 by default for 'localdb' , 'arcadedb' and 'screenscraper' scraping modules. Can be overruled on command line and in config file
Made localdb more thread safe, might've fixed rare issues of resources being mixed up internally
Now works with filenames provided on command line even if they don't include full path
Added resource sources to output
Added 'wonderswan' and 'wonderswancolor' platforms

This is by far the biggest Skyscraper release to date. It's been a long ride but worth it! The biggest news is of course the new compositor!!! Just look at what you can do now!!!

A bunch of fixes and (quite cool) features also made it in. Check the notes for the details, it's too much to go through here. Please enjoy this release, and if you create some awesome artwork.xml files, please share them. The examples I've made myself are mostly to demonstrate certain effects.

Happy scraping guys!

AnalogHero

@muldjord Nice video, the new functions are very good explained. Would be nice if i could preview my changes to artwork.xml like you do.

muldjord

Also, guys if you would be so kind to share the news wherever you see fit (forums, social media), I would appreciate it a lot!

hi_im_daft

"Wait a few days", he says, then releases it about 12 hours later. /:-p What a liar.

EDIT: Also, what an above poster was saying about the "number of players" data. I get the same issue sometimes. Some databases just format it as a "1 - 2" or "1 - 4" players instead of just "2" or "4" players. Been trying to scrape around that, but it is odd/annoying. I'd imagine (like they do) that there's a way to trim the "1 -" from it. Like, if the field is more than 1 character, trim to the last character. Would be troublesome for any games that have more than 9 players though, but I don't know of any games people would be running on a retropie that would fit that criteria.

SammyBoy

@muldjord Thank you SO MUCH for huge effort and update to Skyscraper and I'll definitely make an follow-up to my post on Reddit. But can you tell us how to view artwork.xml previews like you do on your videos please?

muldjord

@sammyboy It's a bit tricky to do. You have to compile it on a Linux machine that ISN'T the RetroPie. I would suggest Ubuntu since the installation procedure on there is exactly the same as on the pi. Then you need to create a rom folder with just 1 single rom inside. Then you need to scrape that rom with sources until you have all of the artwork types for it (start by trying screenscraper, it probably has all of them). Then you open a terminal and type in "watch -n2 Skyscraper -p [platform] -i /[full]/[rom]/[folder] -s localdb --unattend". Be sure to change [platform] and the rom folder. That command will run Skyscraper every 2 seconds, using the localdb without asking any questions, which will result in the exported media to be updated every 2 seconds from the '[homedir]/.skyscraper/artwork.xml' file. Now open the relevant output image in an image viewer that automatically reloads if the image changes. The default one in Ubuntu works.

Now all you need to do is open up an editor and edit '[homedir]/.skyscraper/artwork.xml' and the image will update with your changes every 2 seconds.

So yeah, not so simple unfortunately. It can be done on the pie itself too if you ssh connect to it from another machine through 2 different terminals. On one you run the above command. On the other you edit the artwork.xml. And then you open a file browser and browse to the export folder where the images reside. I DO NOT recommend doing this on the pi, as the command will save data to the SD card every 2 seconds while you're working on it. SD cards do not like that! You've been warned. :)

Concerning the "players" problem, I will fix this soon. I've made note of it and it should be easily doable to just convert any known format to simply be 1 digit. :)

AnalogHero

@muldjord Have to report a bug: When scraping with screenscraper module all games get compared as .hack link and get all the same results!

muldjord

@analoghero I don't understand, please elaborate. What does ".hack link" mean?

AnalogHero

@muldjord Dont know what this is. I wanted to update my dbs as there are no wheels scraped. The command im using was : Skyscraper -p megadrive -s screenscraper --updatedb

muldjord

@analoghero Could you post the output, please? I still don't understand what you mean. Just tested it here with screenscraper on megadrive platform and it works well.

AnalogHero

@muldjord

pi@retropie:~ $ Skyscraper -p megadrive -s screenscraper --updatedb
------------------------------------------
Running Skyscraper v2.3.0 by Lars Muldjord
------------------------------------------
Platform           : 'megadrive'
Scraper module     : 'screenscraper'
Input folder       : '/home/pi/RetroPie/roms/megadrive'
Game list folder   : '/home/pi/RetroPie/roms/megadrive'
Covers folder      : '/home/pi/RetroPie/roms/megadrive/media/covers'
Screenshots folder : '/home/pi/RetroPie/roms/megadrive/media/screenshots'
Wheels folder      : '/home/pi/RetroPie/roms/megadrive/media/wheels'
Marquees folder    : '/home/pi/RetroPie/roms/megadrive/media/marquees'
Local db folder    : 'dbs/megadrive'

Reading and parsing local database, please wait...
Successfully parsed 9285 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

'gamelist.xml' already exists, do you want to overwrite it (y/N)? y
Checking if 'gamelist.xml' is writable?... It is! :)
Do you wish to skip existing entries (y/N)? n

Starting scraping run on 628 files using 4 threads.
Sit back, relax and let me do the work! :)

#1/628 Pass 1 ---- Game 'Empire of Steel (Europe)' found! :) ----
From cache:     NO
Search match:   0 %
Game title:     '.hack-Link' ()
Compare title:  'Empire of Steel'
Platform:       'PSP' ()
Release Date:   ''
Developer:      'Bandai Namco' ()
Publisher:      'CyberConnect2' ()
Players:        '' ()
Tags:           'Role playing games' ()
Rating (0-1):   '' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          NO ()
Marquee:        NO ()

Description: ()
The first game in the .hack series for PSP (and the planned final game for the franchise), .hack//LINK logs player into a new version of its virtual landscape called The World R:X (the "R" stands for "Revision"). Set 10 years after the last .Hack, players take control of Tokio Kuryu, a second year junior-high student. Presented through manga-style visuals, the game's story promises to clear up the mysteries from past entries. Over 100 characters from past .hack games, anime, manga, and books will make an appearance.

Gameplay promises to retain the basics of past titles, with players facing off in battle against enemies as they explore dungeons. The difference here is that you move around in a party of two, with the CPU-controlling the other character. The game will include 33 such CPU-controlled characters. For the PSP game, the battle system has been changed to a more action-heavy combat system.

Elapsed time: 00:00:04
Estimated time: 00:48:06

#2/628 Pass 1 ---- Game 'Spirou (Europe) (En,Fr,De,Es)' found! :) ----
From cache:     NO
Search match:   40 %
Game title:     '.hack-Link' ()
Compare title:  'Spirou'
Platform:       'PSP' ()
Release Date:   ''
Developer:      'Bandai Namco' ()
Publisher:      'CyberConnect2' ()
Players:        '' ()
Tags:           'Role playing games' ()
Rating (0-1):   '' ()
Cover:          YES ()
Screenshot:     YES ()
Wheel:          NO ()
Marquee:        NO ()

Description: ()
The first game in the .hack series for PSP (and the planned final game for the franchise), .hack//LINK logs player into a new version of its virtual landscape called The World R:X (the "R" stands for "Revision"). Set 10 years after the last .Hack, players take control of Tokio Kuryu, a second year junior-high student. Presented through manga-style visuals, the game's story promises to clear up the mysteries from past entries. Over 100 characters from past .hack games, anime, manga, and books will make an appearance.

Gameplay promises to retain the basics of past titles, with players facing off in battle against enemies as they explore dungeons. The difference here is that you move around in a party of two, with the CPU-controlling the other character. The game will include 33 such CPU-controlled characters. For the PSP game, the battle system has been changed to a more action-heavy combat system.

Elapsed time: 00:00:05
Estimated time: 00:30:12

^C

muldjord

@analoghero Ok, that's a game called hack-LINK that gets connected to the wrong data... Would it at all be possible for you to compress your ~/.skyscraper/dbs/megadrive folder into a file and upload it somewhere where I can download it? I would be very interested in figuring out what is going on.

AnalogHero

@muldjord Im zipping it right now. Note that the same thing happens for other system too. Tried amiga, same result.

muldjord

@analoghero I think I have a lead now... I think it's related to how I handle platform detection with the screenscraper module in 2.3.0. What I don't quite understand is why it returns the "hack-LINK" game in the first place, that shouldn't happen at all!

I have to go to work now, your help is appreciated. I really want to squash this bug, so I'll be back later to ask more questions if that's ok. In the meantime, can I ask you to keep your dbs's as they are? I would hate for this problem to go away without me knowing what happened. :)

AnalogHero

@muldjord Uploaded it.

Megadrive.zip

hi_im_daft

Japanese-release only PSP title being returned for all scrapes? That's an odd one. :-|

muldjord

@analoghero Thank you so much, I've downloaded it. Could you also do me the favor of running the command "ls -l /home/pi/RetroPie/roms/megadrive >> /home/pi/megadrive_ls.txt" and send me the megadrive_ls.txt it creates? I need as much information as I can get to reproduce this.