Versatile C++ game scraper: Skyscraper

hi_im_daft

If it uses an amalgam of information from different databases though, how does it know what info is "correct"? Or, at least, more accurate to use? That's the problem I fear I'm going to wind up with.

muldjord

@hi_im_daft You can prioritize sources for each type of information (check the documentation). But you will undoubtedly end up with some faulty entries, that can't be avoided with an automated scraper like this. If you only want correct results, just only ever scrape with 'screenscraper' and 'arcadedb'. Those are based on mame names and file checksums. But then you'll give up the option of scraping platforms that have scripts to start the games. Such as svm for scummvm and cue files for psx.

Bottom line, you can't avoid faulty results with sources that are based on file name searches.

muldjord

In release related news: I think I tracked down the segfault that made Skyscraper crash sporadically. And I might even have fixed the rare problem some have been reporting where image and or resources would have been mixed up.

So, once again, I will be doing testing over the next days. And if I don't run into more trouble, I can start documenting the new compositor and, finally, release 2.3.0... Phew. I have literally been coding on this for 6-10 hours a day for the past week. I hope you guys appreciate this! It's starting to take it's toll here :D

hi_im_daft

@muldjord: I'm sure I can say for many of us here that we definitely appreciate the effort. The system itself is pretty great.

One last silly question, is it possible to set a different UI theme to use different Metadata? Some themes seems to look a lot nicer with the layout you use (screenshot with boxart in the corner), while others have a smaller area for images and squishing yours looks worse.

For example, I think the Skyscraper images look great with Retrorama, but for the NES Mini theme the boxart by itself would look better.

I don't think there's a way to do this one, but I am curious.

AnalogHero

@bleuge Ok if youre using simplemode the script is correct. Sometimes roms doesnt get scraped because the match wasnt close enough. What im doing then is lowering the -m value. For example Skyscraper -p snes -m 50. Answer Y to overwrite and Y to skip existing entries . You can lower the value if you like in a second run.

@hi_im_daft If the information online isnt correct, your scraped data will be so. If you have like 700 snes games for example, i would never know if a few games have incorrect descriptions.

AnalogHero

@hi_im_daft No, you have to decide. Emulationstation always uses the same gamelist.xml, no matter what theme youre using. This would ve a nice addition to ES imho. You should ask the dev of ES!

muldjord

@hi_im_daft If you want to change the look of the artwork exported by Skyscraper you can do that by editing "~/.skyscraper/artwork.xml". This is also well documented, so please check it out. Then, when you're done editing that file, rescrape with '-s localdb' and it will create new artwork from the recipe in artwork.xml. As AnalogHero notes, you can't have different screenshot artwork per ES theme, but if the theme uses the cover/boxart artwork specifically in one theme, and the screenshot in another, then those two can be exported from Skyscraper, and it will use one with one theme, and the other with the other. It's all up to how the ES theme is composed, and not something Skyscraper has control over.

So bottom line, if you have one ES theme only using cover, and another theme only usinge screenshot, then edit artwork.xml so it exports both to your liking. Then it will work as you expect.

With that said, please wait a few days. The new 2.3.0 release has A LOT of eye candy compared to the current release of Skyscraper. It can even render 3D boxart just by adding a <gamebox/> node to an artwork.xml <layer> node. And it will be well documented and easy to use.

hi_im_daft

I'll definitely be looking forward to the new release. Thanks for all the help guys. :)

bleuge

Is it possible to add Bandai Wonderswan and Color platforms?
I can't see it in the platforms source code.
Thanks!

muldjord

@bleuge Sure, it'll be in 2.3.0 when it's done (which is really soon now!)

AnalogHero

@muldjord This is awesome, cant wait. Did you had the time to look into the number of players issue? Its not that important really, just add it to your todo list :)

muldjord

@analoghero I'm sorry, I forgot what that was... Can you refresh my memory?

Aaaaaaaaaaadn, release is imminent now. I've created a quick video showing off the new compositor features and I've completely updated the artwork documentation. Pretty much all I need to do now is go through the last couple of sanity checks and 2.3.0 will be released. Which means it'll be out within an hour probably. :)

bleuge

@muldjord profanitywordhere man, you work pretty fast. Thanks for doing the best scraper around.

muldjord

This post is deleted!

AnalogHero

@muldjord It was the thing that most Emulationstation themes can only handle integer numbers in the number of players field. Some scrapingsources return things like 1 - 2 simultaneous which will display garbage in the theme.

muldjord

Skyscraper 2.3.0 released: https://github.com/muldjord/skyscraper

The ARTWORK release. Check artwork documentation here

MAJOR: Completely rewrote the artwork compositing engine
- Now supports nested layers which anchors to the parent layer for easy placement
- Implemented 'balance' effect that adjusts the colors of the parent layer
- Implemented 'blur' effect that blurs the parent layer
- Implemented 'brightness' effect that adjusts the brightness of the parent layer
- Implemented 'contrast' effect that adjusts the contrast of the parent layer
- Implemented 'frame' effect that allows you to add a graphical frame to the parent layer
- Implemented 'gamebox' effect that turns the parent layer into a nice looking 3D game box
- Implemented 'mask' effect that allows you to mask out certain parts of the parent layer
- Implemented 'opacity' effect that adjusts the opacity of the parent layer
- Implemented 'rounded' effect that rounds the corners of the parent layer
- Implemented 'stroke' effect that outlines the parent layer
- Improved 'shadow' effect to adhere perfectly to softness as radius
Added '-a' command line option for setting custom artwork xml config file
Added 'artworkXml' config file options for setting custom artwork xml config file
Implemented resource system that allows user to place files in '[homedir]/.skyscraper/resources' and use them in the '[homedir]/.skyscraper/artwork.xml' layers and effects
Added 'From cache' boolean to output plus note about '--updatedb'
'simple mode' now also accepts "Y" as a yes answer instead of just "y"
Now also looks for 'jp' region if no english region media is found for 'screenscraper' module
Now always accepts 'screenscraper' results no matter if platform matches or not
Now sets 'minMatch' to 0 by default for 'localdb' , 'arcadedb' and 'screenscraper' scraping modules. Can be overruled on command line and in config file
Made localdb more thread safe, might've fixed rare issues of resources being mixed up internally
Now works with filenames provided on command line even if they don't include full path
Added resource sources to output
Added 'wonderswan' and 'wonderswancolor' platforms

This is by far the biggest Skyscraper release to date. It's been a long ride but worth it! The biggest news is of course the new compositor!!! Just look at what you can do now!!!

A bunch of fixes and (quite cool) features also made it in. Check the notes for the details, it's too much to go through here. Please enjoy this release, and if you create some awesome artwork.xml files, please share them. The examples I've made myself are mostly to demonstrate certain effects.

Happy scraping guys!

AnalogHero

@muldjord Nice video, the new functions are very good explained. Would be nice if i could preview my changes to artwork.xml like you do.

muldjord

Also, guys if you would be so kind to share the news wherever you see fit (forums, social media), I would appreciate it a lot!

hi_im_daft

"Wait a few days", he says, then releases it about 12 hours later. /:-p What a liar.

EDIT: Also, what an above poster was saying about the "number of players" data. I get the same issue sometimes. Some databases just format it as a "1 - 2" or "1 - 4" players instead of just "2" or "4" players. Been trying to scrape around that, but it is odd/annoying. I'd imagine (like they do) that there's a way to trim the "1 -" from it. Like, if the field is more than 1 character, trim to the last character. Would be troublesome for any games that have more than 9 players though, but I don't know of any games people would be running on a retropie that would fit that criteria.

SammyBoy

@muldjord Thank you SO MUCH for huge effort and update to Skyscraper and I'll definitely make an follow-up to my post on Reddit. But can you tell us how to view artwork.xml previews like you do on your videos please?