Versatile C++ game scraper: Skyscraper

muldjord

@circo Thank you for your suggestion, but it's more a question of development time. I would have to rewrite almost the entirety of the scraper system in Skyscraper for this to work as it is not designed for multigame-requests. It's not an overnight change I'm afraid. It's a substantial rewrite that would probably take me months to complete. I do not have that kind of time to work on this project.

circo

@muldjord Ah, I see.
I know from experience that it's difficult to work with APIs that don't fit with the rest :/ But Skyscraper is working really nicely, so at least the rest are cooperating!

chipsnblip

i finally got around to scraping my roms with Skyscraper. a few minor hiccups, but overall it's been really great so far. over the last few days, i noticed quite a few screenshots it was pulling in were not properly cropped, so i would like to share an example of how to autocrop them, for those who are interested.

here's a before & after example:

before

after

it's easy to batch-process a database's artwork after it's been scraped. this requires installing ImageMagick first (apt-get install imagemagick). for example, mogrify -trim /home/pi/.skyscraper/dbs/n64/screenshots/screenscraper/*.*, then it's just a matter of rescraping from the localdb.

i hope this is useful to someone.

muldjord

@chipsnblip Fantastic, thank you for the tip. And that gave me the idea to implement this automatically into my compositor. I can simply look for black borders, and remove them. I'll add that to my todo list. :)

Used2BeRX

@muldjord I didn't even know it was possible to script that. Any chance you could post a standalone script to remove black borders from images?

A few months back I re-shot all 4,236 Title and Action screenshots for NES and FDS to make sure they were all the same size and were taken while using the SONY CSX palette, but just in case I don't want to re-invent that particular wheel every time I tackle a new system down the road, that code of yours might just come in handy. :)

chipsnblip

i wasn't too impressed with the gameboy screenshots that were coming in, so i gave in and downloaded a pack of them from emumovies and imported to my skyscraper database. but something was still a bit off, the grayscale images didn't really fit my theme or look very appealing on my modest 13" CRT screen.

so i set out to make them appear more or less like the puke-green original gameboy using Skyscraper's built in compositor, a simple marquee overlay made in gimp, a snippet of xml, and a dash of bash. the result looks pretty ok, so thought i'd go ahead and share my findings with you all.

Before:
before01
before02

After:
after01
after02

depending on your priorities.xml settings for <screenshot>, you may need to fiddle with the brightness/contrast/etc in the artwork_gb.xml file. in this example i'm using imported screenshots that i downloaded from emumovies, which for the most part are the same ones from screenscraper.fr (i think).

instructions for use:

drop this "marquee" file in /home/pi/.skyscraper/import/marquees/

create a file /home/pi/.skyscraper/artwork_gb.xml and paste the following code into it:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
    <output type="screenshot" width="730" height="480">
        <layer resource="screenshot" x="20" width="433" height="390" align="center" valign="middle">
            <balance red="1" green="-10" blue="0"/>
            <brightness value="20" />
            <contrast value="-45" />
            <rounded radius="10" />
            <stroke width="5" />
        </layer>
        <layer resource="marquee" x="20" width="433" height="390" align="center" valign="middle" mode="hardlight">
            <opacity value="80"/>
            <rounded radius="10" />
            <stroke width="5" />
        </layer>
        <layer resource="cover" height="212" x="0" y="-10" valign="bottom">
            <shadow distance="5" softness="5" opacity="69" />
        </layer>
        <layer resource="wheel" width="250" x="-10" align="right">
            <shadow distance="5" softness="5" opacity="69" />
        </layer>
    </output>
</artwork>

next you can go one of two routes: download this zip package (851.0 kB) containing 1,870 pre-made .png copies of the marquee, which should cover most no-intro rom file names out there, or, open a terminal and run a command that will make copies of the marquee for just your set of roms. here's a basic command that assumes your roms are zip files. if not just exchange both instances of .zip with .7z/.gb:

ls ~/RetroPie/roms/gb | egrep *.zip | sed -e 's/^/cp "gb_marquee.png" "/' | sed 's/.zip/.png"/' > ~/.skyscraper/import/marquees/gb_roms.txt && cd ~/.skyscraper/import/marquees && bash gb_roms.txt && rm gb_roms.txt

the final step of course is to import them into your Skyscraper database, and that's pretty much it.
Skyscraper -p gb -s import --pretend && Skyscraper -p gb -s localdb --nobrackets -a ~/.skyscraper/artwork_gb.xml --region us --updatedb

so feel free to post your results and/or improvements (adding a dot matrix grid to the overlay might look nice..).

i hope this will be useful to someone, even though it barely scratches the surface of what is possible with Skyscraper =)

happy scraping!

muldjord

@chipsnblip Dude, that is friggin' awesome! Glad someone is actually starting to use the compositor :) I haven't seen many do this so far and it really is quite powerful.

muldjord

Skyscraper version 2.5.3 released: https://github.com/muldjord/skyscraper

Added '--allowext' option which will force allow a file extension for the given platform. Thank you to 'herbymachine' for suggesting this
Added 'allowExtension' to the '[homedir]/.skyscraper/config.ini' variables for both 'main' and 'platform' specific sections. This is useful if you wish to permanently add a file extension to all or one platform when scraping
Implemented 'developers' change in 'thegamesdb' API
Implemented 'publishers' change in 'thegamesdb' API
Fixed 'Tags' bug in 'screenscraper' module

So I just wanted this update out there. The 'thegamesdb' module has been updated to include the latest API changes they've made. The API has been quite unstable while they have been working out the kinks, but they say this is the final version.
Also some other stuff in there. You can now add an extension yourself, simply by using '--allowext *.[ext]'. And finally I fixed a bug that made 'Tags' not work with the 'screenscraper' module.

muldjord

Skyscraper version 2.5.5 released: https://github.com/muldjord/skyscraper

Fixes an issue with the 'screenscraper' module which would cause it to not recognize a lot of games due to a region not being recognized by Skyscraper
Changed '--updatedb' to '--refresh'. '--updatedb' still works, but is considered deprecated

Updating is highly recommended if you've had trouble with some games not being recognized by screenscraper even though the filename is in their database. I would like to thank all of the people who reported this issue. It took me a while to realize why it didn't work, so it's a fix that's been a long time coming.

muldjord

2.6.0 on the way. Now with 'mobygames' support. It is extremely slow though. Only scrapes about 100 roms an hour to accomodate their limits. But it is very useful if no other sources provide info.

muldjord

So the MobyGames scraping source implementation is basically done. I did some testing of it last night, and it works well. One thing to note though. As the 360 requests per hour is a KEY limit and not an IP limit, this means that if simultaneous Skyscraper users scrape with 'mobygames' it will probably give back errors.

What are your thoughts on this? I should still implement it, right? I could just give back an error saying something like "Some other Skyscraper user is currently using the 'mobygames' scraping source. Please try again later.". And I will also apply a limit to only scraping 30 games or so per scraping run for the source. Does this seem sensible to you guys?

MobyGames would therefore be a source you could use for a few games that you are having trouble with with the other sources. Not that MobyGames would guarantee a result, but at least then you would have one extra tool in the toolbox to try. :)

muldjord

Skyscraper version 2.6.0 released: https://github.com/muldjord/skyscraper

Added 'mobygames' scraping module

Official release notes: Be aware that the 'mobygames' scraping module has a limit of 360 requests per hour for THE ENTIRE Skyscraper user base. This means that if someone else is currently using it, you might get errors, and Skyscraper will quit. For this reason, it is recommended to ONLY use 'mobygames' for a few games. Using 'mobygames' is also limited to a maximum of 25 roms per scraping run. So please be considerate and never overuse it.

parasven

Solved: Forgot to edit the artwork.xml

@muldjord
I have just discovered your scraper. Do i have to specify a special switch or something so that marquee and wheel files get transfered too?

I can see that marquee and wheel have been downloaded to the localdb but when i scrap from local db there are empty entries in my gamelist.xml

Im on Version 2.6.0
Example:

<gameList>
  <game>
    <path>/home/sven/RetroPie/roms/gb/Mystical Ninja Starring Goemon.zip</path>
    <name>Mystical Ninja Starring Goemon</name>
    <cover />
    <image>/home/sven/.emulationstation/downloaded_images/gb/screenshots/Mystical Ninja Starring Goemon.png</image>
    <marquee />
    <video>/home/sven/.emulationstation/downloaded_images/gb/videos/Mystical Ninja Starring Goemon.mp4</video>
    <rating>0.7</rating>
    <desc>Oh no!  The evil Skull King and his army of evil pirates are plotting to take over Ancient Japan!  Only Goemon, Ebisumaru, Sasuke and Yae stand in their way.  Fight your way through Gadget Castle, The Black Ship Skull, The Demon Cave and Skeleton Island as you make your way to face off against the Skull King himself.

Mystical Ninja Starring Goemon is an overhead action game with light RPG elements, similar to Link&apos;s Awakening.  Players explore towns, dungeons, castles and more, searching for treasure chests and attacking enemies.  There are also shops and inns where you can purchase items or rest to recover your strength.

At the end of each level, there is a boss waiting for you.  However, to defeat them, you must compete in a mini-game called &apos;The Surprise Match&apos;.  These mini-games usually feature button-mashing of some kind or a variation on a memory match game. </desc>
    <releasedate>0000-00-00</releasedate>
    <developer>Konami&amp;nbsp;Computer&amp;nbsp;Entertainment&amp;nbsp;Nagoya&amp;nbsp;Co.,&amp;nbsp;Ltd.</developer>
    <publisher>Konami&amp;nbsp;of&amp;nbsp;America,&amp;nbsp;Inc.</publisher>
    <genre>Action, Role playing games</genre>
    <players>1</players>
    <kidgame>true</kidgame>
  </game>
</gameList>

Output of the Scraper:

------------------------------------------
Running Skyscraper v2.6.0 by Lars Muldjord
------------------------------------------
Platform:           'gb'
Scraper module:     'localdb'
Input folder:       '/home/sven/RetroPie/roms/gb'
Game list folder:   '/home/sven/.emulationstation/gamelists/gb'
Covers folder:      '/home/sven/.emulationstation/downloaded_images/gb/covers'
Screenshots folder: '/home/sven/.emulationstation/downloaded_images/gb/screenshots'
Wheels folder:      '/home/sven/.emulationstation/downloaded_images/gb/wheels'
Marquees folder:    '/home/sven/.emulationstation/downloaded_images/gb/marquees'
Videos folder:      '/home/sven/.emulationstation/downloaded_images/gb/videos'
Local db folder:    'dbs/gb'

Reading and parsing local database, please wait...
Successfully parsed 15 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!


Starting scraping run on 1 files using 4 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) ---- Game 'Mystical Ninja Starring Goemon' found! :) ----
Scraper:        localdb
Search match:   100 %
Compare title:  'Mystical Ninja Starring Goemon'
Result title:   'Mystical Ninja Starring Goemon' (screenscraper)
Platform:       'Game Boy' (screenscraper)
Release Date:   '' (screenscraper)
Developer:      'Konami&nbsp;Computer&nbsp;Entertainment&nbsp;Nagoya&nbsp;Co.,&nbsp;Ltd.' (screenscraper)
Publisher:      'Konami&nbsp;of&nbsp;America,&nbsp;Inc.' (screenscraper)
Players:        '1' (screenscraper)
Ages:           '6+' (screenscraper)
Tags:           'Action, Role playing games' (screenscraper)
Rating (0-1):   '0.7' (screenscraper)
Cover:          YES (screenscraper)
Screenshot:     YES (screenscraper)
Wheel:          YES (screenscraper)
Marquee:        YES (screenscraper)
Video:          YES (screenscraper)

Description: (screenscraper)
Oh no!  The evil Skull King and his army of evil pirates are plotting to take over Ancient Japan!  Only Goemon, Ebisumaru, Sasuke and Yae stand in their way.  Fight your way through Gadget Castle, The Black Ship Skull, The Demon Cave and Skeleton Island as you make your way to face off against the Skull King himself.

Mystical Ninja Starring Goemon is an overhead action game with light RPG elements, similar to Link's Awakening.  Players explore towns, dungeons, castles and more, searching for treasure chests and attacking enemies.  There are also shops and inns where you can purchase items or rest to recover your strength.

At the end of each level, there is a boss waiting for you.  However, to defeat them, you must compete in a mini-game called 'The Surprise Match'.  These mini-games usually feature button-mashing of some kind or a variation on a memory match game.

Elapsed time: 00:00:00
Estimated time left: 00:00:00

---- Scraping run completed! YAY! ----
Writing 15 (0 new) resources to local database, please wait... Success!
Now writing '/home/sven/.emulationstation/gamelists/gb/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Average search match: 100%
Average entry completeness: 100%

Total number of games: 1
Successfully scraped games: 1
Skipped games: 0 (Filenames saved to '[homedir]/.skyscraper/skipped-localdb.txt')

Used command:

Skyscraper -p gb -g /home/sven/.emulationstation/gamelists/gb -o /home/sven/.emulationstation/downloaded_images/gb --refresh --nobrackets --videos --unattend -s localdb

Am i missing something ?

muldjord

2.7.0 is shaping up really nicely. I've added an interactive mode set with '--interactive' which will ask the user instead of auto-choosing which result is best. This has been requested a few times and I really find it useful if you're struggling to get good data on a few roms.
I've also added the 'igdb' (http://igdb.com/) scraping module which is nearly finished. The api for it is pretty good, but it has some huge caveats which means I can't scrape any media for it and it also has issues with scraping publisher and developer. The module needs a user-key so each user will have to request one from them. And there's also a request limit of 100 requests per month for a free membership. You can pay for more if you like. The limit is actually 3000 for normal requests, but I have to use what they call "expanded requests" to get more than just the integer id's of the data which is limited to 100. Quite a difference there.

circo

@muldjord said in Versatile C++ game scraper: Skyscraper:

Does this seem sensible to you guys?

Absolutely, great idea! Having an extra tool can never hurt, and the user limit should somewhat counteract the key limit from moby.

dorkvader

First-time poster, long-time lurker ... I absolutely love Skyscraper, muldjord. After spending hours, days, weeks, months, and probably a year seeking art, creating it when there was none, and then combining multiple components for thousands of custom images, I have tossed aside that laborious process and the results for the impressive uniform look that Skyscraper provides and my all-too-eager and all-too-meager skills don't.

Praise and limited skill set aside, my question: Is there ANY WAY to remove the black outline on the "gamebox" image? I'm talking specifically about the black bar on the top and bottom of the generated gamebox front and the gamebox side. I have tried myriad ways to either stumble my way through this dilemma or work around it. (For example, I trimmed the black border almost entirely from Skyscraper's included .png images, only to have the .png front and side box return after a scraping.)

I have failed in every attempt.

My goal is to use gamebox "spine" art -- title specific when possible, generic platform template as fallback . Unfortunately, the spine art when substituted for the "wheel" art doesn't line up with the box front cover -- e.g, those Atari 2600, 5200, 7800 silver boxes with the rainbow-like bar on the front and side will not align as a single ban. The two sides come justcloseenough so that they should be together but just far enough apart that it's glaringly apparent that they don't.

Thank you, again, for you amazing contribution. And for anyone reading this who hasn't used Skyscraper, trust me when I say, if you really are into the look of your game artwork, spend the few hours or much less to learn and use Skyscraper, rather than the hours and more (much, much more) you will spend doing it yourself: amassing, creating, and then building your artwork collection by hand.

muldjord

@dorkvader First of all, thank you very much for the nice words, I appreciate that a lot :) I am glad you find Skyscraper useful.

Concerning your question, there is currently no way of removing the black bars, and I am having a bit of trouble understanding what you mean. Could you post a screenshot where the flaw is present, so I can figure out what to do about it? That would be great :)

dorkvader

@muldjord THANKS! I'm at work at the moment but will post a screenshot tonight.

muldjord

Skyscraper version 2.7.0 released: https://github.com/muldjord/skyscraper

Added scraping module specific configs in config.ini to allow different userCreds per scraping module
Added 'igdb' scraping module
Added '--interactive' mode which will ask user to choose the best result
Changed release date patterns to regexps for cleaner code
Fixed bug which caused <video> tags to disappear when skipping entries with emulationstation (thank you to ozy24 for reporting this)
Reached commit number 1000!!! I'm gonna celebrate by coding some more
Conformed [homedir] to be '~' everywhere
Added several aliases to platforms to better support all scraping modules
Added option for custom header when initiating network requests
When generating search names, all instances of ' v.X' and ' rev.X' are now removed prior to search
Added '--unattendskip' option to allow for always skipping existing entries without user input (thank you to ozy24 for suggesting this)

You can now set the '--interactive' flag which will cause Skyscraper to prompt for the correct entry whenever a search based scraping module is used (mobygames, thegamesdb, igdb, openretro). For 'screenscraper' and 'arcadedb' this is not enabled as they are checksum and id based, which means they are always correct, so no user input needed.

I've also added the 'igdb' module which needs a key. Request one at their site if you wish to use it. The module/api has some caveats meaning that it won't scrape any media at all as I can't guarantee that the returned media is for the requested platform. I've managed to sortof get around another caveat which means that publisher and developer is not connected to a platform either, so I only use those if only one is returned. But hey, it's another tool in the toolbox. :)

Please try this new release out and let me know if you encounter any errors or inconsistencies. I've done a fair bit of testing myself, but it's software so, you know, bugs/glitches! :)

AnalogHero

@muldjord Awesome. Need to try the interactive mode asap.