Versatile C++ game scraper: Skyscraper

muldjord

So the MobyGames scraping source implementation is basically done. I did some testing of it last night, and it works well. One thing to note though. As the 360 requests per hour is a KEY limit and not an IP limit, this means that if simultaneous Skyscraper users scrape with 'mobygames' it will probably give back errors.

What are your thoughts on this? I should still implement it, right? I could just give back an error saying something like "Some other Skyscraper user is currently using the 'mobygames' scraping source. Please try again later.". And I will also apply a limit to only scraping 30 games or so per scraping run for the source. Does this seem sensible to you guys?

MobyGames would therefore be a source you could use for a few games that you are having trouble with with the other sources. Not that MobyGames would guarantee a result, but at least then you would have one extra tool in the toolbox to try. :)

muldjord

Skyscraper version 2.6.0 released: https://github.com/muldjord/skyscraper

Added 'mobygames' scraping module

Official release notes: Be aware that the 'mobygames' scraping module has a limit of 360 requests per hour for THE ENTIRE Skyscraper user base. This means that if someone else is currently using it, you might get errors, and Skyscraper will quit. For this reason, it is recommended to ONLY use 'mobygames' for a few games. Using 'mobygames' is also limited to a maximum of 25 roms per scraping run. So please be considerate and never overuse it.

parasven

Solved: Forgot to edit the artwork.xml

@muldjord
I have just discovered your scraper. Do i have to specify a special switch or something so that marquee and wheel files get transfered too?

I can see that marquee and wheel have been downloaded to the localdb but when i scrap from local db there are empty entries in my gamelist.xml

Im on Version 2.6.0
Example:

<gameList>
  <game>
    <path>/home/sven/RetroPie/roms/gb/Mystical Ninja Starring Goemon.zip</path>
    <name>Mystical Ninja Starring Goemon</name>
    <cover />
    <image>/home/sven/.emulationstation/downloaded_images/gb/screenshots/Mystical Ninja Starring Goemon.png</image>
    <marquee />
    <video>/home/sven/.emulationstation/downloaded_images/gb/videos/Mystical Ninja Starring Goemon.mp4</video>
    <rating>0.7</rating>
    <desc>Oh no!  The evil Skull King and his army of evil pirates are plotting to take over Ancient Japan!  Only Goemon, Ebisumaru, Sasuke and Yae stand in their way.  Fight your way through Gadget Castle, The Black Ship Skull, The Demon Cave and Skeleton Island as you make your way to face off against the Skull King himself.

Mystical Ninja Starring Goemon is an overhead action game with light RPG elements, similar to Link&apos;s Awakening.  Players explore towns, dungeons, castles and more, searching for treasure chests and attacking enemies.  There are also shops and inns where you can purchase items or rest to recover your strength.

At the end of each level, there is a boss waiting for you.  However, to defeat them, you must compete in a mini-game called &apos;The Surprise Match&apos;.  These mini-games usually feature button-mashing of some kind or a variation on a memory match game. </desc>
    <releasedate>0000-00-00</releasedate>
    <developer>Konami&amp;nbsp;Computer&amp;nbsp;Entertainment&amp;nbsp;Nagoya&amp;nbsp;Co.,&amp;nbsp;Ltd.</developer>
    <publisher>Konami&amp;nbsp;of&amp;nbsp;America,&amp;nbsp;Inc.</publisher>
    <genre>Action, Role playing games</genre>
    <players>1</players>
    <kidgame>true</kidgame>
  </game>
</gameList>

Output of the Scraper:

------------------------------------------
Running Skyscraper v2.6.0 by Lars Muldjord
------------------------------------------
Platform:           'gb'
Scraper module:     'localdb'
Input folder:       '/home/sven/RetroPie/roms/gb'
Game list folder:   '/home/sven/.emulationstation/gamelists/gb'
Covers folder:      '/home/sven/.emulationstation/downloaded_images/gb/covers'
Screenshots folder: '/home/sven/.emulationstation/downloaded_images/gb/screenshots'
Wheels folder:      '/home/sven/.emulationstation/downloaded_images/gb/wheels'
Marquees folder:    '/home/sven/.emulationstation/downloaded_images/gb/marquees'
Videos folder:      '/home/sven/.emulationstation/downloaded_images/gb/videos'
Local db folder:    'dbs/gb'

Reading and parsing local database, please wait...
Successfully parsed 15 resources!

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!


Starting scraping run on 1 files using 4 threads.
Sit back, relax and let me do the work! :)

#1/1 (T1) ---- Game 'Mystical Ninja Starring Goemon' found! :) ----
Scraper:        localdb
Search match:   100 %
Compare title:  'Mystical Ninja Starring Goemon'
Result title:   'Mystical Ninja Starring Goemon' (screenscraper)
Platform:       'Game Boy' (screenscraper)
Release Date:   '' (screenscraper)
Developer:      'Konami&nbsp;Computer&nbsp;Entertainment&nbsp;Nagoya&nbsp;Co.,&nbsp;Ltd.' (screenscraper)
Publisher:      'Konami&nbsp;of&nbsp;America,&nbsp;Inc.' (screenscraper)
Players:        '1' (screenscraper)
Ages:           '6+' (screenscraper)
Tags:           'Action, Role playing games' (screenscraper)
Rating (0-1):   '0.7' (screenscraper)
Cover:          YES (screenscraper)
Screenshot:     YES (screenscraper)
Wheel:          YES (screenscraper)
Marquee:        YES (screenscraper)
Video:          YES (screenscraper)

Description: (screenscraper)
Oh no!  The evil Skull King and his army of evil pirates are plotting to take over Ancient Japan!  Only Goemon, Ebisumaru, Sasuke and Yae stand in their way.  Fight your way through Gadget Castle, The Black Ship Skull, The Demon Cave and Skeleton Island as you make your way to face off against the Skull King himself.

Mystical Ninja Starring Goemon is an overhead action game with light RPG elements, similar to Link's Awakening.  Players explore towns, dungeons, castles and more, searching for treasure chests and attacking enemies.  There are also shops and inns where you can purchase items or rest to recover your strength.

At the end of each level, there is a boss waiting for you.  However, to defeat them, you must compete in a mini-game called 'The Surprise Match'.  These mini-games usually feature button-mashing of some kind or a variation on a memory match game.

Elapsed time: 00:00:00
Estimated time left: 00:00:00

---- Scraping run completed! YAY! ----
Writing 15 (0 new) resources to local database, please wait... Success!
Now writing '/home/sven/.emulationstation/gamelists/gb/gamelist.xml'... Success!!!

---- And here are some neat stats :) ----
Total completion time: 00:00:00

Average search match: 100%
Average entry completeness: 100%

Total number of games: 1
Successfully scraped games: 1
Skipped games: 0 (Filenames saved to '[homedir]/.skyscraper/skipped-localdb.txt')

Used command:

Skyscraper -p gb -g /home/sven/.emulationstation/gamelists/gb -o /home/sven/.emulationstation/downloaded_images/gb --refresh --nobrackets --videos --unattend -s localdb

Am i missing something ?

muldjord

2.7.0 is shaping up really nicely. I've added an interactive mode set with '--interactive' which will ask the user instead of auto-choosing which result is best. This has been requested a few times and I really find it useful if you're struggling to get good data on a few roms.
I've also added the 'igdb' (http://igdb.com/) scraping module which is nearly finished. The api for it is pretty good, but it has some huge caveats which means I can't scrape any media for it and it also has issues with scraping publisher and developer. The module needs a user-key so each user will have to request one from them. And there's also a request limit of 100 requests per month for a free membership. You can pay for more if you like. The limit is actually 3000 for normal requests, but I have to use what they call "expanded requests" to get more than just the integer id's of the data which is limited to 100. Quite a difference there.

circo

@muldjord said in Versatile C++ game scraper: Skyscraper:

Does this seem sensible to you guys?

Absolutely, great idea! Having an extra tool can never hurt, and the user limit should somewhat counteract the key limit from moby.

dorkvader

First-time poster, long-time lurker ... I absolutely love Skyscraper, muldjord. After spending hours, days, weeks, months, and probably a year seeking art, creating it when there was none, and then combining multiple components for thousands of custom images, I have tossed aside that laborious process and the results for the impressive uniform look that Skyscraper provides and my all-too-eager and all-too-meager skills don't.

Praise and limited skill set aside, my question: Is there ANY WAY to remove the black outline on the "gamebox" image? I'm talking specifically about the black bar on the top and bottom of the generated gamebox front and the gamebox side. I have tried myriad ways to either stumble my way through this dilemma or work around it. (For example, I trimmed the black border almost entirely from Skyscraper's included .png images, only to have the .png front and side box return after a scraping.)

I have failed in every attempt.

My goal is to use gamebox "spine" art -- title specific when possible, generic platform template as fallback . Unfortunately, the spine art when substituted for the "wheel" art doesn't line up with the box front cover -- e.g, those Atari 2600, 5200, 7800 silver boxes with the rainbow-like bar on the front and side will not align as a single ban. The two sides come justcloseenough so that they should be together but just far enough apart that it's glaringly apparent that they don't.

Thank you, again, for you amazing contribution. And for anyone reading this who hasn't used Skyscraper, trust me when I say, if you really are into the look of your game artwork, spend the few hours or much less to learn and use Skyscraper, rather than the hours and more (much, much more) you will spend doing it yourself: amassing, creating, and then building your artwork collection by hand.

muldjord

@dorkvader First of all, thank you very much for the nice words, I appreciate that a lot :) I am glad you find Skyscraper useful.

Concerning your question, there is currently no way of removing the black bars, and I am having a bit of trouble understanding what you mean. Could you post a screenshot where the flaw is present, so I can figure out what to do about it? That would be great :)

dorkvader

@muldjord THANKS! I'm at work at the moment but will post a screenshot tonight.

muldjord

Skyscraper version 2.7.0 released: https://github.com/muldjord/skyscraper

Added scraping module specific configs in config.ini to allow different userCreds per scraping module
Added 'igdb' scraping module
Added '--interactive' mode which will ask user to choose the best result
Changed release date patterns to regexps for cleaner code
Fixed bug which caused <video> tags to disappear when skipping entries with emulationstation (thank you to ozy24 for reporting this)
Reached commit number 1000!!! I'm gonna celebrate by coding some more
Conformed [homedir] to be '~' everywhere
Added several aliases to platforms to better support all scraping modules
Added option for custom header when initiating network requests
When generating search names, all instances of ' v.X' and ' rev.X' are now removed prior to search
Added '--unattendskip' option to allow for always skipping existing entries without user input (thank you to ozy24 for suggesting this)

You can now set the '--interactive' flag which will cause Skyscraper to prompt for the correct entry whenever a search based scraping module is used (mobygames, thegamesdb, igdb, openretro). For 'screenscraper' and 'arcadedb' this is not enabled as they are checksum and id based, which means they are always correct, so no user input needed.

I've also added the 'igdb' module which needs a key. Request one at their site if you wish to use it. The module/api has some caveats meaning that it won't scrape any media at all as I can't guarantee that the returned media is for the requested platform. I've managed to sortof get around another caveat which means that publisher and developer is not connected to a platform either, so I only use those if only one is returned. But hey, it's another tool in the toolbox. :)

Please try this new release out and let me know if you encounter any errors or inconsistencies. I've done a fair bit of testing myself, but it's software so, you know, bugs/glitches! :)

AnalogHero

@muldjord Awesome. Need to try the interactive mode asap.

AnalogHero

@muldjord Tried it. The interactive mode is awesome. I wish this would somehow work for screenscraper. But i understand its crc based...

All i can say is that this scraper has to be included with retropie.

muldjord

@analoghero Awesome! Well, I can enable it with screenscraper aswell, but I don't really see how it would be useful as there will always only be one entry to choose from since its checksum based. If you have any thoughts on this, I'd be happy to hear them.

muldjord

Wow, I just went to check out EmuMovies. The community on there is doing some pretty amazing work!

I just downloaded the wheels pack for the Sega Master System, plopped them onto my Pi in the '~/.skyscraper/import/wheels' folder and ran "Skyscraper -p mastersystem -s import" and then "Skyscraper -p mastersystem -s localdb" (-s localdb not really necessary as it is default if it's left out) to make use of the imported data and BAM! I now have wheels for about 70% or my Mastersystem roms. That is pretty amazing. All I had to do was to make sure that my rom naming conventions match that of the pack, which is often Hyperspin.

You guys should seriously check that out.

dorkvader

@muldjord OK, here are some images that better explain my intentions and my problem.

As you can see by the Atari 2600 Adventure "gamebox" with its spine created from a "spine image" resource, the sizing is off. (I used the wrong spine color intentionally to demonstrate the slight gap in the top and bottom, which I cannot fix).
0_1535409335995_Adventure BOX WRONG SPINE.png

For point of reference, I've included an Adventure gamebox that was generated with a wheel resource.
0_1535409292350_Adventure BOX NO SPINE.png

An Atlantis "gamebox" shows the other problem with using a "spine image" as the resource: the side and front colors don't match up.
0_1535409378766_Atlantis BOX AND SPINE.png

Finally, I'm including a mixed-art image of what I'm creating for all the systems: screenshot, wheel, gamebox, and marquee (the cartridge).
0_1535409402133_Atlantis MIX ART.png

Again, I love what you've done and cannot tell you enough how much I appreciate your help. Even if you tell me it's impossible, I'll be happy to end this quest for another!

AnalogHero

@muldjord Maybe there is a way to scrape not matching single roms with the screenscraper module by using a override flag and use a user provided checksum.

For example:

Skyscraper -p megadrive -s screenscraper /home/pi/RetroPie/roms/megadrive/Sonic.zip -crc AC400F6A

muldjord

@dorkvader Ah, yes, I see the problem. Let me think about this. I should be able to add some attributes to "gamebox" that will allow this to work. Great idea to use the marquee resource for cartridge btw, I really like that. Awesome!

@AnalogHero Not a bad idea, I'll give this some thought aswell and find a proper way to implement it. I'll probably just call it '--checksum [sum]' as it doesn't really matter if it's the sha1 or the md5 sum people input. Also, I'll disable it if the user tries to scrape more than 1 rom.

AnalogHero

@muldjord Exactly what i thought. Just for single roms.

Then you can scrape amiga .lhas which arent in their database using a checksum from an .adf entry :)

muldjord

@dorkvader 2.7.1 out now which can do what you are looking for. Just add the 'sidescaling="both"' attribute to the gamebox xml note, and it will autoscale the image to completely fit both width and height. That should do the trick. Please try it out and let me know if it does what you expect.

@AnalogHero Concerning the checksum option, I have decided against it for the simple reason that it is a screenscraper only option, and there's not a lot of payoff with it. For most roms you could just as easily just rename the input file, which would take just as long as getting the sha1's. With that said, for the LHA files that you suggest, I am currently investigating if I can get all of the Retroplay LHA's added to screenscraper. Then we can start linking them to the correct games. :)

dorkvader

THANK YOU SOOOOO MUCH, Muldjord! I am excited to try it out and will let you know the results ASAP!

dorkvader

@muldjord said in Versatile C++ game scraper: Skyscraper:

sidescaling="both"

Works BRILLIANTLY! Thought I'd really test it with a Vectrex box, and as you can see it worked beautifully, matching the graph lines as well as one can expect. 0_1535491463132_Scramble Vectrex box.png

Thank you for your help and for your generosity in time and work. If you have some kind of "donation jar," I will gladly throw a tip your way.