Versatile C++ game scraper: Skyscraper

jura

@muldjord , your tool works amazingly great!
This good example when you are good at something and your are doing something for the community! :)

I have 700 nes and 700 snes and 1000 megadrive roms and scraping those took 30mins.
Which was amazing!

It found 80% of the covers for the roms but for c64 it found only 700 when i have 17 000 (17 thousand) roms. :)
Dunno is it possible to set the match percentage as a parameter to allow lower findings, or is it even necessary.

Any idea how I could increase the result persentage at least for C64?

dylosaur

@muldjord I'm really liking your scraper after trying a couple out. One tiny quip I was hoping I could find a solution for-- is there a way to make it so that the game name matches the file name after running several scrapers/using the localdb? For instance, my Atari 2600 "Bowling" now says "32-In-1 - 30 - Bowling" which is technically correct, but a little more cluttered than I'd like on the screen. Was going to just go in and edit the XML but figured I'd check if there was a command I could use to automate instead.

muldjord

@dylosaur Glad you like it. No, currently there is no way of forcing it to use the file name instead of the search result name.

muldjord

@jura Changing the percentage match has been possible for a looong time :). Use '-m [minimum percentage]'. Remember to read the readme and check the '--help' output. It's all in there.

dylosaur

@muldjord Thanks man. One more question: I want to just scrape screenshots with no drop shadow added. Am I correct in saying that if my config file just says the following, then I'll just get a centered, 500 x 375 screenshot? Not really sure what the X/Y tags are for, is that for position? Just want to make sure it's centered in the box.

[artwork]
screenshotEnabled="true"
screenshotWidth="500"
screenshotHeight="375"
screenshotX="47"
screenshotY="0"

EDIT: Just scanned through this thread and saw your response to somebody else regarding this. Having some trouble getting the import system to work. If I just do screenshots, it seems to work just fine, but when I add textual data on top of that, it no longer works for either the screenshot or the text data. Any solutions here?

dylosaur

@muldjord Here's what my text file looks like for MegaMania - A Space Nightmare:

<game>
<description>Megamania is an arcade action game with gameplay similar to Space Invaders. You control a spaceship at the bottom of the screen and need to defend yourself from numerous waves of attackers. But in Megamania, the attackers much more bizarre than invading aliens. There are hamburgers, diamonds, steam irons, bow ties, space dice, and more! You need to earn points by destroying as many of these strange attackers as you can before your energy supply becomes depleted.</description>
<developer>Activision</developer>
<publisher>Activision</publisher>
<players>1-2</players>
<genre>Action</genre>
<releasedate>1982</releasedate>
</game>

And the definitions file:

<game>
  <description>###DESCRIPTION###</description>
  <developer>###DEVELOPER###</developer>
  <publisher>###PUBLISHER###</publisher>
  <players>###PLAYERS###</players>
  <rating>###RATING###</rating>
  <genre>###TAGS###</genre>
  <releasedate>###RELEASEDATE###</releasedate>
</game>

Actually, I just did some tinkering around, and it seems it can't do pictures and text at the same time. Once I separated them it started working. Just wanted to point that out to you.

muldjord

@dylosaur That is actually a bug, and I think I've found the culprit. I'll try and fix it in a quick release as soon as I get the time. Thank you for notifying me about this.

muldjord

Skyscraper 2.0.5c released. Last Skyscraper release of 2017. Just a quick bugfix release: https://github.com/muldjord/skyscraper

Fixed a bug where the import module wouldn't look for all resource types during the same run. Thank you to 'dylosaur' for notifying me about this.

Merry christmas everyone, and happy new year!

dylosaur

@muldjord No problem ! I love your scraper. Wish I could use it with some of the unsupported systems because it makes changing the sizes of my screenshots a breeze. Keep it up.

muldjord

@dylosaur Let me know which ones you'd like me to include. Adding more systems is quite easy in Skyscraper, as long as the sources I currently support also support the platforms.

dylosaur

@muldjord Oh, cool. That would be great. I'm working on it for a gift.

Right now, the ones I'm looking to add that you don't support are:

Atari Lynx
GCE Vextrex
NeoGeo Pocket Color
Sega 32x

Was also looking to add, but might not get to (because of time):

Virtual Boy
Magnavox-Odyssey2
TurboGrafx16

How do I update the scraper, by the way? Just follow the install steps again?

muldjord

Skyscraper 2.0.6 released!

Ok, so THIS is the last Skyscraper release of 2017 (except bug fixes if something major pops up). Added a bunch of platforms: https://github.com/muldjord/skyscraper

Added support for atarilynx, vectrex, ngpc (Neo Geo Pocket Color), sega32x, virtualboy and videopac platforms.

Merry christmas everyone, and happy new year!

muldjord

@dylosaur I've added the platforms you've requested. Pc-Engine was already in there. Let me know if any of them don't work. I haven't tested them, just implemented them.

Yes, to get the newest release, just follow the steps again, that's the easiest way to do it. :)

dylosaur

@muldjord I just did the Lynx, Vectrex, NGPC, and Sega32x and can confirm they all work perfectly. This will make finishing this gift so much easier. I'll try the others tomorrow and let you know. Thanks so much!

muldjord

Just started working on the new 'scripting' mode ('scripter' branch on github). This will be easier to implement than I initially thought. Might even be out before new year! We'll see. :)

Anyways, I am off for the holidays, have fun everyone!

Rion

@muldjord God Jul/Glædelig Jul/ Merry Christmas

muldjord

@rion Lige over / right back at ya'!

AnalogHero

@muldjord Tried the new version with Atari7800 and it worked fine! Thanks again for this tool and Merry Christmas to all reading :)

muldjord

@analoghero Thank you, glad it works!

muldjord

Skyscraper 2.1.0 released! https://github.com/muldjord/skyscraper

Added 'simple mode'
Refactored and cleaned up all emulation platform specific code

'Simple mode' works by simply typing 'Skyscraper' after you've installed it. It will ask you a bunch of questions. Based on your answers it will generate a script and run it (if you allow it). When the script is done, you will have the optimal result for the chosen platform. This should help a lot of the newcomers to get the best results from the get-go. It basically does multiple scraping runs using the supported scraping modules for the selected platform, and finishes the job by scraping with 'localdb' to bind it all together.

Please try it out and let me know if you have any feedback or discover bugs. I've tested it a bit myself, but I've refactored a lot of code, so bugs can have snuck in through the process.

Happy scraping! And happy new year everyone!