Versatile C++ game scraper: Skyscraper

muldjord

@analoghero Thank you, glad it works!

muldjord

Skyscraper 2.1.0 released! https://github.com/muldjord/skyscraper

Added 'simple mode'
Refactored and cleaned up all emulation platform specific code

'Simple mode' works by simply typing 'Skyscraper' after you've installed it. It will ask you a bunch of questions. Based on your answers it will generate a script and run it (if you allow it). When the script is done, you will have the optimal result for the chosen platform. This should help a lot of the newcomers to get the best results from the get-go. It basically does multiple scraping runs using the supported scraping modules for the selected platform, and finishes the job by scraping with 'localdb' to bind it all together.

Please try it out and let me know if you have any feedback or discover bugs. I've tested it a bit myself, but I've refactored a lot of code, so bugs can have snuck in through the process.

Happy scraping! And happy new year everyone!

AnalogHero

@muldjord Was about to test simple mode with the beta but then 2.1.0 was released and i updated to it.
It worked fine for me with sega32x. It did 2 web runs and the a localdb run.

muldjord

Skyscraper 2.1.1 released: https://github.com/muldjord/skyscraper

Added support for '3do', 'amstradcpc', 'coco', 'daphne', 'dragon32', 'dreamcast', 'fds', 'gameandwatch', 'intellivision', 'ngp', 'oric', 'pc88', 'pcfx', 'saturn', 'sg-1000', 'trs-80' and 'zmachine' platforms
Added 'forcefilename' option which forces the file name to be used in the frontend instead of the one returned by the scraping module
Changed 'gamelist.xml' to 'game list' in 'simple mode' question

Be sure to also check the notes for 2.1.0 if you haven't already (there's some cool new stuff in that one). Happy scraping!

SammyBoy

@muldjord Are you planning to add Fb Alpha as well allow ROM Hacks to be scraped in the future. Because I've got a lot of FB Alpha ROMS and one final question are the thumbnails/artworks that are scraped good quality because I know Steven Selph Scraper isn't in good quality. Sorry for the long message.

Rion

@sammyboy I think you can already scrape fba roms using the Arcade parameter because it's already incorporated by using adb.arcadeitalia.net?

muldjord

@sammyboy The artwork quality completely depends on what the scraping sources deliver. So I can't give you a clear answer to that as I am not the one creating them. I am planning a bit of a rewrite of the artwork part of Skyscraper, to allow for the largest possible artwork for those who needs it. Currently I resize them to save space, but I plan to allow users to disable this. Let me know what your thoughts are on this. Concerning 'fba' I will include it in 2.2.0. But as Rion suggests, it's already possible to scrape them with 'arcade'. But since you request it, of course I will implement it. :)

To all of you, 2.2.0 is just around the corner. Some big stuff in this release. Most importantly Skyscraper now also scrapes for 'wheel' and 'marquee' artwork. Not only that, but I also rewrote the artwork handler completely. A byproduct of this means you will be able to save the artwork directly "as is", completely disabling the artwork compositor. This has been requested quite a few times, so I hope this will be appreciated. It has been one helluwa rewrite. :D

Stay tuned... Need to test stuff, but it's almost ready for release.

SammyBoy

@muldjord Thank you for choosing to add FB Alpha and I will try Skyscraper out today and I know that it will be good. And one last question are you a able to choose what artwork gets scraped? And thank you for hard work and effort.

muldjord

@sammyboy Currently you can only enable/disable video scraping. All artwork types(cover, screenshot, wheel and marquee) will be scraped if it's available from the scraping source. I will try and find a way to enable/disable them individually.

SammyBoy

@muldjord Thank you for all your help.

muldjord

NOTE!!! The new version is not out yet! But below follows some notes I wanted to throw out there before release.

The new image compositor is now pretty much done, and I have to say, it's pretty awesome! Not only do Skyscraper now support wheel and marquee, but it also allows you to set up the output images EXACTLY the way you want it. It will use a format like this:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <!--
      <output>
      - Leave out 'width' and 'height' to use dimensions of first image resource
  -->
  <output type="screenshot" width="640" height="400">
    <!--
	<image>
	- Leave out 'width' or 'height' to keep aspect ratio.
	- Leave out both 'width' and 'height' to use original size.
	- 'align' can be 'left', 'center' or 'right'
	- 'valign' can be 'top', 'middle' or 'bottom'
	- 'x' and 'y' defines top-left coordinate. It is relative to 'align' and 'valign' if used.
	- Image nodes are rendered from bottom to top.
    -->
    <image resource="cover" valign="bottom" y="-15" height="200">
      <shadow distance="10" softness="10" opacity="75"/>
    </image>
    <image resource="screenshot" align="right" valign="top" width="520" height="390"/>
  </output>
  <!-- If an output node contains no image nodes, it will save it exactly as it was delivered by the scraping source. -->
  <output type="cover"/>
  <output type="wheel"/>
  <output type="marquee"/>
</artwork>

As you can see there are 4 different artwork outputs in this particular example. It will output 'cover', 'wheel' and 'marquee' exactly as they are delivered by the scraping source. 'screenshot' is composited of the 'screenshot' with a 'cover' with a drop shadow on top.

If an '<output>' node for any of the types ('cover', 'screenshot', 'wheel', 'marquee') is left out, it will simply not be exported while scraping. All data is still saved in the localdb cache, so you can change the artwork.xml any time you wish, and it will use all of the resources you have scraped and cached.

Any thoughts on this? It's a huge step from the old config.ini way of doing this I think.

Rion

@muldjord Great idé, i like it!

(Vilken jävla bra idé! Jag vill nästan gå så långt och säga att jag blir lite mörkrädd. Så bra var det!)

muldjord

@rion Mörkrädd!? :D Jeg håber ikke det bliver et horror release. :P

ETA for 2.2.0 won't be long - probably a week at the most. I just need to polish some rough edges and do a bunch of testing. Also need to document the new artwork functionality. :)

muldjord

So, I am pretty much ready to release Skyscraper 2.2.0. This is the biggest release since 2.0.0!

But, I am currently trying to figure out what pieces of artwork EmulationStation supports in the gamelist.xml files. I know about 'image' and 'video' but I can't seem to find proper documentation for whether it supports 'boxart', 'wheel' and 'marquee'. Do any of you guys know what xml nodes are needed for these? And are they supported at all by EmulationStation?

AnalogHero

@muldjord Dont know really. Depends on the theme used i guess.

Quick question: Can you now compose the image lets say with screenshot, boxart and marquee or wheel?

muldjord

@analoghero Yes, and you can do it any way you like. With as many layers as you want. You can even create the wheel graphics from cover and screenshot if you want (don't know why anyone would want that, but it's possible) :) It's one very simple xml file that needs to be edited and you are good to go.

Concerning the gamelist.xml types, I would assume the themes use them which means they would probably be listed in gamelist.xml somehow. So my question stands. There must be nodes for them like there is an <image> and <video> node. I just don't know what they are... Might be <wheel> or something...

EDIT: Ok, so I finally found some thread that mentioned a <marquee> gamelist.xml node. And to my surprise it seems that "wheel" and "marquee" are somewhat the same thing... I find it to be quite confusing.

EDIT2: I've decided to use the 'wheel' graphics and insert it into the '<marquee>' node in the EmulationStation gamelist.xml. Makes most sense to me.

SammyBoy

@muldjord is skyscraper 2.2.0 out?

SammyBoy

@sammyboy I'm only asking because I was just downloading skyscraper when I realised that you updated the github files 30mins ago but not the download instructions.

muldjord

@sammyboy Not yet, but almost... Currently trying to do a quick and dirty video on some of the new features.

It will be released in 20 minutes, at midnight where I am.

SammyBoy

If I download 2.1.1 how will I update the scraper?