Versatile C++ game scraper: Skyscraper

muldjord

@Clyde EmulationStation does not have a <cover>node so it would just be non-used media files taking up space (unless like in your case, where you use them with the runcommand). Only recently did I decide to make it populate the <thumbnail> ES node with the cover. But I won't make it default, as the cover is not a thumbnail. But it's a hack to allow users to export a thumbnail by creating a <output type="cover"> node. You will notice that your gamelist now has a <thumbnail> node for each game that is now populated with the cover. It is only used when browsing in grid mode (unless you use a theme that makes use of it otherwise), and is quite a resource hog to be honest.

Clyde

Again, thanks for the explanation. I actually did notice the new <thumbnail> nodes in the gamelist.xml. 😊

raspnoobie

@muldjord Thanks for the clarification; I think I can add the flag to the script, if that is the only thing that is needed. I have been parsing the same games multiple times but ended up with the same 10 games that can't be found :) That's why I was trying to not parse again the ones already parsed

Clyde

Based on my new knowledge about covers in Skyscraper, I made a small script to hardlink all covers in media/covers to their ...-launching counterparts in images as Runcommand launching images (see the RetroPie Docs for more about that). edit: Hardlinks save space by being just additional directory entries for the same data blocks. Other than soft links, they remain valid after the original entries were deleted. If you want real copies, just change the ln command to cp.

I'll share it here for anyone who wants an easy way to use Skyscraper's covers as launching images. Everything behind a # is merely a comment.

system=$1                 # get the system to process (e.g. snes, psx, …)
cd /home/pi/RetroPie/roms # go to the roms directory
mkdir -p $system/images   # create "images" subdir in system dir if not already present
for f in $system/media/covers/* # process all cover images
  do 
    # get the filename without its path
    fb=$(basename "$f")
    # hardlink cover.ext to cover-launching.ext
    ln "$f" $system/images/"${fb%.*}-launching.${fb##*.}"  
    # explanation of the bash-fu:
    # ${fb%.*} = gets the filename without the last extension
    # ${fb##*.} = gets only the extension
  done
cd - > /dev/null  # return to the original directory quietly

Ideally, put the script in /home/pi/bin where it will be found automatically from any other directory and give it a meaningful name, e.g. covers2images. Make it executable with the command chmod u+x /home/pi/bin/covers2images. This way, you can invoke it from any other directory by that name followed by the system to process, e.g. covers2images arcade. It will return to that directory after completion.

The same script in one line for one-time use without a script file:

system=arcade; cd /home/pi/RetroPie/roms; mkdir -p $system/images;for f in $system/media/covers/*; do fb=$(basename "$f"); ln "$f" $system/images/"${fb%.*}-launching.${fb##*.}"; done; cd - > /dev/null

The one-liner will also work from any other dir and return to it afterwards. The only difference is that you'll have to change the system's name after system= manually before running it.

Sharing is caring. ❤️ That said, I'm waiting now for someone who tells me that this could be done much more easily in a different way, or that it is unnecessary for reasons™ alltogether. 😉 Anyway, it was a nice exercise to hone my bash-fu, and although I tested it thoroughly, I welcome any bug report or suggestions to improve it.

edit: launching, not loading 🥴

quicksilver

Im having a strange issue where some of my scraped DOS games are suddenly no longer scraped. I'll randomly find games that I had previously scraped successfully missing, play count, rating etc still is there but all media for that game is missing. When I try to regenerate the gameslist it's like the media files are missing from the cache too. I can't figure out what I'm doing thats causing this as I'm not touching these files at all once they are scraped.

Edit: so i just tried rescraping my "pc" folder and many games that were already scraped and in the cache are getting "game not found" messages.

muldjord

@quicksilver I'd need a specific example where this happens in order to look into it. The pc platform is not special in any way that would cause this as far as I remember.

EDIT: Just tested it with a random PC game and it does not have these issues on my end.

quicksilver

@muldjord yes I figured the issue was too vague. I haven't noticed any pattern to it. I'll keep an eye on it and see if I can provide any additional details.

peligwe

Love Skyscraper and love the fact I can use it to fill in the missing. It works great with theme I am using but I am having the hardest time trying to only use the 'cover' resource in game box view.

I'v tried , produces a blank game box

And, produces no image at all

Not exactly sure what I am doing wrong.

muldjord

@peligwe Try this:

  <output type="screenshot" width="640" height="480">
    <layer resource="cover" height="480" align="center">
      <gamebox side="wheel" rotate="90"/>
    </layer>
  </output>

That will export a screenshot at 640x480 and composite it of a layer that contains the gamebox centered with a height of 480 pixels with the cover artwork in the front (because that what the layer contains).

Your own second examples should also work, but will give you some weird results since you don't tell it what dimensions to work with. This will make it use the dimensions from the screenshot resource from the cache, which will vary between games. Same goes for the cover layer. If you don't give it any dimensions, it will use the covers' original dimensions which also vary between games. Always provide dimensions for either width or height to gain control of this.

For more information about how the artwork exports work, read this. I've just updated the topmost paragraphs to include some information that might help you understand it better.

quicksilver

Having an issue where some of my PC/DOS games are getting double bracket info. For example when I scrape "Oregon Trail II (1995).conf" from screenscraper it will show up in emulation station as "Oregon Trail II (1995)(1995)" or "Star Wars TIE Fighter (Collectors CD-ROM) (1995).conf" becomes "Star Wars TIE Fighter (Collectors CD-ROM) (1995)(Collectors CD-ROM) (1995)". The problem is if I remove the info in the brackets before scraping then the title is not found by screenscraper because I am assuming the name isnt a close enough match.

muldjord

@quicksilver Just tried reproducing this by creating an empty Oregon Trail II (1995).conf, scraped it with ScreenScraper and created a game list with it. I don't have double (1995) in the title in the created gamelist.xml. Could you please paste your config.ini (remember to remove any userCreds lines you might have in it)? And please also provide the command you run in case you have any command-line options set.

EDIT: You are running the latest Skyscraper right? I seem to recall I fixed this ages ago.

EDIT2: Yes, this issue was fixed all the way back in 3.2.3 on August 11th 2019. So this behavior does seem a bit odd to me unless you have something configured that makes it reappear. If you are running a newer version of Skyscraper I would be interested in figuring out what makes this happen, so I can fix this instance as well.

peligwe

@muldjord The worked out great but let's say I want the game box to fill the whole area. How much space do i have to work with vertically?

quicksilver

@muldjord

# --------------------------------------------------------------------
# Skyscraper by Lars Muldjord (https://github.com/muldjord/skyscraper)
# --------------------------------------------------------------------

# This is an example config file for use with Skyscraper. Use it as a template for creating
#   your own. If you copy this file to 'config.ini' it will be used per default.
#   You can always copy config.ini.example to a filename of your choice and make Skyscraper
#   use it with the '-c' command line option.

# You can have several platform sections in one config file. Further down is an example of
#   a generic platform setup. Check '--help' for more info on supported platforms.

# You can have several scraping module sections in one config file. Further down is an example
#   of a generic module setup. Check '--help' for more info on supported modules.

# Remember that most of these settings can also be set as command line options. Doing so
#   will overrule the options seen here except for the scraping module configs which
#   overrules everything else.
# Platform specific configs overrules main configs.

# Uncomment the sections and lines you wish to use:

[main]
##Setting input, gamelist or media folder here, will automatically append '/[platform]' to path.
##If you need better control, use the same variables under a [<platform>] section instead.
#inputFolder="/home/pi/RetroPie/roms"
#gamelistFolder="/home/pi/RetroPie/roms"
#mediaFolder="/home/pi/RetroPie/roms"
#cacheFolder="/home/pi/.skyscraper/cache"
#cacheResize="false"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import"
#unpack="false"
#frontend="emulationstation"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#maxLength="10000"
#threads="2"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
#maxFails="30"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="us,eu,ss,uk,wor,jp"
#minMatch="0"
#artworkXml="artwork.xml"
#relativePaths="false"
#addExtensions="*.zst"
#hints="false"
#subdirs="true"
#spaceCheck="true"
#scummIni="/full/path/to/scummvm.ini"

[psp]
artworkXml="artwork-psp.xml"

[vectrex]
artworkXml="artwork-vectrex.xml"

[ngpc]
artworkXml="artwork-ngpc.xml"

[arcade]
#inputFolder="/home/pi/RetroPie/roms/amiga"
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#cacheFolder="/home/pi/.skyscraper/cache/amiga"
#cacheResize="false"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#importFolder="/home/pi/.skyscraper/import/amiga"
#unpack="false"
#emulator=""
#launch=""
#videos="false"
#videoSizeLimit="42"
#symlink="false"
#brackets="true"
#lang="en"
#region="wor"
#langPrios="en,de,es"
#regionPrios="eu,us,ss,uk,wor,jp"
#minMatch="0"
#maxLength="10000"
#threads="2"
#startAt="filename"
#endAt="filename"
#pretend="false"
#unattend="false"
#unattendSkip="false"
#interactive="false"
#forceFilename="false"
#verbosity="1"
#skipped="false"
artworkXml="artwork-arcade.xml"
#relativePaths="false"
#extensions="*.zip *.uae *.adf"
#addExtensions="*.zst *.rom"
#subdirs="true"

#[<FRONTEND, eg 'pegasus'>]
#artworkXml="artwork.xml"
#emulator=""
#launch=""
#gamelistFolder="/home/pi/RetroPie/roms/amiga"
#mediaFolder="/home/pi/RetroPie/roms/amiga/media"
#skipped="false"
#brackets="true"
#videos="false"
#symlink="false"
#startAt="filename"
#endAt="filename"
#unattend="false"
#unattendSkip="false"
#forceFilename="false"
#verbosity="1"
#maxLength="10000"

[screenscraper]
userCreds=":"
#threads="1"
#minMatch="0"
#maxLength="10000"
#interactive="false"
#cacheCovers="true"
#cacheScreenshots="true"
#cacheWheels="true"
#cacheMarquees="true"
#videos="false"
#videoSizeLimit="42"

Im running skyscraper directly through retropie-setup, latest version (3.5.5)

peligwe

This post is deleted!

peligwe

@muldjord Never mind. Figured it out. Played around with different heights to get the desired effect I needed. However the gamebox view for N64 is cutoff on the edges. Will play around with different width to see if I can fix.

muldjord

Hi guys, I'm currently working on implementing a full Pegasus game list parser into Skyscraper which will allow it to retain user-information from old game lists. This will bring the support for the Pegasus frontend up to speed with the other supported frontends.

When that parser is done and released with the next release of Skyscraper, I will probably leave the project for a while. I want to focus on other projects again (non-coding related, I'm also a musician) and Skyscraper is simply craving too much of my attention. I've noticed that my replies to certain users on Github have become somewhat aggravated and arrogant due to lack of motivation and personal pressure in general, and that tells me it's time to put the tools down and go do something else.

I'm not leaving the community - certainly not. My interest in retro-gaming and RetroPie is a strong as ever. But I've reached a point where Skyscraper does everything I want it to do. And the changes I, personally, want to work on, are so significant that they will need to be moved to a potential Skyscraper 4.x. And my motivation to start work on that is not on the drawing-board for me at the moment.

I will probably be keeping an eye on any scraper-breaking bugs and fix those for the current 3.x version. And I will probably still provide support here and there when I feel like it. And I know you good peeps also do your bit to help out other Skyscraper users (which I am incredibly thankful for!).

Thank you all for using and supporting Skyscraper. I appreciate you all!

quicksilver

@muldjord skyscraper is a awesome tool. Thank you so much for all your hard work and support. You certainly deserve a break!

seriema

@muldjord Thank you for the greatest scraper ever! It has inspired me to so many ideas. Including one I've been meaning to ask you about for some time but wanted to get further in my own project first. I guess it's now or never :) Check it out: https://retropie.org.uk/forum/topic/26709/could-scraper-data-be-cached-online

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

I've noticed that my replies to certain users on Github have become somewhat aggravated and arrogant due to lack of motivation and personal pressure in general, and that tells me it's time to put the tools down and go do something else.

Kudos for realising that by yourself and pulling in the reins.

I will probably be keeping an eye on any scraper-breaking bugs and fix those for the current 3.x version. And I will probably still provide support here and there when I feel like it.

Thanks for that, you lessen my almost immediate concern what will happen if a scraping source changes breaks Skyscraper by changing their API.

Thank you all for using and supporting Skyscraper. I appreciate you all!

Thank you for creating and maintaining this awesome tool!

gold64

@muldjord Thanks for all your work on this, enjoy your other interests!