Versatile C++ game scraper: Skyscraper

Clyde

@muldjord I just updated ES on both my PC running Kubuntu Linux (ES built from source) and my Pi 4 running stock RetroPie 4.7 (ES pre-built binary). Both are v2.9.6rp now, but both don't display the Daphne system with <folder> elements in the gamelist.

So I'm puzzled why your ES works with <folder> entries while mine doesn't. Could you check your version with emulationstation --help?

Ah, do you happen to have „Parse Gamelists Only“ disabled by any chance? Without it, I do see the Daphne system and its games despite the <folder> entries, and I can also start the games successfully. With this option set to ON (my normal setting) however, the Daphne system disappears.

muldjord

@clyde You are correct. If I set "Parse gamelists only" to "On" (I had it on "Off") it starts giving me those errors and only show the <game> entries it seems.

EDIT: I feel like we're looking at some sort of ES bug here. Reading the description of "Parse gamelists only" here (search for it) it seems to suggest that turning it "On" will only show games it already knows about - which makes sense. But <folder> is officially supported, but then suddenly ignored. I don't think that's how it's supposed to work.

EDIT2: Or maybe it is. It does somewhat make sense that a <folder> should only be shown if there's also a <game> entry inside of it. Otherwise it's just an empty folder... So yeah, it points back to the conclusion that Skyscraper should probably add them as <game>entries for Daphne instead of <folder>.

EDIT3: This change is now on master and will be in Skyscraper 3.6.5.

Clyde

@muldjord Funnily, I witnessed your first edit of the post yesterday and decided to go to bed and let you finish your thoughts. Looks like I was the right decision. 😀

Thanks for your ongoing exploration of this effect. One more thing about folders: They can also be used to create subsets of roms in the same ES system, where you can move down into them, seeing the roms inside, and move back up like within the systems themselves.

That said, I found that Skyscraper wouldn't create the Gamelist for roms in subdirectories when it's invoked from the RetroPie setup menu. It just freezes after "Sit back, relax and let me do the work! :)" and even Ctrl-c can't stop it – only the yellow text "User wants to quit…" appears, but it doesn't return to the menu. In the command line however, it works.

muldjord

Skyscraper 3.6.5 out now: https://github.com/muldjord/skyscraper

Now uses one shared networking instance, instead of one per thread
'daphne' platform entries now added as '<game>' entries instead of '<folder>' entries (Thank you to user 'clyde' for helping figure this out)
Added 'nocropblack' cli flag and 'cropBlack="BOOL"' config.ini option that allows user to disable cropping of black borders around screenshots when compositing the final gamelist artwork (Thank you to user 'WindyWinston' for suggesting this)

The major change this time around is fixing a longtime itch I've had with my network connection implementation. Instead of using an entire network manager object for each thread, I now only use one in total, which should optimize memory consumption. Users probably won't notice the difference, it was mainly a thing I wanted to do as a coder since the old way "was ugly".

Some minor stuff also made it in. Check the notes!

Happy scraping!

Clyde

@muldjord Thanks for the credits and for your ongoing work on Skyscraper. 👍

One question about the networking change – this won't change anything on the side of the online databases, or will it? E.g. will the server only "see" one connection now, and may limit it to its per-connection maximum, whereas before, it may have seen multiple connections, and limit each of them separately? Sorry if the answer is common knowledge for network coders, which I'm totally not. 😇

muldjord

@clyde It has no impact on any of the things you mention. The only difference is how much memory Skyscraper requires to run. Before I had one network manager for each thread, which is really silly to have as a network manager in Qt can easily handle all of the required connections with one instance.
Now I have one network manager in total, that simply distributes the connections to the threads when they request them.

So the number of connections to the online databases remain the same. :) There should be no scraping speed decrease due to this change, except for a nano-second here and there if two threads try to communicate with the network manager at the same time (in which case they will stand in line for a nano-second awaiting its turn).

Clyde

@muldjord A nanosecond?! Oh, noooooo! 😮 😉

Seriously, thanks for the explanation!

iKarith

Just want to say thanks muldjord for a really cool batch scraper! When it works, it works amazingly well! When it doesn't … I'm pretty sure it's partly user error on my part, and partly that I need to use a combination of more advanced features I glimpse are there to do what I want, but … not in the way I am trying to use them.

I have a little experience with technical documentation writing, so perhaps if I give you a couple of examples and what I've done, you can tell me how I should have solved these issues. I may also be able to contribute a helper script to do what I think might have helped in one or both of these cases…

The first is this:

~/RetroPie/roms/nes/Tetris (U) [!].7z
~/RetroPie/roms/nes/Tetris (Unl).7z

These both get picked up as Nintendo's Tetris. As you can probably guess, the second is Atari/Tengen Tetris. I looked up on screenscraper.fr that the expected name for the second is "Tetris - The Soviet Mind Game". All right, so I added this line to my aliasMap.csv:

Tetris (Unl);Tetris - The Soviet Mind Game

I removed metadata for both from EmulationStation to be sure what I did would have the correct result and re-scraped. And what I got was that Nintendo Tetris now had no data, and Tengen Tetris had the Nintendo data.

I found a version of the game online as "Tetris - The Soviet Mind Game (Unl) (U).zip" and so renamed mine to match with .7z extension and re-scraped. Worked. But my understanding is that the alias should have done that for me?

The other is Night Trap. I have both Sega CD and Sega CD 32x versions. Again, ScreenScraper has the latter named "Night Trap (32x)", so following conventions seen elsewhere, I've done this:

~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 1).bin
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 1).cue
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 2).bin
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 2).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1) (Track 1).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1) (Track 2).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2) (Track 1).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2) (Track 2).bin

(AFAIK PicoDrive doesn't support zipping these things nor combining them into CHD files, so … I chose to use subdirectories for sanity.)

These are both recognized as the 1992 Sega CD release. (And in fact most of my 32x games are not recognized at all despite having entries on ScreenScraper.) The 1994 Sega CD 32X release is there, however, it's just not being used.

I … haven't solved this one, save for manually editing emulationstation metadata.

The helper script it seems to me might be useful is effectively skyscraper-helper --cache-data <path to rom> to specifically try to re-fetch metadata for a specific ROM either with no data or with incorrect data. I understand that interactive selection is possible, but not through the RetroPie setup script, and the interactive selection might have allowed me to choose which of two possible Tetrises I was looking at. I don't know if that would have worked on Night Trap 32x, but if I could have specified more data to help narrow down the search … as I said, all I need to do is tell screenscraper that I've got the 32x version and it comes right up.

I think this helper script I'd like is probably not real hard to write. Seems that most of the components are in Skyscraper's existing arguments and the rest could be synthesized by combining them, but I need some confirmation on how. Pass any args starting with - (possibly parsing --flags to append to the --flags we're passing), and take anything else (or after --) and treat it as a filename … and then --startat and --endat I think? That's unclear to me.

But I'd be happy to suggest clarification and write the script to interactively attempt to figure these things out if you can clarify.

muldjord

@ikarith This actually seems to be a problem on ScreenScrapers end.

But first, let me clear some things up. The aliasMap.csv file is only used by filename based search modules. ScreenScraper uses checksum hashes or exact filename searches and therefore makes no use of the aliasMap.csv at all. So that won't help you in this case.

If ScreenScraper gives you back a faulty game, it is most likely because the checksum of your file or the exact filename, is connected to a wrong game inside of their database. This is not a rare thing to happen.

My suggestion is to log on to their website and find the specific checksum or filename that matches your files, and see if they are incorrectly connected to the wrong versions of those games. You can then proceed to correct that on their website as well. When the correction has been approved, you can then proceed to rescrape the games.

retro81

Hi @muldjord, first of all thanks for developing this amazing script for retropie!

I have a problem when fetching covers from screenscraper.fr - if the game has a flyer (like Samurai Shodown for NeoGeo) then the box is composed using the game "flyer" instead of the "box2D" art, any ideas why this happens and how to correct it? I'm using the default artwork.xml

Thanks!

retro81

I actually found this was a feature implemented in 3.4.3, any way to fine-tune skyscraper to either use flyers or actual box art? I use the retropie script.

retro81

Hi @muldjord, would it be possible to include the game hard/support media download as an option?

I like the way our ScreenScraper.fr friends do their "recalbox-mixes" including the game support media, here's a sample: https://www.screenscraper.fr/medias/113/176930/mixrbv2(sp).png

muldjord

@retro81 Hi, no, the ScreenScraper specific media won't be supported as they are unique to ScreenScraper and Skyscraper supports many different scraping sources. I don't want to turn Skyscraper into a ScreenScraper scraper.

But you can create a very similar result using Skyscraper's artwork compositor. It's documented here by using the already supported media types and add some custom frames and masks.

EDIT: A quick note: I am not currently active in developing Skyscraper beyond fixing bugs.

muldjord

@retro81 In the current version it will fetch "flyer" for all arcade platforms including "neogeo". But I see that that might not be optimal. So I've removed "neogeo" from that list. From 3.6.7 it will use "box-2D" instead. Thanks for letting me know.

EDIT: 3.6.7 out now with this change included. Please update your Skyscraper version and rescrape the neogeo platform with cache refresh enabled (important!).

retro81

@muldjord thank you for both responses!

Rion

@muldjord Would you consider adding support for Batocera?

muldjord

@rion Just did a Google search and I assume you mean Batocera Linux, right? I won't personally support it, but I don't see why it wouldn't work already. As long as it has Qt5 included and a build chain, you should be able to install it using the description at the Skyscraper github. How to get Qt5 on there is up to the user to figure out. Maybe it's in the Batocera package manager already (if there is such a thing, I have never heard of the distribution before just now).

Rion

@muldjord You should really try it out. It's a great distribution both for sbc's but also for an x86_64 PC.

Skyscraper is not yet available in the Batocera pacman repository. But could probably be in the future.

What i wanted to know is if you would think/consider to implement support for more systems supported by Batocera. I'm completely fine if i could just use it on my Linux machine and scrape the roms there.

Cause i just love how the artwork looks with Skyscraper and there is no better alternative out there! 😃

https://wiki.batocera.org/emulators

muldjord

@rion If they are major platforms that are related to consoles or home computers I will consider it. But I do have a lot of requests to add obscure "homebrew" and or / sub-platforms that I do not wish to support as they are often either too obscure or already supported by "arcade" or others.

Mainly Skyscraper was made to support the RetroPie naming scheme and platforms. I have, however, added some platforms besides that that I felt made sense. I am quite restricting when it comes to adding more, as I don't want to bloat Skyscraper too much.

But if you have a couple of platforms that you feel could fall under the RetroPie umbrella, I'd need the same info as found on the RetroPie supported platform pages, which is:

Platform name and shortname (eg. "Nintendo Entertainment System" / "nes")
File extensions (*.this, *.that)

But no promises I'll add them. :)

Obi-Wan

Not sure where to post this, but I felt it should be mentioned somewhere.

Scraping does not work on Windows with version 3.6.7, but works fine with the previous version, 3.5.8.

I was so happy to find Skyscraper because it seemed like a really nice tool, so I was a bit bummed out about the lack of Windows-support and even more so when I couldn't get it to work. But the second I tried the older 3.5.8 version scraping worked immediately.

I get the same behavior on two separate Windows systems. This reproduces the behavior on both my systems:

Download Skyscraper and unzip to C:\work\
Copy contents of the deploy folder to my user-folder
Open Powershell in Skyscraper-folder (cmd has the same behavior)
In PS, run: .\Skyscraper.exe -s thegamesdb -p snes -i "C:/work/ROMS"

3.6.7 returns Game not found
3.5.8 works fine.

I have no idea if I've missed something else here, but with 3.6.7 I got nowhere and with 3.5.8 everything seems to work as intended.