Versatile C++ game scraper: Skyscraper

Clyde

@muldjord Wow, thanks for all the diagnostic work. What's strange is that my ES doesn't recognise <folder> entries at all and throws the errors I quoted earlier, while it does work for you. I have to rename them to <game> to see them in ES.

So, subject to the different behaviour of your ES and mine regarding <folder> elements, I would propose to write <game> entries in every case for maximum compatibility as long as there are no discernable drawbacks to is.

edit: Also strange is that SS scrapes your Space Ace correctly. Can you tell if it recognises the checksum or does it just use the name? Either way, I'll test this a little more on my side.

muldjord

@clyde Yeah, I think I agree on changing it to <game> for the daphne platform. But your ES should support <folder> unless they've removed it from the dev version of ES or something. I'm just using the regular version I think.

EDIT: I'm guessing it uses the checksum of the zip file inside the daphne/roms folder (ScreenScraper uses the checksum 99% of the time). That's how I've hardcoded it, which is why the daphne platform took some extra work to implement.

EDIT2: Just installed emulationstation-dev instead. It works well with the <folder> entries too.

Clyde

@muldjord I'm using the normal version of ES from the regular RetroPie updates.

$ emulationstation --help
EmulationStation, a graphical front-end for ROM browsing.
Written by Alec "Aloshi" Lofquist.
Version 2.9.6rp, built Nov 14 2020 - 15:29:30

Do you have a newer version? I'm updating mine via SSH right now, but I can't test it before I get home in 5-6 hours.

muldjord

@clyde I run the latest binary releases of ES. But it's been working ever since the Skyscraper version I posted about a few posts up (and thereby the ES that was newest at that time) since I tested it working back then.

Clyde

@muldjord I just updated ES on both my PC running Kubuntu Linux (ES built from source) and my Pi 4 running stock RetroPie 4.7 (ES pre-built binary). Both are v2.9.6rp now, but both don't display the Daphne system with <folder> elements in the gamelist.

So I'm puzzled why your ES works with <folder> entries while mine doesn't. Could you check your version with emulationstation --help?

Ah, do you happen to have „Parse Gamelists Only“ disabled by any chance? Without it, I do see the Daphne system and its games despite the <folder> entries, and I can also start the games successfully. With this option set to ON (my normal setting) however, the Daphne system disappears.

muldjord

@clyde You are correct. If I set "Parse gamelists only" to "On" (I had it on "Off") it starts giving me those errors and only show the <game> entries it seems.

EDIT: I feel like we're looking at some sort of ES bug here. Reading the description of "Parse gamelists only" here (search for it) it seems to suggest that turning it "On" will only show games it already knows about - which makes sense. But <folder> is officially supported, but then suddenly ignored. I don't think that's how it's supposed to work.

EDIT2: Or maybe it is. It does somewhat make sense that a <folder> should only be shown if there's also a <game> entry inside of it. Otherwise it's just an empty folder... So yeah, it points back to the conclusion that Skyscraper should probably add them as <game>entries for Daphne instead of <folder>.

EDIT3: This change is now on master and will be in Skyscraper 3.6.5.

Clyde

@muldjord Funnily, I witnessed your first edit of the post yesterday and decided to go to bed and let you finish your thoughts. Looks like I was the right decision. 😀

Thanks for your ongoing exploration of this effect. One more thing about folders: They can also be used to create subsets of roms in the same ES system, where you can move down into them, seeing the roms inside, and move back up like within the systems themselves.

That said, I found that Skyscraper wouldn't create the Gamelist for roms in subdirectories when it's invoked from the RetroPie setup menu. It just freezes after "Sit back, relax and let me do the work! :)" and even Ctrl-c can't stop it – only the yellow text "User wants to quit…" appears, but it doesn't return to the menu. In the command line however, it works.

muldjord

Skyscraper 3.6.5 out now: https://github.com/muldjord/skyscraper

Now uses one shared networking instance, instead of one per thread
'daphne' platform entries now added as '<game>' entries instead of '<folder>' entries (Thank you to user 'clyde' for helping figure this out)
Added 'nocropblack' cli flag and 'cropBlack="BOOL"' config.ini option that allows user to disable cropping of black borders around screenshots when compositing the final gamelist artwork (Thank you to user 'WindyWinston' for suggesting this)

The major change this time around is fixing a longtime itch I've had with my network connection implementation. Instead of using an entire network manager object for each thread, I now only use one in total, which should optimize memory consumption. Users probably won't notice the difference, it was mainly a thing I wanted to do as a coder since the old way "was ugly".

Some minor stuff also made it in. Check the notes!

Happy scraping!

Clyde

@muldjord Thanks for the credits and for your ongoing work on Skyscraper. 👍

One question about the networking change – this won't change anything on the side of the online databases, or will it? E.g. will the server only "see" one connection now, and may limit it to its per-connection maximum, whereas before, it may have seen multiple connections, and limit each of them separately? Sorry if the answer is common knowledge for network coders, which I'm totally not. 😇

muldjord

@clyde It has no impact on any of the things you mention. The only difference is how much memory Skyscraper requires to run. Before I had one network manager for each thread, which is really silly to have as a network manager in Qt can easily handle all of the required connections with one instance.
Now I have one network manager in total, that simply distributes the connections to the threads when they request them.

So the number of connections to the online databases remain the same. :) There should be no scraping speed decrease due to this change, except for a nano-second here and there if two threads try to communicate with the network manager at the same time (in which case they will stand in line for a nano-second awaiting its turn).

Clyde

@muldjord A nanosecond?! Oh, noooooo! 😮 😉

Seriously, thanks for the explanation!

iKarith

Just want to say thanks muldjord for a really cool batch scraper! When it works, it works amazingly well! When it doesn't … I'm pretty sure it's partly user error on my part, and partly that I need to use a combination of more advanced features I glimpse are there to do what I want, but … not in the way I am trying to use them.

I have a little experience with technical documentation writing, so perhaps if I give you a couple of examples and what I've done, you can tell me how I should have solved these issues. I may also be able to contribute a helper script to do what I think might have helped in one or both of these cases…

The first is this:

~/RetroPie/roms/nes/Tetris (U) [!].7z
~/RetroPie/roms/nes/Tetris (Unl).7z

These both get picked up as Nintendo's Tetris. As you can probably guess, the second is Atari/Tengen Tetris. I looked up on screenscraper.fr that the expected name for the second is "Tetris - The Soviet Mind Game". All right, so I added this line to my aliasMap.csv:

Tetris (Unl);Tetris - The Soviet Mind Game

I removed metadata for both from EmulationStation to be sure what I did would have the correct result and re-scraped. And what I got was that Nintendo Tetris now had no data, and Tengen Tetris had the Nintendo data.

I found a version of the game online as "Tetris - The Soviet Mind Game (Unl) (U).zip" and so renamed mine to match with .7z extension and re-scraped. Worked. But my understanding is that the alias should have done that for me?

The other is Night Trap. I have both Sega CD and Sega CD 32x versions. Again, ScreenScraper has the latter named "Night Trap (32x)", so following conventions seen elsewhere, I've done this:

~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 1).bin
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 1).cue
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 2).bin
~/RetroPie/roms/segacd/Night Trap/Night Trap (USA) (Disc 2).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1) (Track 1).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 1) (Track 2).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2).cue
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2) (Track 1).bin
~/RetroPie/roms/segacd/Night Trap (32X) (USA)/Night Trap (32x) (USA) (Disc 2) (Track 2).bin

(AFAIK PicoDrive doesn't support zipping these things nor combining them into CHD files, so … I chose to use subdirectories for sanity.)

These are both recognized as the 1992 Sega CD release. (And in fact most of my 32x games are not recognized at all despite having entries on ScreenScraper.) The 1994 Sega CD 32X release is there, however, it's just not being used.

I … haven't solved this one, save for manually editing emulationstation metadata.

The helper script it seems to me might be useful is effectively skyscraper-helper --cache-data <path to rom> to specifically try to re-fetch metadata for a specific ROM either with no data or with incorrect data. I understand that interactive selection is possible, but not through the RetroPie setup script, and the interactive selection might have allowed me to choose which of two possible Tetrises I was looking at. I don't know if that would have worked on Night Trap 32x, but if I could have specified more data to help narrow down the search … as I said, all I need to do is tell screenscraper that I've got the 32x version and it comes right up.

I think this helper script I'd like is probably not real hard to write. Seems that most of the components are in Skyscraper's existing arguments and the rest could be synthesized by combining them, but I need some confirmation on how. Pass any args starting with - (possibly parsing --flags to append to the --flags we're passing), and take anything else (or after --) and treat it as a filename … and then --startat and --endat I think? That's unclear to me.

But I'd be happy to suggest clarification and write the script to interactively attempt to figure these things out if you can clarify.

muldjord

@ikarith This actually seems to be a problem on ScreenScrapers end.

But first, let me clear some things up. The aliasMap.csv file is only used by filename based search modules. ScreenScraper uses checksum hashes or exact filename searches and therefore makes no use of the aliasMap.csv at all. So that won't help you in this case.

If ScreenScraper gives you back a faulty game, it is most likely because the checksum of your file or the exact filename, is connected to a wrong game inside of their database. This is not a rare thing to happen.

My suggestion is to log on to their website and find the specific checksum or filename that matches your files, and see if they are incorrectly connected to the wrong versions of those games. You can then proceed to correct that on their website as well. When the correction has been approved, you can then proceed to rescrape the games.

retro81

Hi @muldjord, first of all thanks for developing this amazing script for retropie!

I have a problem when fetching covers from screenscraper.fr - if the game has a flyer (like Samurai Shodown for NeoGeo) then the box is composed using the game "flyer" instead of the "box2D" art, any ideas why this happens and how to correct it? I'm using the default artwork.xml

Thanks!

retro81

I actually found this was a feature implemented in 3.4.3, any way to fine-tune skyscraper to either use flyers or actual box art? I use the retropie script.

retro81

Hi @muldjord, would it be possible to include the game hard/support media download as an option?

I like the way our ScreenScraper.fr friends do their "recalbox-mixes" including the game support media, here's a sample: https://www.screenscraper.fr/medias/113/176930/mixrbv2(sp).png

muldjord

@retro81 Hi, no, the ScreenScraper specific media won't be supported as they are unique to ScreenScraper and Skyscraper supports many different scraping sources. I don't want to turn Skyscraper into a ScreenScraper scraper.

But you can create a very similar result using Skyscraper's artwork compositor. It's documented here by using the already supported media types and add some custom frames and masks.

EDIT: A quick note: I am not currently active in developing Skyscraper beyond fixing bugs.

muldjord

@retro81 In the current version it will fetch "flyer" for all arcade platforms including "neogeo". But I see that that might not be optimal. So I've removed "neogeo" from that list. From 3.6.7 it will use "box-2D" instead. Thanks for letting me know.

EDIT: 3.6.7 out now with this change included. Please update your Skyscraper version and rescrape the neogeo platform with cache refresh enabled (important!).

retro81

@muldjord thank you for both responses!

Rion

@muldjord Would you consider adding support for Batocera?