Versatile C++ game scraper: Skyscraper

muldjord

@muldjord Another thing, do you adhere to the folder layout as specified here?

Clyde

$ tree /home/pi/RetroPie/roms/daphne/
/home/pi/RetroPie/roms/daphne/
├── ace.daphne
│   ├── ace.commands
│   ├── ace.dat
│   ├── ace.m2v
│   ├── ace.ogg
│   └── ace.txt
├── daphne_log.txt
├── dapinput.ini
├── dapinput.ini.backup
├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt
├── gamelist.xml
├── lair2.daphne
│   ├── dl2-00001.dat
│   ├── dl2-00001.m2v
│   ├── dl2-00001.ogg
[... many more dat/m2v/ogg files ...]
│   ├── dl2-44550.m2v
│   ├── dl2-44550.ogg
│   ├── dl2-framefile.txt
│   └── lair2.txt
├── lair.daphne
│   ├── lair.commands
│   ├── lair.dat
│   ├── lair.m2v
│   ├── lair.ogg
│   └── lair.txt
├── media
│   ├── covers
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── marquees
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── screenshots
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── videos
│   │   ├── ace.mp4
│   │   ├── dle21.mp4
│   │   ├── lair2.mp4
│   │   └── lair.mp4
│   └── wheels
├── README.txt
└── roms
    ├── ace.zip
    ├── dle21.zip
    ├── lair2.zip
    └── lair.zip

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

By SS do you mean Skyscraper or Screenscraper in this case? :D

Skyscraper :)

Clyde

@clyde said in Versatile C++ game scraper: Skyscraper:

├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt

There is a m2v missing ~~that I plan to hard link to~~ because my dle21.txt points to ../lair.daphne/lair.m2v (edit: as I just remembered :*) ), but that shouldn't concern my problem, since none of the four games are put correctly in the gamelist.xml by Skyscraper. All have the <folder> element instead of <game>.

muldjord

@clyde Ok, I think that's enough for me to go on, thanks. I might look into it the coming days.

muldjord

@Clyde I've tested it today with lair and lair2 and it works just fine for me, showing the screenshot and data when moving the selection over lair.daphne folder in ES and starting the game when selecting the folder. It should be a <path> in the gamelist.xml since that is what initializes the game. At least on my default daphne installation.

Perhaps something is messed up with your ace files, I can't really know since I don't have that game. But I don't get any ES errors for lair and lair2.

Clyde

@muldjord Here's the gamelist my Skyscraper 3.6.1 is generating for the daphne system:

<?xml version="1.0"?>
<gameList>
  <folder>
    <path>./ace.daphne</path>
    <name>Ace</name>
    <thumbnail>./media/covers/ace.png</thumbnail>
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video>./media/videos/ace.mp4</video>
    <rating>0.1</rating>
    <desc>An old air battle game that features dogfights between 2 biplanes.</desc>
    <releasedate>19760101T000000</releasedate>
    <developer />
    <publisher>Allied Leisure</publisher>
    <genre />
    <players>2</players>
  </folder>
  <folder>
    <path>./lair.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/lair.png</thumbnail>
    <image>./media/screenshots/lair.png</image>
    <marquee>./media/marquees/lair.png</marquee>
    <video>./media/videos/lair.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./dle21.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/dle21.png</thumbnail>
    <image>./media/screenshots/dle21.png</image>
    <marquee>./media/marquees/dle21.png</marquee>
    <video>./media/videos/dle21.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./lair2.daphne</path>
    <name>Dragon&apos;s Lair 2 - Time Warp</name>
    <thumbnail>./media/covers/lair2.png</thumbnail>
    <image>./media/screenshots/lair2.png</image>
    <marquee>./media/marquees/lair2.png</marquee>
    <video>./media/videos/lair2.mp4</video>
    <rating>0.9</rating>
    <desc>Princess Daphne has been spirited away to a wrinkle in time by the Evil Wizard Mordroc who plans to force her into marriage. Only you, Dirk the Daring, can save her. Transported by a bumbling old time machine, you begin the rescue mission. But you must hurry, for once the Casket of Doom has opened, Mordroc will place the Death Ring upon Daphne&apos;s finger in marriage and she will be lost forever in the Time Warp!</desc>
    <releasedate>19910616T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Leland Corporation</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>1</players>
  </folder>
</gameList>

The <path> element is not the problem, but the <folder> where there should be <game>, and it occurs for all of my four daphne games ace, dle21, lair and lair2.

Would the complete output of Skyscraper help you? If so, could you please tell me the best way to generate it?

As for ace, could you maybe copy your lair directory and then rename it and everything in it to mirror an ace installation to see if SS would scrape the MAME game Ace instead of Space Ace? (As long as SS won't recognise the files' checksums and ignore their naming.)

Just some suggestions to dive deeper into those two issues that you may sacrifice your time for or not at your discretion.

muldjord

@clyde Sorry, I misspoke, it should be <folder> since it is a folder. Why do you want it to be a <game>? I don't quite understand (maybe you know something I don't).

It works well on my system, simply showing the game name while hovering over the folder and starting the game when selecting it.

EDIT: Your gamelist.xml looks fine to me. I can't really figure out what might cause your issue or what the issue is, sorry.

EDIT2: I'll try and test it with Space Ace. Stay tuned.

muldjord

@clyde Got my hands on Space Ace and tested it. Scraped it with Skyscraper using the -s screenscraper source. Found it perfectly with title and artwork. Shows up perfectly in ES and launches and plays just fine. Here's my gamelist entry for Space Ace:

  <folder>
    <path>./ace.daphne</path>
    <name>Space Ace</name>
    <thumbnail />
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video />
    <rating>0.9</rating>
    <desc>Space Ace is a laserdisc video game produced by Don Bluth Studios, Cinematronics and Advanced Microcomputer Systems (later renamed RDI Video Systems). It was unveiled in October 1983, just four months after the Dragon&apos;s Lair game, then released in Spring 1984, and like its predecessor featured film-quality animation played back from a laserdisc.

The gameplay is also similar, requiring the player </desc>
    <releasedate>19840424T000000</releasedate>
    <developer>Advanced Microcomputer Systems</developer>
    <publisher>Cinematronics</publisher>
    <genre>Plateform / Run Jump, Platform, Action</genre>
    <players>1</players>
    <kidgame>true</kidgame>
  </folder>

This works as expected on my setup, so I can't reproduce your error it seems.

EDIT: I do notice some odd behaviour from ES though. If they are present as <folder> in gamelist.xml ES suddenly adds it as a <game> as well when it writes the gamelist.xml back to disk when quitting ES. So it suddenly has both a <folder> entry for it and a <game> entry.

I think this is an ES bug. It shouldn't add that by itself. Perhaps it gets confused since the folder has a . in it (eg ace.daphne).

ES doesn't do this for psx games that are also folders (eg. 3D Lemmings (USA)).

EDIT2: Yeah, I think we've found some sort of bug in ES. When I add a . to one of the folders in one of my psx games (eg. 3d.lemmings) ES suddenly also adds that game as a <game> when it quits. This should probably not happen.

EDIT3: Maybe you're right, maybe it simply should be a <game> entry. Maybe I am misunderstanding the use of it. I juse assumed, since it is a folder, that obviously it should be a <folder> entry.

EDIT4: The behaviour seems to be described here and might be intentional. I can't figure out if I should change it to <game> though... Probably should. Comments?

Clyde

@muldjord Wow, thanks for all the diagnostic work. What's strange is that my ES doesn't recognise <folder> entries at all and throws the errors I quoted earlier, while it does work for you. I have to rename them to <game> to see them in ES.

So, subject to the different behaviour of your ES and mine regarding <folder> elements, I would propose to write <game> entries in every case for maximum compatibility as long as there are no discernable drawbacks to is.

edit: Also strange is that SS scrapes your Space Ace correctly. Can you tell if it recognises the checksum or does it just use the name? Either way, I'll test this a little more on my side.

muldjord

@clyde Yeah, I think I agree on changing it to <game> for the daphne platform. But your ES should support <folder> unless they've removed it from the dev version of ES or something. I'm just using the regular version I think.

EDIT: I'm guessing it uses the checksum of the zip file inside the daphne/roms folder (ScreenScraper uses the checksum 99% of the time). That's how I've hardcoded it, which is why the daphne platform took some extra work to implement.

EDIT2: Just installed emulationstation-dev instead. It works well with the <folder> entries too.

Clyde

@muldjord I'm using the normal version of ES from the regular RetroPie updates.

$ emulationstation --help
EmulationStation, a graphical front-end for ROM browsing.
Written by Alec "Aloshi" Lofquist.
Version 2.9.6rp, built Nov 14 2020 - 15:29:30

Do you have a newer version? I'm updating mine via SSH right now, but I can't test it before I get home in 5-6 hours.

muldjord

@clyde I run the latest binary releases of ES. But it's been working ever since the Skyscraper version I posted about a few posts up (and thereby the ES that was newest at that time) since I tested it working back then.

Clyde

@muldjord I just updated ES on both my PC running Kubuntu Linux (ES built from source) and my Pi 4 running stock RetroPie 4.7 (ES pre-built binary). Both are v2.9.6rp now, but both don't display the Daphne system with <folder> elements in the gamelist.

So I'm puzzled why your ES works with <folder> entries while mine doesn't. Could you check your version with emulationstation --help?

Ah, do you happen to have „Parse Gamelists Only“ disabled by any chance? Without it, I do see the Daphne system and its games despite the <folder> entries, and I can also start the games successfully. With this option set to ON (my normal setting) however, the Daphne system disappears.

muldjord

@clyde You are correct. If I set "Parse gamelists only" to "On" (I had it on "Off") it starts giving me those errors and only show the <game> entries it seems.

EDIT: I feel like we're looking at some sort of ES bug here. Reading the description of "Parse gamelists only" here (search for it) it seems to suggest that turning it "On" will only show games it already knows about - which makes sense. But <folder> is officially supported, but then suddenly ignored. I don't think that's how it's supposed to work.

EDIT2: Or maybe it is. It does somewhat make sense that a <folder> should only be shown if there's also a <game> entry inside of it. Otherwise it's just an empty folder... So yeah, it points back to the conclusion that Skyscraper should probably add them as <game>entries for Daphne instead of <folder>.

EDIT3: This change is now on master and will be in Skyscraper 3.6.5.

Clyde

@muldjord Funnily, I witnessed your first edit of the post yesterday and decided to go to bed and let you finish your thoughts. Looks like I was the right decision. 😀

Thanks for your ongoing exploration of this effect. One more thing about folders: They can also be used to create subsets of roms in the same ES system, where you can move down into them, seeing the roms inside, and move back up like within the systems themselves.

That said, I found that Skyscraper wouldn't create the Gamelist for roms in subdirectories when it's invoked from the RetroPie setup menu. It just freezes after "Sit back, relax and let me do the work! :)" and even Ctrl-c can't stop it – only the yellow text "User wants to quit…" appears, but it doesn't return to the menu. In the command line however, it works.

muldjord

Skyscraper 3.6.5 out now: https://github.com/muldjord/skyscraper

Now uses one shared networking instance, instead of one per thread
'daphne' platform entries now added as '<game>' entries instead of '<folder>' entries (Thank you to user 'clyde' for helping figure this out)
Added 'nocropblack' cli flag and 'cropBlack="BOOL"' config.ini option that allows user to disable cropping of black borders around screenshots when compositing the final gamelist artwork (Thank you to user 'WindyWinston' for suggesting this)

The major change this time around is fixing a longtime itch I've had with my network connection implementation. Instead of using an entire network manager object for each thread, I now only use one in total, which should optimize memory consumption. Users probably won't notice the difference, it was mainly a thing I wanted to do as a coder since the old way "was ugly".

Some minor stuff also made it in. Check the notes!

Happy scraping!

Clyde

@muldjord Thanks for the credits and for your ongoing work on Skyscraper. 👍

One question about the networking change – this won't change anything on the side of the online databases, or will it? E.g. will the server only "see" one connection now, and may limit it to its per-connection maximum, whereas before, it may have seen multiple connections, and limit each of them separately? Sorry if the answer is common knowledge for network coders, which I'm totally not. 😇

muldjord

@clyde It has no impact on any of the things you mention. The only difference is how much memory Skyscraper requires to run. Before I had one network manager for each thread, which is really silly to have as a network manager in Qt can easily handle all of the required connections with one instance.
Now I have one network manager in total, that simply distributes the connections to the threads when they request them.

So the number of connections to the online databases remain the same. :) There should be no scraping speed decrease due to this change, except for a nano-second here and there if two threads try to communicate with the network manager at the same time (in which case they will stand in line for a nano-second awaiting its turn).

Clyde

@muldjord A nanosecond?! Oh, noooooo! 😮 😉

Seriously, thanks for the explanation!