Versatile C++ game scraper: Skyscraper

muldjord

@Clyde Yeah, they seem to be hit hard this time. I hope they aren't about to shut down... :S

EDIT: Ok, just visited your link. So they are working on it. Good to know. :) Thanks.

Clyde

Hi,
I just noticed that Skyscraper doesn't seem to generate a valid gamelist.xml for Daphne, at least not from the file structure for Daphne that's described in the Docs.

Since that structure requires directories instead of archive files, Skyscraper writes a <folder> element instead of <game>:

<?xml version="1.0"?>
<gameList>
  <folder>
    <path>./ace.daphne</path>
...

This makes Emulation Station ignore the game's entry:

Nov 19 23:26:24 lvl2: 	Parsing XML file "/home/pi/RetroPie/roms/daphne/gamelist.xml"...
Nov 19 23:26:24 lvl1: 	gameList: folder doesn't already exist, won't create
Nov 19 23:26:24 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/daphne/ace.daphne", skipping.

Can you confirm this? Should I open an issue on Github?

Thanks
Clyde

Clyde

And another question about Daphne:

SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

Thanks again
Clyde

muldjord

@clyde I spend quite a lot of time getting Daphne to work some time ago I think, but I can't remember how it works. I'll have to spend some time looking into that when I get the time.

EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

Clyde

@muldjord Of course, take your time. And please tell me if I can help you with that in any way (other than coding, which I am not proficient in).

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

Alas, I'm using version 3.6.1 already, and I have four roms in daphne/roms (ace.zip, dle21.zip, lair.zip, lair2.zip).

muldjord

@clyde said in Versatile C++ game scraper: Skyscraper:

And another question about Daphne:

SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

Thanks again
Clyde

By SS do you mean Skyscraper or Screenscraper in this case? :D

muldjord

@muldjord Another thing, do you adhere to the folder layout as specified here?

Clyde

@muldjord Yes, I do:

$ tree /home/pi/RetroPie/roms/daphne/
/home/pi/RetroPie/roms/daphne/
├── ace.daphne
│   ├── ace.commands
│   ├── ace.dat
│   ├── ace.m2v
│   ├── ace.ogg
│   └── ace.txt
├── daphne_log.txt
├── dapinput.ini
├── dapinput.ini.backup
├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt
├── gamelist.xml
├── lair2.daphne
│   ├── dl2-00001.dat
│   ├── dl2-00001.m2v
│   ├── dl2-00001.ogg
[... many more dat/m2v/ogg files ...]
│   ├── dl2-44550.m2v
│   ├── dl2-44550.ogg
│   ├── dl2-framefile.txt
│   └── lair2.txt
├── lair.daphne
│   ├── lair.commands
│   ├── lair.dat
│   ├── lair.m2v
│   ├── lair.ogg
│   └── lair.txt
├── media
│   ├── covers
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── marquees
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── screenshots
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── videos
│   │   ├── ace.mp4
│   │   ├── dle21.mp4
│   │   ├── lair2.mp4
│   │   └── lair.mp4
│   └── wheels
├── README.txt
└── roms
    ├── ace.zip
    ├── dle21.zip
    ├── lair2.zip
    └── lair.zip

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

By SS do you mean Skyscraper or Screenscraper in this case? :D

Skyscraper :)

Clyde

@clyde said in Versatile C++ game scraper: Skyscraper:

├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt

There is a m2v missing ~~that I plan to hard link to~~ because my dle21.txt points to ../lair.daphne/lair.m2v (edit: as I just remembered :*) ), but that shouldn't concern my problem, since none of the four games are put correctly in the gamelist.xml by Skyscraper. All have the <folder> element instead of <game>.

muldjord

@clyde Ok, I think that's enough for me to go on, thanks. I might look into it the coming days.

muldjord

@Clyde I've tested it today with lair and lair2 and it works just fine for me, showing the screenshot and data when moving the selection over lair.daphne folder in ES and starting the game when selecting the folder. It should be a <path> in the gamelist.xml since that is what initializes the game. At least on my default daphne installation.

Perhaps something is messed up with your ace files, I can't really know since I don't have that game. But I don't get any ES errors for lair and lair2.

Clyde

@muldjord Here's the gamelist my Skyscraper 3.6.1 is generating for the daphne system:

<?xml version="1.0"?>
<gameList>
  <folder>
    <path>./ace.daphne</path>
    <name>Ace</name>
    <thumbnail>./media/covers/ace.png</thumbnail>
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video>./media/videos/ace.mp4</video>
    <rating>0.1</rating>
    <desc>An old air battle game that features dogfights between 2 biplanes.</desc>
    <releasedate>19760101T000000</releasedate>
    <developer />
    <publisher>Allied Leisure</publisher>
    <genre />
    <players>2</players>
  </folder>
  <folder>
    <path>./lair.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/lair.png</thumbnail>
    <image>./media/screenshots/lair.png</image>
    <marquee>./media/marquees/lair.png</marquee>
    <video>./media/videos/lair.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./dle21.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/dle21.png</thumbnail>
    <image>./media/screenshots/dle21.png</image>
    <marquee>./media/marquees/dle21.png</marquee>
    <video>./media/videos/dle21.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./lair2.daphne</path>
    <name>Dragon&apos;s Lair 2 - Time Warp</name>
    <thumbnail>./media/covers/lair2.png</thumbnail>
    <image>./media/screenshots/lair2.png</image>
    <marquee>./media/marquees/lair2.png</marquee>
    <video>./media/videos/lair2.mp4</video>
    <rating>0.9</rating>
    <desc>Princess Daphne has been spirited away to a wrinkle in time by the Evil Wizard Mordroc who plans to force her into marriage. Only you, Dirk the Daring, can save her. Transported by a bumbling old time machine, you begin the rescue mission. But you must hurry, for once the Casket of Doom has opened, Mordroc will place the Death Ring upon Daphne&apos;s finger in marriage and she will be lost forever in the Time Warp!</desc>
    <releasedate>19910616T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Leland Corporation</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>1</players>
  </folder>
</gameList>

The <path> element is not the problem, but the <folder> where there should be <game>, and it occurs for all of my four daphne games ace, dle21, lair and lair2.

Would the complete output of Skyscraper help you? If so, could you please tell me the best way to generate it?

As for ace, could you maybe copy your lair directory and then rename it and everything in it to mirror an ace installation to see if SS would scrape the MAME game Ace instead of Space Ace? (As long as SS won't recognise the files' checksums and ignore their naming.)

Just some suggestions to dive deeper into those two issues that you may sacrifice your time for or not at your discretion.

muldjord

@clyde Sorry, I misspoke, it should be <folder> since it is a folder. Why do you want it to be a <game>? I don't quite understand (maybe you know something I don't).

It works well on my system, simply showing the game name while hovering over the folder and starting the game when selecting it.

EDIT: Your gamelist.xml looks fine to me. I can't really figure out what might cause your issue or what the issue is, sorry.

EDIT2: I'll try and test it with Space Ace. Stay tuned.

muldjord

@clyde Got my hands on Space Ace and tested it. Scraped it with Skyscraper using the -s screenscraper source. Found it perfectly with title and artwork. Shows up perfectly in ES and launches and plays just fine. Here's my gamelist entry for Space Ace:

  <folder>
    <path>./ace.daphne</path>
    <name>Space Ace</name>
    <thumbnail />
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video />
    <rating>0.9</rating>
    <desc>Space Ace is a laserdisc video game produced by Don Bluth Studios, Cinematronics and Advanced Microcomputer Systems (later renamed RDI Video Systems). It was unveiled in October 1983, just four months after the Dragon&apos;s Lair game, then released in Spring 1984, and like its predecessor featured film-quality animation played back from a laserdisc.

The gameplay is also similar, requiring the player </desc>
    <releasedate>19840424T000000</releasedate>
    <developer>Advanced Microcomputer Systems</developer>
    <publisher>Cinematronics</publisher>
    <genre>Plateform / Run Jump, Platform, Action</genre>
    <players>1</players>
    <kidgame>true</kidgame>
  </folder>

This works as expected on my setup, so I can't reproduce your error it seems.

EDIT: I do notice some odd behaviour from ES though. If they are present as <folder> in gamelist.xml ES suddenly adds it as a <game> as well when it writes the gamelist.xml back to disk when quitting ES. So it suddenly has both a <folder> entry for it and a <game> entry.

I think this is an ES bug. It shouldn't add that by itself. Perhaps it gets confused since the folder has a . in it (eg ace.daphne).

ES doesn't do this for psx games that are also folders (eg. 3D Lemmings (USA)).

EDIT2: Yeah, I think we've found some sort of bug in ES. When I add a . to one of the folders in one of my psx games (eg. 3d.lemmings) ES suddenly also adds that game as a <game> when it quits. This should probably not happen.

EDIT3: Maybe you're right, maybe it simply should be a <game> entry. Maybe I am misunderstanding the use of it. I juse assumed, since it is a folder, that obviously it should be a <folder> entry.

EDIT4: The behaviour seems to be described here and might be intentional. I can't figure out if I should change it to <game> though... Probably should. Comments?

Clyde

@muldjord Wow, thanks for all the diagnostic work. What's strange is that my ES doesn't recognise <folder> entries at all and throws the errors I quoted earlier, while it does work for you. I have to rename them to <game> to see them in ES.

So, subject to the different behaviour of your ES and mine regarding <folder> elements, I would propose to write <game> entries in every case for maximum compatibility as long as there are no discernable drawbacks to is.

edit: Also strange is that SS scrapes your Space Ace correctly. Can you tell if it recognises the checksum or does it just use the name? Either way, I'll test this a little more on my side.

muldjord

@clyde Yeah, I think I agree on changing it to <game> for the daphne platform. But your ES should support <folder> unless they've removed it from the dev version of ES or something. I'm just using the regular version I think.

EDIT: I'm guessing it uses the checksum of the zip file inside the daphne/roms folder (ScreenScraper uses the checksum 99% of the time). That's how I've hardcoded it, which is why the daphne platform took some extra work to implement.

EDIT2: Just installed emulationstation-dev instead. It works well with the <folder> entries too.

Clyde

@muldjord I'm using the normal version of ES from the regular RetroPie updates.

$ emulationstation --help
EmulationStation, a graphical front-end for ROM browsing.
Written by Alec "Aloshi" Lofquist.
Version 2.9.6rp, built Nov 14 2020 - 15:29:30

Do you have a newer version? I'm updating mine via SSH right now, but I can't test it before I get home in 5-6 hours.

muldjord

@clyde I run the latest binary releases of ES. But it's been working ever since the Skyscraper version I posted about a few posts up (and thereby the ES that was newest at that time) since I tested it working back then.