Versatile C++ game scraper: Skyscraper

theshadowzero

@muldjord i've got a quick question about skyscraper, i've tried messing with the --cache settings but haven't been able to quite hit my use case. i scraped all my roms over a couple days with skyscraper from screenscraper, but i had videos disabled. now, i'd like to add videos, but i haven't been able to figure out how to get skyscraper to just scrape videos. because the roms already have metadata in the cache, skyscraper seems to want to either skip them entirely, or with a --cache refresh completely scrape them fresh. is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

ive tried running it with just --flags videos and it asks me if i want to skip already existing game list entries. saying yes obviously skips all the roms, and saying no will show that it finds the rom, show all the metadata and media i have (with a big red NO() for video), and then do nothing and proceed to the next rom. if i scrape with --cache refresh it finds the rom and the video and will download and include it. i was under the impression skyscraper would automatically add media that is missing but that doesn't seem to be the case, or more likely i am doing something wrong. guidance would be appreciated! i don't want to spend another 2-3 days hammering screenscraper.fr to get videos when i already have 90% of the scrapes i need

edit: i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless

muldjord

@theshadowzero said in Versatile C++ game scraper: Skyscraper:

is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

No, it will always scrape the metadata since it's "free". When a game is looked up, the metadata is already there in the data received from the server. So it would be pretty silly to not use it.
It's the media that might take some time to grab - especially the videos of course.

I would simply rescrape all of the games with videos enabled. That's the easiest for this use case. You will need --cache refresh for that to work.

i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless

Correct, you cannot use --cache edit for this use case. It can only edit metadata and add metadata. Not media.

muldjord

Skyscraper 3.5.9 released: https://github.com/muldjord/skyscraper

Implemented the new IGDB v4 authentication method. IGDB will now work again, and requires free credentials. Read more about that here
Improved memory consumption when handing entries back to main thread
Added '.au3' file extension to id script exception list (Thank you to 'o0alucard0o' for reporting this)

Most prominent feature is the updated IGDB v4 API authentication method. This was broken due to IGDB moving away from their old v3 API. This is now fixed and requires free personal credentials. Read more about that here.

Let me know if you run into issues.

Clyde

@muldjord Perfect timing as ScreenScraper is currently down! 😉

Seriously, thanks for still working on this awsome tool. 👍

muldjord

@Clyde Yeah, they seem to be hit hard this time. I hope they aren't about to shut down... :S

EDIT: Ok, just visited your link. So they are working on it. Good to know. :) Thanks.

Clyde

Hi,
I just noticed that Skyscraper doesn't seem to generate a valid gamelist.xml for Daphne, at least not from the file structure for Daphne that's described in the Docs.

Since that structure requires directories instead of archive files, Skyscraper writes a <folder> element instead of <game>:

<?xml version="1.0"?>
<gameList>
  <folder>
    <path>./ace.daphne</path>
...

This makes Emulation Station ignore the game's entry:

Nov 19 23:26:24 lvl2: 	Parsing XML file "/home/pi/RetroPie/roms/daphne/gamelist.xml"...
Nov 19 23:26:24 lvl1: 	gameList: folder doesn't already exist, won't create
Nov 19 23:26:24 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/daphne/ace.daphne", skipping.

Can you confirm this? Should I open an issue on Github?

Thanks
Clyde

Clyde

And another question about Daphne:

SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

Thanks again
Clyde

muldjord

@clyde I spend quite a lot of time getting Daphne to work some time ago I think, but I can't remember how it works. I'll have to spend some time looking into that when I get the time.

EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

Clyde

@muldjord Of course, take your time. And please tell me if I can help you with that in any way (other than coding, which I am not proficient in).

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

Alas, I'm using version 3.6.1 already, and I have four roms in daphne/roms (ace.zip, dle21.zip, lair.zip, lair2.zip).

muldjord

@clyde said in Versatile C++ game scraper: Skyscraper:

And another question about Daphne:

SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

Thanks again
Clyde

By SS do you mean Skyscraper or Screenscraper in this case? :D

muldjord

@muldjord Another thing, do you adhere to the folder layout as specified here?

Clyde

@muldjord Yes, I do:

$ tree /home/pi/RetroPie/roms/daphne/
/home/pi/RetroPie/roms/daphne/
├── ace.daphne
│   ├── ace.commands
│   ├── ace.dat
│   ├── ace.m2v
│   ├── ace.ogg
│   └── ace.txt
├── daphne_log.txt
├── dapinput.ini
├── dapinput.ini.backup
├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt
├── gamelist.xml
├── lair2.daphne
│   ├── dl2-00001.dat
│   ├── dl2-00001.m2v
│   ├── dl2-00001.ogg
[... many more dat/m2v/ogg files ...]
│   ├── dl2-44550.m2v
│   ├── dl2-44550.ogg
│   ├── dl2-framefile.txt
│   └── lair2.txt
├── lair.daphne
│   ├── lair.commands
│   ├── lair.dat
│   ├── lair.m2v
│   ├── lair.ogg
│   └── lair.txt
├── media
│   ├── covers
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── marquees
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── screenshots
│   │   ├── ace.png
│   │   ├── dle21.png
│   │   ├── lair2.png
│   │   └── lair.png
│   ├── videos
│   │   ├── ace.mp4
│   │   ├── dle21.mp4
│   │   ├── lair2.mp4
│   │   └── lair.mp4
│   └── wheels
├── README.txt
└── roms
    ├── ace.zip
    ├── dle21.zip
    ├── lair2.zip
    └── lair.zip

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

By SS do you mean Skyscraper or Screenscraper in this case? :D

Skyscraper :)

Clyde

@clyde said in Versatile C++ game scraper: Skyscraper:

├── dle21.daphne
│   ├── dle21.commands
│   ├── dle21.dat
│   ├── dle21.ogg
│   └── dle21.txt

There is a m2v missing ~~that I plan to hard link to~~ because my dle21.txt points to ../lair.daphne/lair.m2v (edit: as I just remembered :*) ), but that shouldn't concern my problem, since none of the four games are put correctly in the gamelist.xml by Skyscraper. All have the <folder> element instead of <game>.

muldjord

@clyde Ok, I think that's enough for me to go on, thanks. I might look into it the coming days.

muldjord

@Clyde I've tested it today with lair and lair2 and it works just fine for me, showing the screenshot and data when moving the selection over lair.daphne folder in ES and starting the game when selecting the folder. It should be a <path> in the gamelist.xml since that is what initializes the game. At least on my default daphne installation.

Perhaps something is messed up with your ace files, I can't really know since I don't have that game. But I don't get any ES errors for lair and lair2.

Clyde

@muldjord Here's the gamelist my Skyscraper 3.6.1 is generating for the daphne system:

<?xml version="1.0"?>
<gameList>
  <folder>
    <path>./ace.daphne</path>
    <name>Ace</name>
    <thumbnail>./media/covers/ace.png</thumbnail>
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video>./media/videos/ace.mp4</video>
    <rating>0.1</rating>
    <desc>An old air battle game that features dogfights between 2 biplanes.</desc>
    <releasedate>19760101T000000</releasedate>
    <developer />
    <publisher>Allied Leisure</publisher>
    <genre />
    <players>2</players>
  </folder>
  <folder>
    <path>./lair.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/lair.png</thumbnail>
    <image>./media/screenshots/lair.png</image>
    <marquee>./media/marquees/lair.png</marquee>
    <video>./media/videos/lair.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./dle21.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <thumbnail>./media/covers/dle21.png</thumbnail>
    <image>./media/screenshots/dle21.png</image>
    <marquee>./media/marquees/dle21.png</marquee>
    <video>./media/videos/dle21.mp4</video>
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>2</players>
  </folder>
  <folder>
    <path>./lair2.daphne</path>
    <name>Dragon&apos;s Lair 2 - Time Warp</name>
    <thumbnail>./media/covers/lair2.png</thumbnail>
    <image>./media/screenshots/lair2.png</image>
    <marquee>./media/marquees/lair2.png</marquee>
    <video>./media/videos/lair2.mp4</video>
    <rating>0.9</rating>
    <desc>Princess Daphne has been spirited away to a wrinkle in time by the Evil Wizard Mordroc who plans to force her into marriage. Only you, Dirk the Daring, can save her. Transported by a bumbling old time machine, you begin the rescue mission. But you must hurry, for once the Casket of Doom has opened, Mordroc will place the Death Ring upon Daphne&apos;s finger in marriage and she will be lost forever in the Time Warp!</desc>
    <releasedate>19910616T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Leland Corporation</publisher>
    <genre>Platform, Plateform / Run Jump</genre>
    <players>1</players>
  </folder>
</gameList>

The <path> element is not the problem, but the <folder> where there should be <game>, and it occurs for all of my four daphne games ace, dle21, lair and lair2.

Would the complete output of Skyscraper help you? If so, could you please tell me the best way to generate it?

As for ace, could you maybe copy your lair directory and then rename it and everything in it to mirror an ace installation to see if SS would scrape the MAME game Ace instead of Space Ace? (As long as SS won't recognise the files' checksums and ignore their naming.)

Just some suggestions to dive deeper into those two issues that you may sacrifice your time for or not at your discretion.

muldjord

@clyde Sorry, I misspoke, it should be <folder> since it is a folder. Why do you want it to be a <game>? I don't quite understand (maybe you know something I don't).

It works well on my system, simply showing the game name while hovering over the folder and starting the game when selecting it.

EDIT: Your gamelist.xml looks fine to me. I can't really figure out what might cause your issue or what the issue is, sorry.

EDIT2: I'll try and test it with Space Ace. Stay tuned.

muldjord

@clyde Got my hands on Space Ace and tested it. Scraped it with Skyscraper using the -s screenscraper source. Found it perfectly with title and artwork. Shows up perfectly in ES and launches and plays just fine. Here's my gamelist entry for Space Ace:

  <folder>
    <path>./ace.daphne</path>
    <name>Space Ace</name>
    <thumbnail />
    <image>./media/screenshots/ace.png</image>
    <marquee>./media/marquees/ace.png</marquee>
    <video />
    <rating>0.9</rating>
    <desc>Space Ace is a laserdisc video game produced by Don Bluth Studios, Cinematronics and Advanced Microcomputer Systems (later renamed RDI Video Systems). It was unveiled in October 1983, just four months after the Dragon&apos;s Lair game, then released in Spring 1984, and like its predecessor featured film-quality animation played back from a laserdisc.

The gameplay is also similar, requiring the player </desc>
    <releasedate>19840424T000000</releasedate>
    <developer>Advanced Microcomputer Systems</developer>
    <publisher>Cinematronics</publisher>
    <genre>Plateform / Run Jump, Platform, Action</genre>
    <players>1</players>
    <kidgame>true</kidgame>
  </folder>

This works as expected on my setup, so I can't reproduce your error it seems.

EDIT: I do notice some odd behaviour from ES though. If they are present as <folder> in gamelist.xml ES suddenly adds it as a <game> as well when it writes the gamelist.xml back to disk when quitting ES. So it suddenly has both a <folder> entry for it and a <game> entry.

I think this is an ES bug. It shouldn't add that by itself. Perhaps it gets confused since the folder has a . in it (eg ace.daphne).

ES doesn't do this for psx games that are also folders (eg. 3D Lemmings (USA)).

EDIT2: Yeah, I think we've found some sort of bug in ES. When I add a . to one of the folders in one of my psx games (eg. 3d.lemmings) ES suddenly also adds that game as a <game> when it quits. This should probably not happen.

EDIT3: Maybe you're right, maybe it simply should be a <game> entry. Maybe I am misunderstanding the use of it. I juse assumed, since it is a folder, that obviously it should be a <folder> entry.

EDIT4: The behaviour seems to be described here and might be intentional. I can't figure out if I should change it to <game> though... Probably should. Comments?