Versatile C++ game scraper: Skyscraper

muldjord

@AlCzervik And you did generate the gamelist for it after grabbing the data, right?

KN4THX

I am having an issue where screenscraper says states that anonymous users only get 1 thread and I am locked to that but Screenscraper shows my account as having 6 threads and my scrape total on Screenscaper's site continually climbs while I am scraping on my Pi. I edited the config file to use 6 threads and obviously my user:pass is correct or it wouldn't show up on the site. Has anyone had the same trouble?

AlCzervik

@muldjord Yep. Tried it a few times.

muldjord

@KN4THX said in Versatile C++ game scraper: Skyscraper:

I edited the config file to use 6 threads and obviously my user:pass is correct or it wouldn't show up on the site. Has anyone had the same trouble?

You don't need to set threads=6 it will do this automatically when you've added:

[screenscraper]
userCreds="USER:PASS"

Please verify that you have it entered exactly like that in /home/USER/.skyscraper/config.ini. Also please verify what it says when it starts scraping. Something like Starting scraping run on X files using Y threads. Does it say 1 threads or 6 threadsthere?

AlCzervik

@muldjord A few more points. Per the RetroPie Daphne emulator site: https://retropie.org.uk/docs/Daphne/ the file structure is a bit different. From EmulationStation I just see 1 game entitled Lair. In the file structure under Daphne, you have two folders: lair.daphne and roms. The lair.daphne folder has the content (videos, screenshots, etc), the roms folder has the lair.zip file. So could that be the cause of the issue?

muldjord

@AlCzervik Oh, so it's not a file called lair.daphne it's a folder, I didn't know that. I don't have any Daphne games, so it would be cool if we could figure this out.

Well, basically I'm guessing, if you enter the roms folder inside the daphne folder, you will probably see the artwork and so on, right? It scrapes the lair.zip file from that folder and that's what's added to the gamelist.

What file do you select to start a game?

AlCzervik

@muldjord I think you're right. I'm sure it's stored 1 level lower as that's where the lair.zip is located. The file I select to start the game is the "Lair" file that is shown via EmulationStation so I assume we need to map the artwork somehow to that top level.

AlCzervik

@muldjord this is what it looks like:

muldjord

@AlCzervik I've been reading up on this and this is actually a bit tricky. EmulationStation actually launches the game when you select a gamename.daphne folder. This is quite unusual. No other platform I am aware of does this. They all launch using files. Skyscraper and other scrapers don't scrape folders, they scrape files, meaning that the gamename.daphne folder is not scraped, which is why you don't have any game information and artwork on it. But, the roms/lair.zip files is scraped, but this file is hidden inside of EmulationStation it would seem.

So I'll need to figure out how to deal with that.

AlCzervik

@muldjord Gotcha. Yeah, Daphne is quite different but once operational it works like a charm. Let me know how I can help troubleshoot/test!

muldjord

@AlCzervik This is now working in 3.5.0 which has just been released. Just update Skyscraper and run Skyscraper -p daphne and it should work without having to rescrape the data. Remember to exit EmulationStation first.

muldjord

Skyscraper 3.5.0 released: https://github.com/muldjord/skyscraper

For 'screenscraper', 'thegamesdb', 'mobygames' and 'arcadedb' Skyscraper will now no longer fetch the artworks if user has requested it to remain uncached (Thank you to user 'herbymachine' for the request)
Added '--skipexistingmedia' option to allow for faster gamelist generation in cases where some artwork already exists in the gamelist media folder (Thank you to 'jacrify' for suggesting this)
Added '--flags' command-line option that collects certain flag-like options into one
Added deprecated warnings to all options that are now moved to the '--flags' option
Added '*.cue' extension to '3do' platform (Thank you to 'therealdealneil' for the suggestion)
Implemented a hack for the 'daphne' platform that will add the game info to the 'game.daphne' folders instead of the 'game.zip' files (Thank you to 'AlCzervik' for pointing this out)

Several new features made it into this one that have been a long time coming. For one, when scraping any platform and user has requested any particular artwork resource not to be cached, the artwork will no longer be fetched from the sources either. This saves bandwidth and will provide faster resource gathering. So if you use the --nocovers and so on options (which are now deprecated, read on below), this will be useful to you.
A somewhat similar feature is the option to now skip any artwork generation if the artwork already exists in the media output folder. This can be done with the flags --flags skipexistingTYPEs where TYPE is cover, screenshot and so on. Remember the sat the end, so it's --flags skipexistingcovers. So basically, if the only thing you've changed is the textual data and you want to regenerate the gamelist only with that, you can now skip some or all of the artwork compositing to save a lot of time.

So, this is 3.5.0, and a rather substantial change to the command-line options have also snuck in. I was getting increasingly annoyed with how bloated the Skyscraper --help output was getting. It shows way too many options that are often only used in niche cases. I wanted to do away with that. So all options that do not require a parameter, which means they are basically flags, have now been collected in the --flags FLAG1,FLAG2 option. To see all available flags simply type Skyscraper -p PLATFORM --flags help. I will probably make this a bit smarter in the coming releases. Specifically I want it to only show flags that are relevant in relation to what mode Skyscraper was started in. For instance, if -s MODULEisn't set, it shouldn't show the --flags forcefilename option in the flags help text as it is not relevant when -s MODULEis left out. So it will be a contextual flags help. For now though, it just shows all flags.

@mitu The old options are still there, but should be corrected in the retropie script sometime soon'ish, as I plan to remove them from Skyscraper entirely in a few months. I won't do so without warning of course, but for now they are clearly marked as Deprecated in the Skyscraper --help output. If you want me to I can look over the script myself and see if I can figure out how to migrate them to the new --flags unattend,forcefilename,... format. Let me know what you think.

AlCzervik

@muldjord Still not showing with v3.5.0. What can I do to help? When I kill EmulationStation I do see the following error (not sure if this is related at all):
lv10: Error finding/creating FileData for "/home/pie/RetroPie/roms/daphne/roms"

muldjord

@AlCzervik I just tested it on lair, works fine.

Please recheck that Skyscraper has been updated to 3.5.0 at the top when you run it. For a more thorough fix, first exit EmulationStation (Very important!) then try running:

$ Skyscraper -p daphne -s screenscraper --refresh --flags unattend
$ Skyscraper -p daphne --flags unattend

That should do it. For it to work you need the lair.daphne folder with the files inside. And there needs to be a roms/lair.zip file, just as it describes on the RetroPie daphne page you linked above.

AlCzervik

@muldjord I reconfirmed 3.5, ran your commands, and still no art. I then removed everything from the daphne folder, restarted and no game (as expected). I then re-added all the files to lair.daphne and the roms folders under daphne. So the daphne folder only had lair.daphne and roms and the roms folder contained lair.zip. Rebooted and Dragon's lair was operational. Played a few rounds to confirm. I then exited EmulationStation again, re-ran the two commands you list above and still nothing via EmulationStation. I once again see the following error after quitting EmulationStation now: lv10: Error finding/creating FileData for "/home/pie/RetroPie/roms/daphne/roms", skipping. I think that has to be part of the issue. Lair is operational so I know the folders are setup correctly. Let me know what screenshots you'd like.

muldjord

@AlCzervik Can you please paste the contents of /home/pi/RetroPie/roms/daphne/gamelist.xml here as a codeblock? I'm kind of thinking that maybe some other gamelist is overriding it. I just need to verify that it even exists and that the contents look as it is supposed to.

AlCzervik

@muldjord Here you go:

<?xml version="1.0"?>
<gameList>
<folder>
<path>/home/pi/RetroPie/roms/daphne/roms</path>
<name>Dragon's Lair</name>
<cover />
<image>/home/pi/RetroPie/roms/daphne/media/screenshots/lair.png</image>
<marquee>/home/pi/RetroPie/roms/daphne/media/marquees/lair.png</marquee>
<video />
<rating>0.8</rating>
<desc>Originally released in the arcades as a laserdisc game, Dragon's Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
<releasedate>19830601T000000</releasedate>
<developer>Sullivan Bluth Interactive</developer>
<publisher>Cinematronics</publisher>
<genre>Platform, Plateform / Run Jump, Action / Adventure</genre>
<players>2</players>
</folder>
</gameList>

muldjord

@AlCzervik Thanks, something is definitely wrong with that. <path>/home/pi/RetroPie/roms/daphne/roms</path> should be <path>/home/pi/RetroPie/roms/daphne/lair.daphne</path>. I find that a bit odd as it worked when I tested it here. I'll look into it.

muldjord

@AlCzervik 3.5.1 out now where this is fixed. It only happens when you only have a single rom for the daphne platform. Which is why it didn't happen on my setup where i have two. But it's fixed for instances where the is only one as well now.

Please update and run Skyscraper -p daphne.

mitu

Not quite, but progress. Here's what it generated in my gamelist:

  <folder>
    <path>/home/pi/RetroPie/roms/daphne/lair.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <cover />
    <image>/home/pi/RetroPie/roms/daphne/media/screenshots/lair.png</image>
    <marquee>/home/pi/RetroPie/roms/daphne/media/marquees/lair.png</marquee>
    <video />
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump, Action / Adventure</genre>
    <players>2</players>
  </folder>



<folder>
    <path>/home/pi/RetroPie/roms/daphne/astron.daphne</path>
    <name>Astron Belt</name>
    <cover />
    <image>/home/pi/RetroPie/roms/daphne/media/screenshots/astron.png</image>
    <marquee>/home/pi/RetroPie/roms/daphne/media/marquees/astron.png</marquee>
    <video />
    <rating>0.6</rating>
    <desc>In Astron Belt, you fly through the universe battling alien ships to make your way to fight the main Alien Battle Cruiser.  Along the way, you fly across alien planets, through tunnels, through trenches, and get involved in a few astro-dogfights with enemy space fighters</desc>
    <releasedate>19830710T000000</releasedate>
    <developer>Sega</developer>
    <publisher>Sega</publisher>
    <genre>Shoot&apos;em Up, Shooter, Shooter / Plane, 1st person</genre>
    <players>1</players>
  </folder>

Note that I have a single game (lair.daphne), and the roms sub-folder has multiple entries (for multiple games), which have been thoroughly scraped :). The gamelist has all the .zip scraped data for each zip found under roms, but with paths similar to the entry above.

Daphne is a weird system.