Versatile C++ game scraper: Skyscraper

AlCzervik

@muldjord Still not showing with v3.5.0. What can I do to help? When I kill EmulationStation I do see the following error (not sure if this is related at all):
lv10: Error finding/creating FileData for "/home/pie/RetroPie/roms/daphne/roms"

muldjord

@AlCzervik I just tested it on lair, works fine.

Please recheck that Skyscraper has been updated to 3.5.0 at the top when you run it. For a more thorough fix, first exit EmulationStation (Very important!) then try running:

$ Skyscraper -p daphne -s screenscraper --refresh --flags unattend
$ Skyscraper -p daphne --flags unattend

That should do it. For it to work you need the lair.daphne folder with the files inside. And there needs to be a roms/lair.zip file, just as it describes on the RetroPie daphne page you linked above.

AlCzervik

@muldjord I reconfirmed 3.5, ran your commands, and still no art. I then removed everything from the daphne folder, restarted and no game (as expected). I then re-added all the files to lair.daphne and the roms folders under daphne. So the daphne folder only had lair.daphne and roms and the roms folder contained lair.zip. Rebooted and Dragon's lair was operational. Played a few rounds to confirm. I then exited EmulationStation again, re-ran the two commands you list above and still nothing via EmulationStation. I once again see the following error after quitting EmulationStation now: lv10: Error finding/creating FileData for "/home/pie/RetroPie/roms/daphne/roms", skipping. I think that has to be part of the issue. Lair is operational so I know the folders are setup correctly. Let me know what screenshots you'd like.

muldjord

@AlCzervik Can you please paste the contents of /home/pi/RetroPie/roms/daphne/gamelist.xml here as a codeblock? I'm kind of thinking that maybe some other gamelist is overriding it. I just need to verify that it even exists and that the contents look as it is supposed to.

AlCzervik

@muldjord Here you go:

<?xml version="1.0"?>
<gameList>
<folder>
<path>/home/pi/RetroPie/roms/daphne/roms</path>
<name>Dragon's Lair</name>
<cover />
<image>/home/pi/RetroPie/roms/daphne/media/screenshots/lair.png</image>
<marquee>/home/pi/RetroPie/roms/daphne/media/marquees/lair.png</marquee>
<video />
<rating>0.8</rating>
<desc>Originally released in the arcades as a laserdisc game, Dragon's Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
<releasedate>19830601T000000</releasedate>
<developer>Sullivan Bluth Interactive</developer>
<publisher>Cinematronics</publisher>
<genre>Platform, Plateform / Run Jump, Action / Adventure</genre>
<players>2</players>
</folder>
</gameList>

muldjord

@AlCzervik Thanks, something is definitely wrong with that. <path>/home/pi/RetroPie/roms/daphne/roms</path> should be <path>/home/pi/RetroPie/roms/daphne/lair.daphne</path>. I find that a bit odd as it worked when I tested it here. I'll look into it.

muldjord

@AlCzervik 3.5.1 out now where this is fixed. It only happens when you only have a single rom for the daphne platform. Which is why it didn't happen on my setup where i have two. But it's fixed for instances where the is only one as well now.

Please update and run Skyscraper -p daphne.

mitu

Not quite, but progress. Here's what it generated in my gamelist:

  <folder>
    <path>/home/pi/RetroPie/roms/daphne/lair.daphne</path>
    <name>Dragon&apos;s Lair</name>
    <cover />
    <image>/home/pi/RetroPie/roms/daphne/media/screenshots/lair.png</image>
    <marquee>/home/pi/RetroPie/roms/daphne/media/marquees/lair.png</marquee>
    <video />
    <rating>0.8</rating>
    <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
    <releasedate>19830601T000000</releasedate>
    <developer>Sullivan Bluth Interactive</developer>
    <publisher>Cinematronics</publisher>
    <genre>Platform, Plateform / Run Jump, Action / Adventure</genre>
    <players>2</players>
  </folder>



<folder>
    <path>/home/pi/RetroPie/roms/daphne/astron.daphne</path>
    <name>Astron Belt</name>
    <cover />
    <image>/home/pi/RetroPie/roms/daphne/media/screenshots/astron.png</image>
    <marquee>/home/pi/RetroPie/roms/daphne/media/marquees/astron.png</marquee>
    <video />
    <rating>0.6</rating>
    <desc>In Astron Belt, you fly through the universe battling alien ships to make your way to fight the main Alien Battle Cruiser.  Along the way, you fly across alien planets, through tunnels, through trenches, and get involved in a few astro-dogfights with enemy space fighters</desc>
    <releasedate>19830710T000000</releasedate>
    <developer>Sega</developer>
    <publisher>Sega</publisher>
    <genre>Shoot&apos;em Up, Shooter, Shooter / Plane, 1st person</genre>
    <players>1</players>
  </folder>

Note that I have a single game (lair.daphne), and the roms sub-folder has multiple entries (for multiple games), which have been thoroughly scraped :). The gamelist has all the .zip scraped data for each zip found under roms, but with paths similar to the entry above.

Daphne is a weird system.

AlCzervik

@muldjord Worked! And it looks glorious !
F453C513-0BC8-417A-8A0A-E9CB15ED3656.jpeg

muldjord

@AlCzervik Awesome! Thanks for testing

@mitu Sure is. To my knowledge it is also the only system that launches from a folder and not a file. I played a bit of Dragon's Lair 2. Damn it's hard! But it works pretty well.

Did you see my mention in https://retropie.org.uk/forum/topic/11826/versatile-c-game-scraper-skyscraper/1420 about the (soon-to-be) deprecated flags that are now moved to --flags? I can give it a go updating it myself if you like.

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

Did you see my mention in https://retropie.org.uk/forum/topic/11826/versatile-c-game-scraper-skyscraper/1420 about the (soon-to-be) deprecated flags that are now moved to --flags? I can give it a go updating it myself if you like.

Yes, I'll have to update the script to make use of them. I wouldn't mind if you do it instead of me :), but it's no trouble.

muldjord

@mitu Great, I'll look into it at some point. But we should probably hold out on it until there's a reasonable chance that people have updated to the 3.5.x versions. Otherwise it won't work for pre-3.5.x users when the script is using the --flags options.

KN4THX

@muldjord Thank you, it seems to have been my fault but the result was strange. I failed to have the user"pass line uncommented properly so it was not detecting my thread count and said I was logged in as an anonymous user (which limited me to 1 thread).

What was weird was my scrape count kept going up in my SS account when monitoring it in real time. Not sure if that is any relevant info to you but everything works dandy now and you are much appreciated in every way. I thank you and hope you are staying safe.

muldjord

@KN4THX Hm, that makes little sense. If the entire userCreds="USER:PASS" line was commented I can't see how ScreenScraper would know it is you who is scraping. It has to have been entered somewhere, somehow. Maybe it wasn't entirely commented? I dunno. Glad you like the scraper and got it working.

muldjord

@mitu I've forked and updated the skyscraper.sh script to use the --flags FLAG1,FLAG2 format. I've also added version checks for at least 3.5 to the gathering and generating functions that will display a message in case the user needs to update. Lastly I've updated all the places where the old flags were mentioned in the help texts so they now show the new --flags versions.

Should I just create a pull request for the RetroPie-Setup project?

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

Should I just create a pull request for the RetroPie-Setup project?

Why not ? I'll take a look - just be careful with those tabs :).

muldjord

@mitu Yeah, I know. I think I got it right... Not 100% sure though. :D I'll make a pull request.

EDIT: Here it is: https://github.com/RetroPie/RetroPie-Setup/pull/3119

muldjord

Skyscraper 3.5.3 released: https://github.com/muldjord/skyscraper

MAJOR: Added video conversion support through config.ini option 'videoConvertCommand="COMMAND %i %o"' option. See docs for more details.
Removed deprecated '<cover>' node and replaced it with '<thumbnail>' for ES gamelist generation (Thank you to 'yisraeldov' for letting me know)
Added simple error handling for media network requests
'--flags help' and '--cache help' no longer requires '-p PLATFORM' to be set
Changed 'worldofspectrum' scraping module to use https

Most prominent new feature is the addition of video conversion options. The videos delivered by the scraping sources can vary quite a lot in codec and color format. The new videoConvertCommand="COMMAND %i %o" config.ini option allows you to funnel the videos through an external command, before saving them to the cache. This all happens on the fly.
I have added an example command in the config.ini.example file. Obviously, if you use ffmpeg as the example uses, you need to install it first. For RetroPie and other Debian-derived distros you can install it with sudo apt install ffmpeg. Read more about the possibilities here. The video conversion makes good use of the --verbosity N option. So be sure to use --verbosity 3 when testing your own commands to ensure everything works as expected.

Let me know if you test this out.

ohmycommodore

Using it a week or so ago, it completely missed on some major systems like Sega and SNES and more. I'll have to try it again. Odd that I could see it scraping for those systems but in the end presented me with nothing.

ohmycommodore

I recall I was going to try ScreenScraper as a source this next time.