Help Understanding Scraper Commands to Add Video

aidy80's

Pi Model or other hardware: B+
Power Supply used: Original Pi PSU
RetroPie Version Used: 4.5.1
Built From: retropie-4.5.1 -rpi2_rpi3.img.gz
USB Devices connected: PS3 controller & usb keyboard
Controller used: PS3 controller
Error messages received: none as such
Guide used: (Mention if you followed a guide) Retropie docs Scraper

Hi

I've been working on my Retropie build for a few months now, managed to get my chosen emulators working with my ROMS, nothing too fancy, Arcade, SNES, Neo Geo, Megadrive. I've done some scraping with the built in Emulationstation Scraper, Steven Selph Scraper, Skyscraper and after trying them all I'm settling on box art mainly, then add videos and adjust the theme so that the box art transitions to the video. I've read another post saying that is possible... If not I'll stick with videos.

I've managed to source some videos for my ROMs elsewhere and what I'm struggling with despite reading the docs/guides is, if I save my videos in their own 'videos' folder, in the ROMS folder, what scraper command line can I use to update the game list for each game in each system with an entry for the relevant video clip? I'm trying to avoid having to update each game in the game list manually! I want the scraper to look for the video file and update the game list.

muldjord

You can adjust the Skyscraper artwork.xml to only export the cover art. Read more about that here. Basically your /home/pi/.skyscraper/artwork.xml file should contain something like this:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="cover" width="640" height="480">
    <layer resource="cover" height="480" align="center" valign="middle"/>
  </output>
</artwork>

Then edit /home/pi/.skyscraper/config.ini and uncomment the [main] section and set video="true". That will enable video scraping in general for Skyscraper. It should look like this:

[main]
video="true"

If you haven't already, go to screenscraper.fr and create a user account. This is important! Not just for Skyscraper, but for any scraper using the screenscraper source. When you have your accound credentials, again edit config.ini and add the following:

[screenscraper]
userCreds="USER:PASSWORD"

A note though, there's currently a bug on ScreenScraper's side that broke the login. So even though you have typed it in correctly, it won't log in correctly until they fix this. Hopefully today or in the coming days (I have reported the problem to them, but haven't heard back yet). But it will somewhat work without it, so you can continue, just know that it might be unstable without user-credentials.

Then scrape each platform with the following command:

Skyscraper -p PLATFORM -s screenscraper

And lastly generate the gamelist for each platform with the following command:

Skyscraper -p PLATFORM

And that's it! It will now have boxart as the main artwork, but if a video was found, it will use that instead.

aidy80's

@muldjord Thanks for the comprehensive response. Using the above, will that ensure that only existing local files are used by the scraper? I already have box art from the Emulationstation Scraper and some videos I sourced so I don't want the scraper to download or cache anything else.

muldjord

@aidy80-s You can use the esgamelist import module in Skyscraper to import all of your existing data into the Skyscraper cache. Use the following command for each platform to do so:

Skyscraper -p PLATFORM -s esgamelist

This will import all gamelist entries inside <video>, <cover>, <image> and <marquee> (remember to set videos="true" as described above first!).

After you've done this, for good measure, you should make a backup of your gamelists. Skyscraper WILL overwrite them when you generate gamelists using it.

Then either continue to gather information from ScreenScraper afterwards, or simply generate the gamelists again. Grabbing information from different sources won't overwrite any of the ones you have. Skyscraper caches one per source per platform. And you can prioritize them too.

aidy80's

@muldjord Thanks for your help. This sounds great but, if I import the gamelist into Skyscraper it won't have an entry for a <video> yet (or will it?) Because that's what I'm trying to achieve. If I import my gamelist it will just import the box art because that's all the gamelist has at the minute.

Let me use Neo-Geo as an example, I've got my box art in: /home/pi/.emulationstation/downloaded_images/neogeo/

I've got some videos downloaded from elsewhere that I want to add to the gamelist so that I can then edit the carbon theme to display the box art for a few seconds and then play the video.

I've saved the videos in the neogeo/roms folder in a folder called videos. I want to run a command that will just update the gamelist to point to the right video.

Maybe I'm being a bit too ambitious, it's quite tricky learning this stuff.

muldjord

@aidy80-s said in Help Understanding Scraper Commands to Add Video:

@muldjord Thanks for your help. This sounds great but, if I import the gamelist into Skyscraper it won't have an entry for a <video> yet (or will it?) Because that's what I'm trying to achieve. If I import my gamelist it will just import the box art because that's all the gamelist has at the minute.

That is correct. If you just have the videos downloaded somewhere and they aren't already in your gamelist.xml files, I would strongly recommend importing the videos into Skyscraper using the import module. Read more about that here.

But really, all this manual work you are doing, it's kind of unnecessary. ScreenScraper already have many of the videos in their database, so you just need to scrape your platforms using that as a source and you won't have to do all this manual work at all.

aidy80's

@muldjord Thanks a lot, I was going to say that actually - I'm trying to avoid waiting ages to download lots of video clips but it's actually taking me more time to figure out! 😄 I just thought there would be a simple command for either Steven Selph Scraper or Skyscraper that would achieve what I've described. It looks like I will have to re-scrape instead. Either way thanks for taking the time to help me out, I'm nearly there with my Retropie build and it's a lot of fun!

aidy80's

@muldjord So I tried the steps you mentioned above, got some video clips but also ended up with some complete blanks and little pics of arcade machines when I was expecting cover art. I've somehow managed to mess my game lists up and lost all the manual edits I made. I think I must have done something in the wrong order! Oh well these things happen. I hate to moan as building this Retropie system has been a lot of fun up until now, but doing something that sounds like it should be relatively simple with metadata is turning into a bit of a headache.

I just want to scrape original front cover/box art (whatever it is called, nothing fancy rotated or 3D), videos, description, ratings, no of players, category and end up with a neat and tidy gamelist. Not fussed about marquee or 'wheel'. I'm starting again with Neo Geo. All help greatly appreciated.

muldjord

@aidy80-s Most of the arcade platforms don't have "covers" as they are arcade machines. So that's the reason for those.

What you describe is pretty simple once you get the hang of it - and can be achieved by following my advice above. What you need is the experience working with this to be able to understand why things happen the way they do. And that will come with failures, as you mention. Just keep at it, you'll get it soon enough. You'll find that people are quite willing to help.

Let me know if you have further questions, I'll try to answer them to the best of my ability.

aidy80's

@muldjord Thanks. I've started again with Neo Geo, using the built in Emulationstation scraper as it's the only one that seems to download the Neo Geo cover box art from screenscraper. Then at least I've got a starting point i.e. the cover art I want and a gamelist.

muldjord

@aidy80-s I've just changed this in Skyscraper so it now uses the cover artwork for all the arcade platforms (including neogeo) as you request. It definitely looks better. It will be in Skyscraper 3.4.3 which will be released in a bit when I've tested it.

muldjord

3.4.3 is released now with this fixed (and tested). However, they are having server issues on top of this too, so it might be best to wait a few days before trying again. It kind of works right now, but some times their server times out.

aidy80's

@muldjord Great! I'll try Skyscraper again in a couple of days. In the meantime I tried copying the videos over to the local import location (created a subfolder of neogeo first) gathered resources, then generated a gamelist but it then only populated the video field and the image field is now blank. I then chose the local location to import from my gamelist (in the roms/neogeo folder) the idea being that it imports all the metadata from the gamelist first, then adds the videos on top. Still doesn't seem to want to generate a gamelist with both image and video populated doing it that way.

aidy80's

@muldjord Hey! Just tried something else now, moved my cover art to the local import folder (renamed all 140+ ROM files so that they didn't have an -image suffix 😄 anyone know how to batch rename?) and made sure my videos were in there as well so everything is renamed just like the roms. Regathered, then generated game list, reloaded Emulationstation... Nearly there. Both image and video are populated in the metadata 😃 but I forgot about artwork manipulation that Skyscraper does so I've ended up being able to toggle between some small 3D looking boxes and some video clips. Even the videos are shrunk for some reason, I thought they would fill the size of the black square area on the right of the carbon theme like I've seen in YouTube clips. I do think I understand a bit more about how it works so thanks again for your help - but I've reached beyond excruciating levels of frustration with this now, so I'm giving up for a few days. This Retropie is amazing technology but I'm completely baffled as to why there isn't a quick and simple tool for what I'm trying to achieve.

WeirdH

@aidy80-s said in Help Understanding Scraper Commands to Add Video:

anyone know how to batch rename?

Try this.

muldjord

@aidy80-s You can use the command rename with a regular expression. For instance, if you want to remove all instances of -image from all filenames, you should just be able to type:

rename 's/-image//g' *.png

*.png must be the suffix of of your image files of course. I have not tested this, so please try it on a couple of files first. Once you run the command it can't be undone, so consider making a backup of your image files first.

What you're missing right now, is probably just the correct setup of your Skyscraper artwork.xmlfile. Have you tried the one I posted a bit further up? It should just give you the clean boxart.

How big the video is is not controlled by Skyscraper, so I think that needs to be set up in the ES theme somehow.

but I've reached beyond excruciating levels of frustration with this now, so I'm giving up for a few days

I know how that feels, just keep at it, it will get a lot easier the more you work with it. For any specific use case, there will be a relatively simple solution. But it does take a bit of learning-by-doing to find it. The tools are there.

aidy80's

@muldjord I couldn't keep away... You are absolutely right, got to stick at it and...

...I've had a breakthrough!

I discovered/used a program called Skraper on my PC. Switched on the expert settings and chose which images I want to scrape. I let in run and when it had finished, rebooter Emulationstation. I now have box art and videos. I also noticed it had downloaded 'wheel' images and I thought these would look nice as marquees in the theme, but these were in the XML file as 'thumbnails'. I scraped again but chose to put these in a folder called 'marquees'. I downloaded Note++, loaded up the gamelist and did a find & replace of the word thumbnail with marquee and then resaved the gamelist. I now have box art, video and a nice looking logo at the top!

Not finished yet though... I want the art to fade into the video. So I then loaded up file manager in Retropie, copied the carbon theme over to a new location called 'themes' in the emulationstation folder, navigated to the 'carbon.xml' and chose to edit it.

I then did a CTRL+W to find this bit '<showSnapshotDelay>false</showSnapshotDelay>' and changed the word false to true. Above that is a line that says for how many seconds to delay and I went with 3 (so put in <delay>3</delay>).

Saved, exited, rebooted and it's working great! Well, I've only done it for Neo Geo but I'm hoping to repeat the steps for my other systems.

I should perhaps rename the theme so as not to get confused but I can't figure out how to do that. I want to add in rating and release date under where it says 'Players:' but I can't quite figure that out. Yet.

muldjord

Awesome! There are many ways to achieve what you are looking for. Glad to hear you found one that works well for you. :)

aidy80's

@muldjord Thanks! I'll still be giving Skyscraper another go of course! There's still lots more scraping to be done.