sselph scraper and video previews, 1 game = 1 Video tweak (+ download link)

bobibrown

Hi everyone,

sselph scraper is really cool and working great, but I made a change so it doesn't download multiple videos if you have full sets, like me.

For example, right now if you have 6 roms of Kirby's Adventure for NES, it will download 6 times the exact same video, for pretty much nothing. When a rom is only few KB, a video on the other hand is often 2 to 8 MB, so it takes a looot of space on your SD CARD.

If you want to free some space on your sd card, you can rescrap using my little tweak. It's not optimal, and I hope sselph will take the time to implement something better (even though the only downside that you will see below is not that big of a deal), because as of right now I'm renaming the videos by their "clean" name found on TheGamesDB.net.

So basically, for Kirby on NES, what the modified scraper will get you :

Kirby's Adventure (Canada)-image.jpg
Kirby's Adventure (Europe)-image.jpg
Kirby's Adventure (France)-image.jpg
Kirby's Adventure (Germany)-image.jpg
Kirby's Adventure (USA) (Rev A)-image.jpg
Kirby's Adventure (USA)-image.jpg
Kirby's Adventure-video.mp4

instead of:

Kirby's Adventure (Canada)-image.jpg
Kirby's Adventure (Canada)-video.mp4
Kirby's Adventure (Europe)-image.jpg
Kirby's Adventure (Canada)-video.mp4
Kirby's Adventure (France)-image.jpg
Kirby's Adventure (France)-video.mp4
Kirby's Adventure (Germany)-image.jpg
Kirby's Adventure (Germany)-video.mp4
Kirby's Adventure (USA) (Rev A)-image.jpg
Kirby's Adventure (USA) (Rev A)-video.mp4
Kirby's Adventure (USA)-image.jpg
Kirby's Adventure (USA)-video.mp4

In the gamelist.xml file, all roms has a link to that same single video file now.

What happen though, is that when you watch your roms on retropie and you switch between versions, the video keeps playing as long as you don't switch to a different game. It's kind of cool, I really like it, but at the same time, you're missing on the alternative covers if you don't switch fast enough, so I don't know how to feel about that.

You can check this behavior in this video:

The only downside to me, is that it could become a little messy if you're willing to access the sd card and check the jpgs and videos, as for some games, you will have this kind of results:

A Boy and his Blob - Trouble on Blobolonia-video.mp4
David Crane's A Boy and His Blob - Trouble on Blobolonia (USA)-image.jpg
David Crane's A Boy and His Blob - Trouble on Blobolonia (Europe)-image.jpg
David Crane's A Boy and His Blob - Trouble on Blobolonia (Europe) (Rev A)-image.jpg

So you might think the video is missing when listing your files by ascending filenames, but it's there, it just don't use the same naming system. But it's not a problem for retropie/emulstation, as it is linked correctly in the XML file.

I had no knowledge of the go language, or how to configure and compile from command lines, so it was a pain to achieve, trying to understand how Steven Selph's script was working too. So I just made very little changes to reach that goal. It still took me something like 12-14 hours though, since I had to get familiar with command lines and the script itself.

If you want to give it a try

here is the source on github:
https://github.com/zcstrife/scraper/tree/patch-1

If you don't know how to, or don't want to bother installing go, setting it up and compile the source yourself, you can download the binary here:
http://uptobox.com/1kyofnlmj3rm

Connect to your retopie via ftp/ssh, and copy this file to:
/opt/retropie/supplementary/scraper

You might have to give yourself permission to overwrite the file, so you need to go into command line, and type this:
sudo chown -R pi /opt/retropie/supplementary/scraper
Where "pi" is your username on the machine (should be by default)

Then, when the file is copied, scrap like usual. Don't forget to remove everything scraped before, before rescraping again!
Access your raspberry pi via your file explorer, and delete the "roms/(system you want to scrap)/images/" directory.
You can also delete the gamelist.xml file located at roms/(system you want to scrap)/gamelist.xml

I would recommand to make a test where you don't have a lot of games first, just to check if you like it. Sega32x doesn't have many games, and you will be able to see how much space you're saving!

As of right now, I scraped all my NES, SNES, MASTER SYSTEM, MEGADRIVE(GENESIS), SEGA 32x, PC ENGINE, ATARI 7800, and everything worked just fine!
Before, I was not able to have all these fulls sets with video preview on my 32GB sd card, I had to stop scraping leaving the SNES full set with no video preview. Now it fits, and I also have about 6 GB free left!
Anyway, I'm very satisfied, and I felt like I had to share it with the community!

If you want to revert back to the original scraper, you'll just have to update from source from the retropie setup menu.

Enjoy, and let me know what you think!

bobibrown

I guess no one cares but me!
Anyway, if it helps just one person in the futur, that's good enough

cyperghost

@bobibrown I think it is nothing about you.
If I speak just for myself I don't care about scraping ...
I'm still using the IO theme that is one of the sleakest themes out there ;) I think people would rather don't install/renew a scraper that is already standarized.

Because the scraping and the webspace/bandwidth used is the same as you ask for a picture and a video for every specific ROM. With your modification the video file is just renamed same in depence of ROM main name and the scraping process isn't speeded up.

If would undergo that task I had done following:
Instead of modifiying the scraper I would modify the gamelists and delete unused video files then your are able to rebuild file names as intendend like

Kirby's Adventure (ALL RELEASES)-video.mp4

Don't give up with your coding skills. Lot's of people here released great stuff and not every script/coding is used and much less is installed as default into the RetroPie repos ... But that's the neat thing here - every user can choose what is the best for his setup ;)

detron

@bobibrown

I am interested, but did not see this post until now. I think there may be many people who do not see it for days, or even weeks.

I am as interested in scraping as I am in playing. but that is just me

bobibrown

@cyperghost Actually I only ask for a video once for every game, since there was already a condition that's checking if the video path exists. And it does right after it downloads the video for the first rom / version of the game. Then for the others versions, it just add a line with the path to the xml instead of downloading and converting the video again. So I guess it's speeding up the process and limits bandwidth usage too, right?
I didn't have to change anything there as it was in favour of my changes.

I tried to make as little changes as possible, since there are things I still don't get in the script.

But also doing it your way, I would have had to use regex and I didn't want to (I forgot how to write regex).

@detron @cyperghost I was just trying to be funny, it's ok if it doesn't get any attention. I know there are very few people who want full sets AND video previews too, if it helps those people, that's good enough. But I still think this is an improvement from the original script, it shouldn't be downloading a video that already exists but with a different name.