Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?

Dochartaigh

Pi Model or other hardware: Raspberry Pi 4 8GB
Power Supply used: Official
RetroPie Version Used: 4.7.3
Built From: Pre made SD Image on RetroPie website
USB Devices connected: Wireless keyboard/mouse combo, Xbox 360 wireless adapter, 2x hard-wired fightsticks
Controller used: Hori PS3 USB Fightstick
Error messages received: N/A (just looking for advice)
Verbose log (if relevant): N/A

Hello, I want to try using a USB drive to hold all my ROMS. My questions aren't about that process, but about how I can store the other large files on the USB Drive too - such as all the game artwork, video snaps, and the gamelist.xml's required to match the art to the rom's themselves. (maybe I can setup all my customized config files to load from the USB as well???)

Is there a setting to turn-on which will do this? Right now I'm currently running Skyscraper with the option to save the "media" folders (with ALL the art/videos) + the gamelist.xml's (with the game descriptions too) in the individual folders in /home/pi/RetroPie/roms (instead of the usual default location NOT in those roms folders). I'm doing that as a first consolidation step, but don't know if that's the correct way to go about this...

Ideally, if I can get this part down, I'll have the two longest steps (copying over ROMs, and scraping those roms complete with videos snaps) basically eliminated! In theory on a fresh install of RetroPie I should then be able to turn on usbromservice, slap in a USB drive with these folders and literally have everything all ready to go - complete with art and cool video snaps! That would be AWESOME!!!

So how is this done? Does the system automatically look in the /home/pi/RetroPie/roms individual folders FIRST for the gamelists? Or do I have to instruct it to do this? What happens if the system sees all these games first and does NOT add all the accompanying art/videos - how do I get it back to using all my already-downloaded art/videos which are now in the 'media' folder inside each rom folder?

I'm also one who commonly adds, moves, creates new game collections. I usually just 'restart EmulationStation' and it'll refresh my game lists and all the new games show-up (without art at first of course). Will it still work this way on this new setup? Or will I always have to run Skyscraper (which seems like it might now be handling all the gamelist generation since I chose the option for Skyscraper to move all the media/art/videos AND the gamelist.xml files into each individual rom folder -- or does it just not work that way)?

The above (roms and their corresponding art/video-snaps and gamelists) is most important to me, but also wanted to ask about what else --like settings/configs-- can be saved to the USB drive, and be pulled by the root RetroPie setup on the SD card. For example things from the 20+ controller configs I have, to the dozens of mame/fbneo games I had to manually change dipswitch settings on (most to enable 3-4 player mode), to custom button settings for games whose control isn't setup even close to how the real arcade cab controls were setup, etc. etc. etc. (I have in my notes no less than maybe ~20 common-sense tweaks I currently manually do for pretty simple things like the above).

I made a topic back in 2017 about the "FROM_RETROPIE" "TO_RETROPIE" auto-syncing settings folders but was told this wasn't ever documented what exactly it does, and I never got it to work right (and seems to only backup /opt/retropie/configs I believe, which many of my setting tweaks aren't in).

Anyway, thank you for giving this a read and I hope my google-fu didn't fail me when I researched all this (and looking forward to being schooled and learning something new lol!).

jamrom2

@Dochartaigh

Yes... you can do all of this and more.

Get WinSCP, insert your Pi's IP address, and browse the folder structure like any other PC. It cannot be done easily without it. Just copy whatever you want to where ever you want.

You just need to enable "show hidden folders/files" in WinSCP - google it.

You can find the gamelist.xml files in the hidden Directory /home/pi/.emulationstation

you will see the gamelist folder and the images folder there as well.

That's pretty much it...

cdaters

I am assuming that you are running out of space on your SD card? I'd just clone your SD card to your USB drive, using Clonezilla or similar. All of my Pis (with the exception of my Pi400--which has a 512GB SD), are in Argon One cases (one is in a NESPi4 case) and are running off of M.2 SSD 1 or 2TB HDs.

Keep in mind, cloning will destructively overwrite any data you currently have on the USB drive, since it will be effectively making that drive an exact copy of your SD card, so make sure you backup anything you'd like to keep before you begin this process.

Update your firmware on the Pi to Boot from USB (if you haven't already), Use GParted or something similar to resize your data partition to fill the entirety of the USB drive (because the clone will be the same size as your SD card), commit your changes, and you're done!

Boot from your USB drive and everything will be there!

Dochartaigh

@jamrom2 said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

@Dochartaigh

Yes... you can do all of this and more.

Get WinSCP, insert your Pi's IP address, and browse the folder structure like any other PC. It cannot be done easily without it. Just copy whatever you want to where ever you want.

You just need to enable "show hidden folders/files" in WinSCP - google it.

You can find the gamelist.xml files in the hidden Directory /home/pi/.emulationstation

you will see the gamelist folder and the images folder there as well.

That's pretty much it...

Hi! - thank you for responding but I don't think you're understanding exactly what I'm asking about though.

I have no problem finding the gamelists, or moving them (in fact Skyscraper already moved the games lists for me + ALL the art/videos too - both into each systems /home/pi/retropie/roms individual folders), the question is HOW to configure a FRESHLY installed system how to know to read the gamelists AND art/videos from a non-default location, such as on the USB Drive, in that drives ROMs folder -- that's the mystery I'm trying to figure out!

@cdaters said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

I am assuming that you are running out of space on your SD card? I'd just clone your SD card to your USB drive, using Clonezilla or similar. All of my Pis (with the exception of my Pi400--which has a 512GB SD), are in Argon One cases (one is in a NESPi4 case) and are running off of M.2 SSD 1 or 2TB HDs.

Hello. No, I'm not running out of space on my SD card at all. Like I tried to explain in my initial post (and maybe failed at ;) my premise is to have everything (games and scraped art) on an easily removable USB Drive - so I can have a USB Drive with all my games, gamelists, artwork/screenshots/box-art/video-snaps which I can pop into a completely different Raspberry Pi RetroPie setup (even a different model Pi), and automatically have all the games (pulled from their gamelists) + ALL the art/videos show up on that other system all set and ready to go. That's what I don't know how to do since I assume another default-setup RetroPie will NOT by default read gamelists and scraped art off a default location such as the USB Drive I'm planning to use.

The RetroPie directions also say that when you turn on usbromservice it AUTOMATICALLY copies over all your games to the new external USB Drive - so no cloning needed in this regard.

@cdaters said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

Update your firmware on the Pi to Boot from USB (if you haven't already), Use GParted or something similar to resize your data partition to fill the entirety of the USB drive (because the clone will be the same size as your SD card), commit your changes, and you're done!

Boot from your USB drive and everything will be there!

I did NOT know about this! And is indeed VERY cool! Do you find boot-up speeds, compiling a new emulator, or other high bandwidth stuff is actually faster when the OS is running off the external drive, with its faster read/write speeds?

Anyway, the above while being super cool, isn't going to work for me since I own a bunch of different Pi Zero's, RP3 and RP3B+'s, Pi Zero 2 W, RPi4... and the OS itself isn't 100% compatible from unit to unit (I think B to B+ is ok to switch a SD card from one to the other...think Zero 2 W as well except for a LAN or something driver, but you can't put a SD card from a Pi 4 into the original Zero or 3 because it just won't even boot).

Folly

@Dochartaigh

Gamelists, accompanied with media, can also be in a roms directory if that is what you are looking for.

Dochartaigh

@Folly said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

@Dochartaigh

Gamelists, accompanied with media, can also be in a roms directory if that is what you are looking for.

Yup, that's what I'm looking to do. Do you have a link of how this is accomplished? Does the system look for the gamelists + art on the USB Drive FIRST? (in those individual /home/pi/RetroPie/roms -- or where ever the new folders are on the USB Drive).

Also looking for any options about having all my settings/config-files on, and read from, the USB drive too.

Folly

@Dochartaigh

I don't have a link, but we use this structure for our project over here :
https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-and-mame-standalone
The front-end contains the possibility to download our predefined gamelist+media.

And here we are trying to make more :
https://retropie.org.uk/forum/topic/32188/creating-es-gamelists-with-media-for-mame-lr-mess
Perhaps something to contribute to.

You can download and see the examples on google-drive here :
https://drive.google.com/drive/folders/1f_jXMG0XMBdyOOBpz8CHM6AFj9vC1R6m
These are created in such a way that these are the rom directories.
Take for instance videopac :
https://drive.google.com/drive/folders/12OKl3s5Dw4FmZkkeLj0EZmoS8K14zD_W
These files are placed in :
/home/pi/RetroPie/roms/videopac

If you have existing gamelists+media you can copy the gamelist to the specific rom directory and it should work for the names.
Though, you also have to copy the media and change the locations in the gamelist.xml otherwise your media isn't found.
You can simply do that with find and replace in a text editor.
Have a good look at the locations in our gamelists.

cdaters

@cdaters said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

Update your firmware on the Pi to Boot from USB (if you haven't already), Use GParted or something similar to resize your data partition to fill the entirety of the USB drive (because the clone will be the same size as your SD card), commit your changes, and you're done!

Boot from your USB drive and everything will be there!

I did NOT know about this! And is indeed VERY cool! Do you find boot-up speeds, compiling a new emulator, or other high bandwidth stuff is actually faster when the OS is running off the external drive, with its faster read/write speeds?

Anyway, the above while being super cool, isn't going to work for me since I own a bunch of different Pi Zero's, RP3 and RP3B+'s, Pi Zero 2 W, RPi4... and the OS itself isn't 100% compatible from unit to unit (I think B to B+ is ok to switch a SD card from one to the other...think Zero 2 W as well except for a LAN or something driver, but you can't put a SD card from a Pi 4 into the original Zero or 3 because it just won't even boot).

I have not really noticed any improved I/O speeds using USB media like this, especially between using m.2/SSD vs spinning disk platters (older HDs). But I do get more drive space this way.

What you are proposing is very cool, you are looking for a portable RetroPie set-up much in the same vein as a Portable Apps USB flashdrive. I don't know if this is realistic in the sense that your emulators on each Pi that you run that USB drive on would have to be configured the same way. This includes your controller configs, etc.

I know that most of that data can be user-controlled/configured and is stored in the /opt/retropie/configs directory on the Pi. For example, like the docs kind of allude to here and here, in terms of locating user configs elsewhere on the system. I supposed you could configure that location as a symbolic link and point it at that same location on your USB drive, but some data is certainly kept in /etc/emulationstation too. I may have to experiment with this idea and see if it can be done.

Keep in mind though, that the base RetroPie installs on all the Pis you want to do this with, will have to be configured similarly and I can't expand my mind to follow all those threads just yet, lol! I am definitely into Teflon hour and getting ready to head home for the day.

Dochartaigh

@Folly said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

you also have to copy the media and change the locations in the gamelist.xml otherwise your media isn't found.
You can simply do that with find and replace in a text editor.

Thank you for that information, but TBH the entire point of me making this post is basically to NOT have to manually edit .XML files (and like 30+ of them for all the systems I run). Just trying to see how in RetroPie you can turn on the feature to have the gamelist.xml's and the media folder w/ artwork switched over to the individual rom folders. Skyscraper seems to do this, but that process can take days or WEEKS even (when it's acting up and timing out 99.99% of the time) – so it is possible, just don't know how...

@cdaters said in Moving art/video-snaps/gamelists/and-other-settings to USB Drive ALONG with ROMs?:

What you are proposing is very cool, you are looking for a portable RetroPie set-up much in the same vein as a Portable Apps USB flashdrive. I don't know if this is realistic in the sense that your emulators on each Pi that you run that USB drive on would have to be configured the same way. This includes your controller configs, etc.

Right now I'll be happy if I can get my first step figured out: how to get my games + scraped art onto the USB drive, and have a fresh system read from the USB for those two things (nobody yet has been able to tell me a good way to point RetroPie to the USB drive for those things...).

My premise to do this is simple: scraping is STILL a nightmare, ~SIX years later (since my 1st build). Granted these last two spats have been the best ever, but with a six year history of it timing out forever (even with extra threads on Screenscraper.fr), and taking me literally, no joke, days to even WEEKS to scrape my 30+ systems properly, I have very high doubts it'll continue to be good.

So, when I can get an entire Pi setup, ALL my customizations, all roms working, everything – in a couple hours, minus artwork... but then it can STILL take anywhere from like 6-12ish hours (best case scenario), to DAYS to WEEKS to scrape the art for 30+ systems. This is why I want a USB drive with all games, then in each individual rom directory have the gamelist.xml + 'media' folder with all the art/videos so I never, ever, have to deal with a scraping nightmare ever again.

...I'll deal with the settings later (which BTW, the gamelists have NOTHING to do with any of my other tweaks - they literally just say the game name + where to find the art/videos-snaps... so they work on ANY emulator on ANY model of Pi!).