Storing ROMs and other data on a remote drive

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

The drive itself is mounted unter /media/usb0, while its directories are mounted separately onto their RetroPie conterparts using the bind option of Linux mounts (i.e. mounting dirs into dirs).

Oh that's really interesting! I wasn't aware of that. Do you do this setup manually or do you have them in some scripts? Anything you can link to? This is what I'm working on at the moment:
https://github.com/seriema/retro-cloud/blob/copy-roms/raspberry-pi/local/switch-to-file-share.sh

I'm not used to Linux and network drives etc. I'd be really happy for any feedback on my project. 😃

mitu

@seriema said in Storing ROMs and other data on a remote drive:

Do you see any downsides? For example I noticed that Skyscraper follows symlinks, so I have a script to rewrite the gamelists again to point to $HOME/RetroPie/roms. That's the only downside I've encountered so far.

Skyscraper has parameters for the location of the produced media and where the gamelist generation location.

Thanks! I haven't been able to find any docs on es_settings.cfg.

There's not a lot of settings and most of them are just a mirror of what's available in the interface. Look at the source to get the available options (https://github.com/RetroPie/EmulationStation/blob/master/es-core/src/Settings.cpp).

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

As for a general setup recommendation to speed things up, don't store the media/artwork scraped in the ROMs folder. If you don't use Parse ~~Metadata~~ Gamelists Only option, then EmulationStation will have fewer files to scan. You can arrange the gamelist.xml to reference the metadata from an additional folder ($mountpoint/media/system/{video,marquee,image}) and leave the ROMs folder just for ROMs and saves.

Clyde

I did it manually. If you know what to do, it's really simple. There are two ways to do a bind mount: manually and automatically at system start via /etc/fstab. Taking my setup as example:

sudo mount --bind /media/usb0/roms /home/pi/RetroPie/roms

But this will vanish after a reboot, so I've added this line to my /etc/fstab:

/media/usb0/roms /home/pi/RetroPie/roms none bind,bg 0 0

The option bg lets the system try again in the background if the initial mount fails. I don't remember if this is a relic from my first setup attempts, or if it's still necessary for my local drive. But it can be useful especially for network mounts.

@mitu said in Storing ROMs and other data on a remote drive:

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

Ahh I forgot about that advantage.

If you don't use Parse Metadata Only option, then EmulationStation will have fewer files to scan.

You Keep Using That Word, I Do Not Think It Means What You Think It Means.

I think you mean Parse Gamelists Only. 😉

seriema

@mitu said in Storing ROMs and other data on a remote drive:

Skyscraper has parameters for the location of the produced media and where the gamelist generation location.

Yes I think that's what I'm using (see my .skyscraper/config.ini). But since it follows symlinks it ends up with mountpoint/.... I'll try bind instead.

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

So that means one is meant to mount on top of the existing $HOME/RetroPie/roms directory? No need to move anything away to say $HOME/RetroPie/roms.bak?

As for a general setup recommendation to speed things up, don't store the media/artwork scraped in the ROMs folder.

Good to know. I'm storing these three separate at the moment:

gamelists="$RETROCLOUD_RPI_MOUNT_POINT/.emulationstation/gamelists"
downloadedMedia="$RETROCLOUD_RPI_MOUNT_POINT/.emulationstation/downloaded_media"
roms="$RETROCLOUD_RPI_MOUNT_POINT/RetroPie/roms"

I was symlinking them to their regular places on the RPi but will definitely change that to bind instead.

Seems like I'm on the right track then? My next to add are the save files (which I'll configure to not be in /roms), and then the controller configs. For the controller configs I'm currently looking at:

/opt/retropie/configs/all/retroarch
/opt/retropie/configs/all/emulationstation/es_input.cfg
/opt/retropie/configs/all/emulationstation/es_temporaryinput.cfg

@Clyde said in Storing ROMs and other data on a remote drive:

There are two ways to do a bind mount: manually and automatically at system start via /etc/fstab.

Thank you for the tips! Can you confirm that you haven't moved away the /home/pi/RetroPie/roms or its contents, like I did in my script?

Instead of /etc/fstab I'm using autostart.sh, which seems preferred in RetroPie 4+ according to the docs.

Clyde

@seriema said in Storing ROMs and other data on a remote drive:

So that means one is meant to mount on top of the existing $HOME/RetroPie/roms directory? No need to move anything away to say $HOME/RetroPie/roms.bak?

That's correct. The mount (bind or normal) "covers" the original contents of the mount point, so that they become inaccessible until you unmount it again. It is even possible to put several mounts on top of each other. Each would "hide" the one before it. This normally doesn't make sense (edit: other than e.g. for fallback mounts in case the main mount fails), but it is possible nonetheless.

Thank you for the tips! Can you confirm that you haven't moved away the /home/pi/RetroPie/roms or its contents, like I did in my script?

Yes, because it isn't necessary for the above reasons.

Instead of /etc/fstab I'm using autostart.sh, which seems preferred in RetroPie 4+ according to the docs.

I do not know why that is, and I'm using the fstab method for many 4.x versions without any problems, because it is the common way to do auto-mounts in most Linux distributions. Maybe @mitu can shine some light on that remark in the Wiki?

seriema

@Clyde Sorry I could have sworn I at least had upvoted your reply last weekend. I only work on these things on the weekends, so I'll be trying this tomorrow or later on. Either way, you've been a great help. Thank you!

mitu

@Clyde said in Storing ROMs and other data on a remote drive:

I do not know why that is, and I'm using the fstab method for many 4.x versions without any problems, because it is the common way to do auto-mounts in most Linux distributions. Maybe @mitu can shine some light on that remark in the Wiki?

I don't know why it's phrased that way or if it's indeed a preferred method vs. the just using fstab. The only advantage I see if that - with this method - the remote disc is mounted before EmulationStation starts, so there's a smaller chance for EmulationStation to start with an empty roms folder.

Clyde

@mitu Thanks for your reply. One could also use the option bootwait in the fstab to ensure that the boot process will wait for the remote mount, right?

mitu

@Clyde bootwait is not an /etc/fstab option, is something set by raspi-config for boot process to wait until the Pi gets an IP address. That might provide enough delay to allow the network to start and mount the drive before EmulationStation starts, so it's an useful addition.

Clyde

@mitu I mean the option that was talked about here: https://unix.stackexchange.com/questions/53456/what-is-the-difference-between-nobootwait-and-nofail-in-fstab

But I don't know if it is depricated by now like nobootwait, as someone in that old question remarked. Maybe you'd have to use x-systemd.device-timeout for that too now.

seriema

@mitu said in Storing ROMs and other data on a remote drive:

The only advantage I see if that - with this method - the remote disc is mounted before EmulationStation starts, so there's a smaller chance for EmulationStation to start with an empty roms folder.

Yes that's what I use it for too, but there's a caveat to it that I missed the first time. The autostart.sh looks like this:

emulationstation #auto

Meaning the mounting has to be added to the top of the file. Otherwise it's too late to be of any use.

Commonly I just append to files with >> or tee -a. But due to the above I end up doing sed magic:

sudo sed -i "1s+^+$mntCmd\n+" /opt/retropie/configs/all/autostart.sh

Clyde

@seriema Thanks for sharing your solution. Like nearly always with console magic, there are multiple ways to add a line at the beginning of a file. 😎

seriema

@seriema said in Storing ROMs and other data on a remote drive:

@Clyde [...] I'll be trying this tomorrow or later on.

I hate threads that end like that and the timestamp is 5 years ago 😁

In the end I didn't change my solution. I spent a whole day mulling over it, and it basically comes down to all the scenarios I'd need to handle.

My previous (and current) approach

Setup:

mount /mnt/retro-cloud
move /home/pi/RetroPie/roms -> /home/pi/RetroPie/roms.local
rsync /home/pi/RetroPie/roms.local -> /mnt/retro-cloud/roms
symlink /mnt/retro-cloud/roms -> /home/pi/RetroPie/roms

Boot: Step 1.

Drawback: If the mount doesn't work, you can't play the local ROMs.
Counter: The idea is that you keep all your ROMs in retro-cloud, and connect it to everything you run emulators on. RetroPie being just one of them. So you wouldn't have anything locally to fall back on anyway. It also makes the copying (step 3) optional during setup, and can be run at any time after boot.

Suggested approach in this thread

Setup:

mount /mnt/retro-cloud
rsync /home/pi/RetroPie/roms -> /mnt/retro-cloud/roms
mount /mnt/retro-cloud/roms -bind -> /home/pi/RetroPie/roms

Boot: 1 and 3.

Drawbacks:

If the ROMs aren't copied in step 2, they can never be copied to retro-cloud after boot.
Boot script is messier. Step 3 should not run if step 1 failed.

Clyde

@seriema said in Storing ROMs and other data on a remote drive:

Boot script is messier. Step 3 should not run if step 1 failed.

That should be avoidable by && between 1. and 3. (bash for "only execute the latter if the former finished successfully").

mount retro-cloud && mount --bind roms

That said, if there's nothing in the local roms directory, it shouldn't matter if 3. runs if 1. fails. The roms directory will be empty no matter if 1. is executed or not in this case. :)

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

That should be avoidable by && between 1. and 3.

Good tip with &&, thanks! I still need them written separately somewhere, so I can run step 2 in-between during setup. Unless I want to have steps 1 and 3 copied in two places (RetroPie's autostart.sh and my setup.sh) I'd need to make them available in a separate script and call them as methods? It's not insurmountable, it's just meh 😁

Hopefully there's nothing in the local folder. The main reason for me to go down the bind mount path was for the fallback functionality (play local roms if VM mount failed). If I'm not going for the fallback then I don't see the drawbacks of my current symlinked approach?

Clyde

@seriema said in Storing ROMs and other data on a remote drive:

If I'm not going for the fallback then I don't see the drawbacks of my current symlinked approach?

On a quick glance, I don't see any either. The fallback is a nice idea, though. Apart from that, I mostly prefer bind-mounts out of "sympathy". They just feel somewhat more elegant to me.

I think that some people prefer symlinks if there's no actual reason for mounts and vice versa. I just happen to be among the latter. :)

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

You can set it in the file /home/pi/.emulationstation/es_settings.cfg via the line <bool name="ParseGamelistOnly" value="true" />.

Finally got around to try that out. I didn't notice any real difference in starting time. It spends 80%-ish of the EmulationStation load screen with "Loading system config". The total time from console to be in the UI is ~1 minute regardless if ParseGamelistOnly is set to true or false.

It might be my setup, because there's no es_settings.cfg file available so I had to create it myself:

<?xml version="1.0"?>
<bool name="ParseGamelistOnly" value="true" />