Storing ROMs and other data on a remote drive

seriema

Thanks @mitu . I'm currently doing this:

Mount the drive at /mnt/retro-cloud
Copy $HOME/RetroPie/roms to /mnt/retro-cloud/roms
Move $HOME/RetroPie/roms to $HOME/RetroPie/roms.bak
Symlink /mnt/retro-cloud/roms to $HOME/RetroPie/roms

I'm not sure what difference the symlink will have compared to mounting in the RetroPie location, but I suspect some file lookups could be affected? I've found that EmulationStation takes a long time to start, and not just the first time but almost every time. Would you say that there's a difference between symlink or mounting directly? Am I having unnecessary steps? I'm scripting all of the steps so I'm avoiding steps like using the samba share through my PC to copy stuff.

mitu

Why symlink instead of directly mounting the roms folder ? It has no real advantage as far as ROM loading goes.

I've found that EmulationStation takes a long time to start, and not just the first time but almost every time.

Depending on how many ROMs you have, reading the list over the network has an impact. If you don't add new ROMs often, consider using the Parse Metadata Only option in EmulationStation so that the remote folders are not scanned on every start.

Clyde

I would also recommend to store any scraped images or videos locally so that they load as quickly as possible when you're browsing your games in Emulation Station. Most scrapers have an option for the artwork location. You may want to see to it that this options is not set to the roms folder.

seriema

@mitu

Why symlink instead of directly mounting the roms folder ?

It holds more than the ROMs. It has the gamelists and images for EmulationStation as well. Perhaps this diagram clarifies:
https://raw.githubusercontent.com/seriema/retro-cloud/develop/diagrams/filestructure.svg

I'm hoping to store more things further on.

I did create two mounts for a while, so they could be mounted directly on $HOME/RetroPie and $HOME/.emulationstation, but startup often got delayed and one or both of the mounts would fail. I don't know why but I decided that the fewer points of failure, the better.

Depending on how many ROMs you have, reading the list over the network has an impact. If you don't add new ROMs often, consider using the Parse Metadata Only option in EmulationStation so that the remote folders are not scanned on every start.

Oh that would help! I saw your post in another thread where you posted a screenshot of EmulationStation. Is that it? Is it possible to set from the console or change a config file?

seriema

I would also recommend to store any scraped images or videos locally so that they load as quickly as possible when you're browsing your games in Emulation Station.

@Clyde Yes it's a slight issue right now that there's a slight lag when navigating the ROM's in EmulationStation. I'm storing all the media in the remote drive because of the space they take, and the time it takes to re-scrape everything whenever I want to set up my library on a new RetroPie installation. It's a trade-off I'm living with for now that I would like to improve it with time. A fallback could be copying from the remote drive if there's enough space locally, but I'm hoping to discover something smarter.

Clyde

@seriema said in Storing ROMs and other data on a remote drive:

Is that it? Is it possible to set from the console or change a config file?

I think that's what @mitu meant. You can set it in the file /home/pi/.emulationstation/es_settings.cfg via the line <bool name="ParseGamelistOnly" value="true" />. Mind that .emulationstation is a hidden directory, as are all directories in Linux whose name begin with a dot.

seriema

@mitu said in Storing ROMs and other data on a remote drive:

It has no real advantage as far as ROM loading goes.

Do you see any downsides? For example I noticed that Skyscraper follows symlinks, so I have a script to rewrite the gamelists again to point to $HOME/RetroPie/roms. That's the only downside I've encountered so far.

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

You can set it in the file /home/pi/.emulationstation/es_settings.cfg via the line <bool name="ParseGamelistOnly" value="true" />.

Thanks! I haven't been able to find any docs on es_settings.cfg. I've searched the github wikis and the only mention on retropie.org.uk outside of the forum is https://retropie.org.uk/docs/Child-friendly-EmulationStation/ Do you know where I can read more about what's supported and not? Or is it just trial and error, and forum?

Clyde

@seriema Most of the settings in es_settings.cfg should be also available through ES' graphical menus, and be rather self-explanatory and/or mentioned in the Docs. Are there any specific lines in the file that you want to know more about?

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

and/or mentioned in the Docs

There's no mention of that file:

I was just looking for something to browse, to see if there's anything else that might interest me. Hard to say if there's anything specific in it I'm looking for.

If it was possible, I'd like to just mount my drive on /mnt/retro-cloud and then change configs to look there instead of on the RPi. It'd be preferable over moving and mounting/symlinking on top of places where the files usually are.

Clyde

Alas, I don't know of a full documentation of that file. Maybe others here do.

As for mounting directly or changing the config files, both have their benefits and disadvantages. I prefer mounting to symlinks for the following reasons:

It seems more clean to me that way (the weakest reason 😉 ).
Secondly, I can switch romsets or artwork collections just by re-mounting, which is much easier than to change many paths in multiple config files, more or less depending on what's to change.
Finally, I usually avoid to mess with config files that may change with a future update of RetroPie.

That said, I'm using a slightly different way to deal with multiple directories on an external drive. The drive itself is mounted unter /media/usb0, while its directories are mounted separately onto their RetroPie conterparts using the bind option of Linux mounts (i.e. mounting dirs into dirs).

seriema

@Clyde said in Storing ROMs and other data on a remote drive:

The drive itself is mounted unter /media/usb0, while its directories are mounted separately onto their RetroPie conterparts using the bind option of Linux mounts (i.e. mounting dirs into dirs).

Oh that's really interesting! I wasn't aware of that. Do you do this setup manually or do you have them in some scripts? Anything you can link to? This is what I'm working on at the moment:
https://github.com/seriema/retro-cloud/blob/copy-roms/raspberry-pi/local/switch-to-file-share.sh

I'm not used to Linux and network drives etc. I'd be really happy for any feedback on my project. 😃

mitu

@seriema said in Storing ROMs and other data on a remote drive:

Do you see any downsides? For example I noticed that Skyscraper follows symlinks, so I have a script to rewrite the gamelists again to point to $HOME/RetroPie/roms. That's the only downside I've encountered so far.

Skyscraper has parameters for the location of the produced media and where the gamelist generation location.

Thanks! I haven't been able to find any docs on es_settings.cfg.

There's not a lot of settings and most of them are just a mirror of what's available in the interface. Look at the source to get the available options (https://github.com/RetroPie/EmulationStation/blob/master/es-core/src/Settings.cpp).

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

As for a general setup recommendation to speed things up, don't store the media/artwork scraped in the ROMs folder. If you don't use Parse ~~Metadata~~ Gamelists Only option, then EmulationStation will have fewer files to scan. You can arrange the gamelist.xml to reference the metadata from an additional folder ($mountpoint/media/system/{video,marquee,image}) and leave the ROMs folder just for ROMs and saves.

Clyde

I did it manually. If you know what to do, it's really simple. There are two ways to do a bind mount: manually and automatically at system start via /etc/fstab. Taking my setup as example:

sudo mount --bind /media/usb0/roms /home/pi/RetroPie/roms

But this will vanish after a reboot, so I've added this line to my /etc/fstab:

/media/usb0/roms /home/pi/RetroPie/roms none bind,bg 0 0

The option bg lets the system try again in the background if the initial mount fails. I don't remember if this is a relic from my first setup attempts, or if it's still necessary for my local drive. But it can be useful especially for network mounts.

@mitu said in Storing ROMs and other data on a remote drive:

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

Ahh I forgot about that advantage.

If you don't use Parse Metadata Only option, then EmulationStation will have fewer files to scan.

You Keep Using That Word, I Do Not Think It Means What You Think It Means.

I think you mean Parse Gamelists Only. 😉

seriema

@mitu said in Storing ROMs and other data on a remote drive:

Skyscraper has parameters for the location of the produced media and where the gamelist generation location.

Yes I think that's what I'm using (see my .skyscraper/config.ini). But since it follows symlinks it ends up with mountpoint/.... I'll try bind instead.

@Clyde has a good idea - you can use bind mounts instead of symlinks - so if the remote system is not available, the local data can be used instead.

So that means one is meant to mount on top of the existing $HOME/RetroPie/roms directory? No need to move anything away to say $HOME/RetroPie/roms.bak?

As for a general setup recommendation to speed things up, don't store the media/artwork scraped in the ROMs folder.

Good to know. I'm storing these three separate at the moment:

gamelists="$RETROCLOUD_RPI_MOUNT_POINT/.emulationstation/gamelists"
downloadedMedia="$RETROCLOUD_RPI_MOUNT_POINT/.emulationstation/downloaded_media"
roms="$RETROCLOUD_RPI_MOUNT_POINT/RetroPie/roms"

I was symlinking them to their regular places on the RPi but will definitely change that to bind instead.

Seems like I'm on the right track then? My next to add are the save files (which I'll configure to not be in /roms), and then the controller configs. For the controller configs I'm currently looking at:

/opt/retropie/configs/all/retroarch
/opt/retropie/configs/all/emulationstation/es_input.cfg
/opt/retropie/configs/all/emulationstation/es_temporaryinput.cfg

@Clyde said in Storing ROMs and other data on a remote drive:

There are two ways to do a bind mount: manually and automatically at system start via /etc/fstab.

Thank you for the tips! Can you confirm that you haven't moved away the /home/pi/RetroPie/roms or its contents, like I did in my script?

Instead of /etc/fstab I'm using autostart.sh, which seems preferred in RetroPie 4+ according to the docs.

Clyde

@seriema said in Storing ROMs and other data on a remote drive:

So that means one is meant to mount on top of the existing $HOME/RetroPie/roms directory? No need to move anything away to say $HOME/RetroPie/roms.bak?

That's correct. The mount (bind or normal) "covers" the original contents of the mount point, so that they become inaccessible until you unmount it again. It is even possible to put several mounts on top of each other. Each would "hide" the one before it. This normally doesn't make sense (edit: other than e.g. for fallback mounts in case the main mount fails), but it is possible nonetheless.

Thank you for the tips! Can you confirm that you haven't moved away the /home/pi/RetroPie/roms or its contents, like I did in my script?

Yes, because it isn't necessary for the above reasons.

Instead of /etc/fstab I'm using autostart.sh, which seems preferred in RetroPie 4+ according to the docs.

I do not know why that is, and I'm using the fstab method for many 4.x versions without any problems, because it is the common way to do auto-mounts in most Linux distributions. Maybe @mitu can shine some light on that remark in the Wiki?

seriema

@Clyde Sorry I could have sworn I at least had upvoted your reply last weekend. I only work on these things on the weekends, so I'll be trying this tomorrow or later on. Either way, you've been a great help. Thank you!

mitu

@Clyde said in Storing ROMs and other data on a remote drive:

I do not know why that is, and I'm using the fstab method for many 4.x versions without any problems, because it is the common way to do auto-mounts in most Linux distributions. Maybe @mitu can shine some light on that remark in the Wiki?

I don't know why it's phrased that way or if it's indeed a preferred method vs. the just using fstab. The only advantage I see if that - with this method - the remote disc is mounted before EmulationStation starts, so there's a smaller chance for EmulationStation to start with an empty roms folder.

Clyde

@mitu Thanks for your reply. One could also use the option bootwait in the fstab to ensure that the boot process will wait for the remote mount, right?

mitu

@Clyde bootwait is not an /etc/fstab option, is something set by raspi-config for boot process to wait until the Pi gets an IP address. That might provide enough delay to allow the network to start and mount the drive before EmulationStation starts, so it's an useful addition.