Issues when mounting more than just ROMs over NFS

code_dredd

Hello.

TL;DR:

Why is it that mounting splashscreens over NFS causes the videos/intros to not play before EmulationStation loads? How do I fix this so that they do play?

Details

I currently have 2 RetroPie installations, one on a Pi 3B+ and another in a Pi 4B. They're working fine. I have a NAS where ROMs are stored and I can mount/access the ROMs on the NAS over NFS from both systems, so far without issues.

Every other post I've seen seems to limit the NFS access to the RetroPie/roms directory and the docs only show steps to do this with ROMs. I don't recall seeing any warnings or known issues if you try mounting more than that, and I didn't see why it should/would be limited to only ROMs, so I decided to mount the entire contents of the $HOME/pi/RetroPie from the NAS over NFS. The idea here was that I'd avoid duplicating contents across boards, primarily BIOS and splashscreens.

In any case, the only issue I've noticed is that if I mount $HOME/RetroPie from the NAS, then the splash screen intros do not play before EmulationStation loads. I tried mounting only the splashscreens directory, instead of the whole $HOME/RetroPie, but it made no difference - the intros still don't play. This behavior is exactly the same in both the 3B+ and 4B boards.

Using grep -i to check for nfs and error doesn't seem to show anything interesting. This is a short sample:

$ grep nfs /var/log/syslog
Apr  1 20:50:01 retropi3 kernel: [    1.902054] FS-Cache: Netfs 'nfs' registered for caching
Apr  1 20:50:01 retropi3 systemd[1]: nfs-config.service: Succeeded.
Apr  1 20:50:01 retropi3 kernel: [    1.903232] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
Apr  1 20:50:10 retropi3 nfsidmap[608]: nss_getpwnam: name 'retropie' not found in domain 'skynet.lan'
Apr  1 21:00:37 retropi3 nfsidmap[792]: nss_getpwnam: name 'retropie' not found in domain 'skynet.lan'
Apr  1 21:04:01 retropi3 systemd[1]: nfs-config.service: Succeeded.

The relevant /etc/fstab entries are as follows:

area51:/mnt/tank/users/retropie/roms /home/pi/RetroPie/roms nfs4 defaults 0 0
area51:/mnt/tank/users/retropie/splashscreens /home/pi/RetroPie/splashscreens nfs4 defaults 0 0

Note that the host name is resolved by a Pi-Hole in my LAN.

Also, the directories do get mounted successfully and permissions are correct - i.e., no issues accessing/playing ROMs have been observed and the splashscreen folder contents are visible/accessible.

I had also already set the "Network at Boot" option to "yes" in the raspi-config to wait for networking, which I thought would've been sufficient. In short, my end goal is to mount as much as possible over NFS:

$HOME/RetriPie/roms: Already done, no issues (so far).
$HOME/RetriPie/splashscreens: Mounts fine, but intros don't play at boot - why?
$HOME/RetriPie/BIOS: I can do this, but should I? If not, why not?
$HOME/RetriPie/retropiemenu: I can do this, but should I? If not, why not?

Am I simply not supposed to mount anything else other than roms over NFS? I'm at least not aware of a technical reason for why I shouldn't be able to mount splashscreens and BIOS over NFS along with roms... though I can't say I'm as certain about retropiemenu (I assume content in this one could change based on themes, etc.).

Any advice on something I may be missing?

Thanks.

Edits

I've just run into an issue where the RPi 4B, several minutes after loading EmulationStation, will get "stuck" and freezes. This also happens on the shell and the only way to restore things is by rebooting the Pi 4B. This is with only the roms/ directory mounted over NFS and nothing more; this was never observed prior to the NFS-based approach. Not sure if it happens every time, but it has happened several times already. (I've not tested the 3B+ to see if it has the same behavior, but I've not seen it there yet.)

Relevant info:

Pi Model or other hardware: 3B+ and 4B (Both are CanaKits, with their associated power supplies, etc.)
Power Supply used: The ones included with the CanaKit bundles
RetroPie Version Used:
- Pi 3B+: 4.7.7 (fcb80cbe)
- Pi 4B: 4.7.8 (88a8574)
Built From: Initial setup with NOOBs included microSD card and then installed with the web-based script.
USB Devices connected: 1 Logitech Gamepad
Error messages received: None
How to replicate the problem:
- Mount the $HOME/RetroPie/splashscreens directory over NFS with an entry under /etc/fstab.
- Reboot the Raspberry Pi with sudo reboot.
- Wait for system to boot and notice your splashcreen intro didn't play as it used to do before.

mitu

Your issue is most likely the fact that the splashscreen folder is not yet mounted when the splashscreen service starts. Because the splashscreen starts early, the network might not even be up at the time.

code_dredd

@mitu I suspected that, but isn't that what setting the "Network at Boot" to "yes" is supposed to avoid? Is there a way to confirm/verify whether this is the real cause and, if so, a way to address it?

mitu

@code_dredd said in Issues when mounting more than just ROMs over NFS:

I suspected that, but isn't that what setting the "Network at Boot" to "yes" is supposed to avoid?

That's a Raspberry Pi OS configuration that instructs the dhcpcdservice to block until a network address is acquired. The splashscreen service doesn't care/know about the network.

Is there a way to confirm/verify whether this is the real cause and, if so, a way to address it?

You can probably check by looking at the asplashscreen service log or copying the splashscreen files locally (which I think it works anyway). If you wish to wait for network before starting the splashscreen, you can add a dependency for the asplashscreen service to the network service and/or remote-fs systemd target.

code_dredd

@mitu Thanks for replying.

If you wish to wait for network before starting the splashscreen, you can add a dependency for the asplashscreen service to the network service and/or remote-fs systemd target.

I tried several things these last couple of days, but have not been successful. (Admittedly, I've not had to setup or modify systemd unit files before, so I'm not sure where the issue is.)

First, I tried creating my own unit file. Here's the unit file:

[Unit]
Description=NFS share for RetroPie contents
DefaultDependencies=no
Conflicts=umount.target
After=network-online.target remote-fs.target
Before=umount.target

[Mount]
What=area51:/mnt/tank/users/retropie
Where=/home/pi/RetroPie
Type=nfs4
Options=defaults

[Install]
WantedBy=multi-user.target

To be fair, it works when I run systemctl start home-pi-RetroPie.mount, but not when I reboot the Pi board. (I had run systemctl enable home-pi-RetroPie.mount before rebooting the board.) I also tried Before=umount.target asplashscreen.service, thinking it'd make my unit run before the splash screen service unit, but it made no difference.

I did the above b/c I was trying to avoid modifying the built-in ones (e.g., in case of an upgrade wanting to modify them would end up destroying my changes or running into conflicts). That said, I also tried updating the asplashscreen.service unit. The changes, below, boil down to adding network-online.target and remote-fs.target to the After= and Wants= lines:

[Unit]
Description=Show custom splashscreen
DefaultDependencies=no
After=network-online.target remote-fs.target console-setup.service
Wants=network-online.target remote-fs.target console-setup.service
ConditionPathExists=/opt/retropie/supplementary/splashscreen/asplashscreen.sh

This also failed to work as I had expected. Question: Do you have more specific suggestions on what changes I should actually make here?

Thanks.

code_dredd

I added a new line, WantedBy=asplashscreen.service, to the [Install] section of my unit file, in addition to the original/existing WantedBy=multi-user.target, and that made it work. Is this the correct way of doing this or are there better ways (e.g., faster, more reliable/predictable, etc)?

Thanks.

mitu

@code_dredd said in Issues when mounting more than just ROMs over NFS:

Is this the correct way of doing this

Yes.

code_dredd

@mitu

Yes

?...

code_dredd

@mitu Thanks. Now that your edit has clarified this, do you have any ideas regarding the system freezes I've noted?

I notice them when trying to do simple things, such as listing directory contents with the ls command. They seem to happen within a few minutes of the system booting to EmulationStation - even if you've exited EmulationStation back to the shell immediately after reaching the menu.

The Pi board log shows nfs: server area51 not responding, timed out, even though we're in the same LAN and the NAS is idling, so it's definitely not a workload issue.

Future Readers

For your reference, this is what I ended up doing:

I removed all the NFS mount entries from the /etc/fstab file and am relying on systemd service unit files (see below).
Due to the increased wait time, which was more than I had expected, I've decided to mount only the following /home/pi/RetroPie sub-directories over NFS: roms, BIOS, and retropiemenu. (This means that 3 service unit files had to be created.)

Systemd Service Unit Files - RetroPie Sub-Directory Mounts

You will need three (3) service units, one per mount. These go under the /etc/systemd/system/ directory and must have the following names:

home-pi-RetroPie-BIOS.mount
home-pi-RetroPie-retropiemenu.mount
home-pi-RetroPie-roms.mount

Note that, if you fail to use the naming convention expected by systemd, systemd will let you know about it with an error.

(In general, the naming convention is to name them after the paths you intend to mount, replacing the slashes (/) for dashes (-) - e.g., /home/pi/RetroPie/roms would be home-pi-RetroPie-roms.mount. The command systemd-escape -p --suffix=mount <mountpoint-path> will tell you what your service unit file(s) should be named as, replacing <mountpoint-path> for your desired paths - e.g., /home/pi/RetroPie/roms.)

The 3 service units look exactly the same as below, except for the Description under [Unit] and, more importantly, the [Mount] section, which has different values for the What= and Where= lines respectively. (Obviously, these will be different for you, but should be self-explanatory.) The one for my BIOS mount is shown below:

# home-pi-RetroPie-BIOS.mount
[Unit]
Description=NFS share for RetroPie BIOS files
DefaultDependencies=no
After=network-online.target remote-fs.target
Before=umount.target
Conflicts=umount.target

[Mount]
What=area51:/mnt/tank/users/retropie/BIOS
Where=/home/pi/RetroPie/BIOS
Type=nfs4
Options=defaults

[Install]
WantedBy=multi-user.target

Note that, since these are not required before the asplashscreen.service, they don't need to declare that service as a dependency in the [Install] section.

Systemd Service Unit File - RetroPie Directory Mount

This one is in case you don't really care about the extra time needed to mount $HOME/RetroPie/splashscreens and want to mount the entire $HOME/RetroPie directory over NFS:

# home-pi-RetroPie.mount
[Unit]
Description=NFS share for RetroPie
DefaultDependencies=no
After=network-online.target remote-fs.target
Before=umount.target asplashscreen.service
Conflicts=umount.target

[Mount]
What=area51:/mnt/tank/users/retropie
Where=/home/pi/RetroPie
Type=nfs4
Options=defaults

[Install]
WantedBy=multi-user.target
WantedBy=asplashscreen.service

For this one, you do want to add the last WantedBy=asplashscreen.service line. This makes systemd wait until your $HOME/RetroPie directory has been mounted over NFS before launching the asplashscreen.service, which is the one that would run your splashscreen video intros. (Not doing this will get systemd to launch the splashscreen service before the NFS mounts are in place and the intro videos won't be found/played.)

Hope this helps and I'll update if I find out what's causing the unpredictable/unexpected hang-ups I've started to observe after switching over to NFS mounts.

mitu

@code_dredd said in Issues when mounting more than just ROMs over NFS:

The Pi board log shows nfs: server area51 not responding, timed out, even though we're in the same LAN and the NAS is idling, so it's definitely not a workload issue.

If the NFS server doesn't respond, then the 'freezes' are normal. Is the network connection stable on the Pi ? Have you tried using NFSv3 instead ?

code_dredd

@mitu

If the NFS server doesn't respond, then the 'freezes' are normal.

Indeed, but I'm not aware of any reason as to why the server would "fail to respond". I've also noticed that it doesn't happen 100% of the time. For example, after rebooting the Pi (which I did while writing my previous post), the Pi worked fine, being online for about 10 minutes without running into the problem. (It'd normally happen within the first few minutes.)

Note also that this is with the Pi only. If I mount the same NFS shares on my Ubuntu desktop, I never see this problem.

Is the network connection stable on the Pi ?

Yes. Note that this Pi board had been in operation for over a month prior to me trying the NFS-based approach and no other issues had been observed.

Have you tried using NFSv3 instead ?

No. I didn't really intend on using it b/c I was looking forward to the NFSv4 improvements over NFSv3, including performance, better file locking/handling, etc. That being said, I suppose I can give it a try and see what happens...

Is there any reason to think that NFSv4 is the actual cause of the issue here?

code_dredd

It seems NFSv3 didn't really fix the issue and ran into others (e.g., weird user IDs on remapping). I've gone back to NFSv4 and increased the number of NFS servers/processes from the default of 4 up to 5 to see if that helps.

code_dredd

@code_dredd said in Issues when mounting more than just ROMs over NFS:

It seems NFSv3 didn't really fix the issue and ran into others (e.g., weird user IDs on remapping). I've gone back to NFSv4 and increased the number of NFS servers/processes from the default of 4 up to 5 to see if that helps.

It didn't help.