Emulation station freezing

roslof

@mahoneyt944 said in Emulation station freezing:

4.6.8 worked fine. 4.7.1 had freezes with the same gpu setting. So any commits in that time should be suspect. ES would be my target since it's all that froze for me. The pi still works fine other than that, as in its still working over network etc.

Thank you @mahoneyt944
Are you currently running the experimental ES or non-experimental?

mahoneyt944

@roslof I believe I updated from binary. So latest stable.

roslof

@mahoneyt944 said in Emulation station freezing:

@roslof I believe I updated from binary. So latest stable.

Again, helpful. It's still early, but my issue is starting to look like a source-built ES-dev build. The only way I'm testing this is by simply letting the video screensaver run until it freezes, or until I give a ton of time.

Like you, @mahoneyt944, I appear stable with non-dev (so good news for RetroPie in general as dev is experimental).

If it turns out to be source-built ES-dev, I'll zero-in on whether or not it's how it's being built, or if a breaking change came in.

UPDATE: Freezes with latest pre-built binary of ES-dev. Convinced issue is indeed with EmulationStation-dev. Will manually build from older commits (looks like May timeframe has some good targets). It also would appear that my issue is something fairly new, as this thread started in Nov 2020...

UPDATE #2: When the video freezes, it appears that vcgencmd is hanging as seen in htop (status D):

D - uninterruptible sleep (usually IO)
Uninterruptable processes are USUALLY waiting for I/O following a page fault. The process/task cannot be interrupted in this state, because it can't handle any signals; if it did, another page fault would happen and it would be back where it was.

mahoneyt944

@roslof my arcade cabinet "retropie" runs 24/7 so I would always find it frozen when returning after a period of time on the screensaver "slideshow". Once I started testing things though I found ES would sometimes randomly freeze when scrolling menus / navigating around. So I'm not sure it's limited to screensavers. In my case it seemed to be anytime the GPU was getting a jolt of action within ES.

roslof

@mahoneyt944 and you were overclocked for GPU?

mahoneyt944

@roslof I was using gpu_freq=500 before I updated. Never an issue. After updating, I could no longer use this setting in config.txt

There are other setting that effect GPU, so there maybe others effected too.

roslof

@mitu since ES stable seems fine for me, I went ahead and created a debug version of EmulationStation-dev leveraging the existing RetroPie script and adding params+=(-DCMAKE_BUILD_TYPE=Debug)

Ran GDB and needed to handle incoming SIG32s for each new video that ran.

It didn't take long before I got a freeze in the video screensaver. No information in GDB. where provided in the tail I captured here:
GDB ES-Dev Screensaver Freeze Tail

Here are my current ES settings

A few things:

The information above is from latest ES-dev, latest kernel and software and an updated RetroPie setup script.
Prior to this test, I downgraded ES-dev to a build from May that I'm confident used to work, but now it doesn't (suggesting some non-ES influence is affecting ES-dev)
Events I recall since May that may be affecting: a) New RetroPie changes b) a sudo apt full-upgrade which updated the Kernel and VLC c) worked with the wine/box86 project which installed some development mesa drivers (apparently shouldn't affect current drivers, but unclear)

I haven't tried rolling back the Kernel yet, but that's next on my list. If that doesn't provide insights, I'll keep playing with older versions of ES-dev and maybe at least figure out what's not playing nicely with my system.

Does GDB "where" from my log provide any insights to you?

Hope this information is helpful.

mitu

@roslof said in Emulation station freezing:

Does GDB "where" from my log provide any insights to you?

The gdb trace shows the application is running somewhere in libvlccore, not sure if it's where the the crash/hang happens.
vcgencmd hanging may indicate a firmware/GPU crash or error.

Leaving aside a problem with ES, if you want to bisect a problem with the firmware you can use rpi-update and selectively update the just RPI firmware to see which revision is at fault (e.g. run it with sudo SKIP_KERNEL=1 rpi-update <git-revision>).

roslof

@mitu I confirmed that the initial freeze reported in November is not the same as the freeze I am experiencing. I also confirmed that this commit in ES-dev from 26-Feb is causing my RPi4B freeze. Only trying to understand why, since I'm certain that when I first received this commit, everything was fine with my system, and had been for quite sometime. I only started noticing random freezes after updating RPi software with sudo apt update and sudo apt full-upgrade on June 12th, 2021.

HYPOTHESIS #1: Something in the commit may be incompatible with the latest vlc-bin: 3.0.12-0+deb10u1+rpt2.

I ruled out overclocking, as I receive the freeze as often overclocking as not overclocking.

Kernel and firmware also may be ruled out. I rolled back to January, well before I noticed issues and freeze still occurs. Possible user-error on my part, but I doubt it.

I also compiled a version of vlc (version 11) but I failed to get it to render with the screensaver (or video snaps)... I may try again to prove or disprove my hypothesis above.

Beyond that, I'm out of ideas. Something changed since that commit. Not sure how many people are running ES-dev. The improvements are remarkable. But to remain stable, I'll have to stick with stable.

Hope this information is helpful.

mitu

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

roslof

@mitu said in Emulation station freezing:

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

That's correct. Standard RetroPie image and using fkms and my tests were with overclocking turned off, although I also tested with overclocking on, and the behavior is constant.

As instructed I added core_freq_min=500. I also insured all overclock-related settings were commented out.

There is no change in behavior. The screensaver crashed after about 20 videos. FWIW: The timing here is seemingly random. Could freeze immediately or could take up to an hour swapping videos every 10 seconds (during testing).

Adding: Every emulator (about 80-90) work perfectly. No known freezes or crashes. Only the screensaver exhibits the freeze.

roslof

Is anybody aware of a way to downgrade the vlc dependencies from latest (eg. downgrade vlc 3.0.12-0+deb10u1+rpt2) to any older version?

These are the three (3) main vlc-related dependancies for EmulationStation:

libvlc-dev libvlccore-dev vlc

For VLC alone, I was able to download vlc 3.0.11.1 and compile it without issue following these guidelines, but that isn't enough.

After running sudo make install files are copied around, but the system still sees VLC as the latest 3.0.12 -- So I'm not doing something right.

Then would also need to handle for libvlc-dev libvlccore-devof which I haven't even scratched the surface.

Seems rolling back VLC dependencies is not easy.

mitu

@roslof Compiling the sources from videolan.org will not get you the same thing as the packages distributed with Raspberry Pi OS. The version distributed by the RPI folks is patched with RPI specific support and is found at https://github.com/rpi-distro/vlc.

If you wish to downgrade your VLC version, you can download the VLC related packages from http://archive.raspberrypi.org/debian/pool/main/v/vlc/ and install them with dpkg -i.

I gave it a brief test and I don't get a crash/freeze with the stable version, but only briefly tested with the -dev version. Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

roslof

@mitu said in Emulation station freezing:

Top notch information here. Thank you!

Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

I could describe the freeze as this: When the problem kicks in, it's usually in-between videos. The screen is usually black and ES becomes unresponsive to any input. Network/SSH continues to work, but vcgencmd can no longer be used, as it will hang until Ctrl-C or it will lock-up the system completely.

At this point, sudo reboot from ssh will not work. You must kill power to the pi in order to restart it properly.

FWIW the issue with vcgencmd was easy to see when the bashwelcometweak is enabled. When starting a new ssh session, the script uses vcgencmd to read the CPU temperature, and gets stuck until Ctrl-C is pressed. htop shows a status of "D" (uninterruptible sleep).

roslof

@mitu said in Emulation station freezing:

The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

The timing of when the issue occurs is variable, and can be accelerated by shortening the time videos swap via ES Video Screensaver options, or by pressing right on the gamepad to advance to the next video. When I do this, I can usually reproduce the issue in less than 5 minutes. If I let it run natually with 35-second swap intervals, it could take an hour or more before the issue is seen.

In short: The faster you can cycle through videos, the more quickly the problem can be replicated.

dankcushions

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

roslof

@dankcushions said in Emulation station freezing:

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

roslof

@roslof said in Emulation station freezing:

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

Well that didn't take long. Same 'freeze' symptom in about the same amount of time (roughly 20-30 videos).

Will play around with older VLC builds and such later today and see what I can find. There is a possibility that the newer VLC isn't playing nice with some type of randomly selected video, so I'll see if I can catch something there, too.

mitu

I've been testing by leaving the screensaver for about 1h and then enabling the controls and switching manually between videos. So far I haven't encountered any freezes.
I only have a hand-full of videos, if the issue might be triggered by one of the videos, I may be spared because of the small selection I have available . FWIW I have gpu_mem set to 256, no overclock or other GPU/CPU related settings though.

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

roslof

@mitu said in Emulation station freezing:

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

I'll spend time later today trying this:

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

Assuming this is printing to standard output? When I see (an) affected video(s), I will isolate them in a single system and see if it's 100% reproducable.