Emulation station freezing

roslof

@mitu said in Emulation station freezing:

@roslof said in Emulation station freezing:

Q: When updating Raspberry Pi software, is the output logged somewhere? I'd like to see what changed. It had been awhile, so A LOT changed.

The updates are logged to /var/log/apt/history.log or one of the gz files in the same folder.

Will investigate now. Thank you mitu.

roslof

Logs show that I ran apt full-upgrade on 2021-06-12.
Issues started on that date. Should rule-out an ES update, correct? I upgraded RetroPie regularly.

Kernel was updated
12 VLC related updates
...and lots more :|

Full log from history.log.1

Might start with rolling back VLC... then test for awhile
Then try ~~to find the previous stable kernel and try that next~~ my previous kernel (which, if I'm reading the log correctly was kernel dated: 20210303.

Also, this thread...
A change from Dec/Jan affected VLC. Should be unrelated, if my previous kernel was working well (20210303)...

mahoneyt944

I can't confirm what change occured exactly as I didn't dig that far. Though I did test different combinations of updates back when I discovered the effected setting, with and without updating the kernel, etc. Without testing every commit from that time one by one, it seemed to follow the ES update not the kernel. After removing the before mentioned gpu setting the freezing has been resolved and that was good enough for me. I've updated again fully, just 2 days ago, and I am still not experiencing issues. Mind you my build runs literally 24/7, so I'm confident in the results with my build.

This was just my experience though, if nothing else maybe a starting point for someone to dig further.

4.6.8 worked fine. 4.7.1 had freezes with the same gpu setting. So any commits in that time should be suspect. ES would be my target since it's all that froze for me. The pi still works fine other than that, as in its still working over network etc.

roslof

@mahoneyt944 said in Emulation station freezing:

4.6.8 worked fine. 4.7.1 had freezes with the same gpu setting. So any commits in that time should be suspect. ES would be my target since it's all that froze for me. The pi still works fine other than that, as in its still working over network etc.

Thank you @mahoneyt944
Are you currently running the experimental ES or non-experimental?

mahoneyt944

@roslof I believe I updated from binary. So latest stable.

roslof

@mahoneyt944 said in Emulation station freezing:

@roslof I believe I updated from binary. So latest stable.

Again, helpful. It's still early, but my issue is starting to look like a source-built ES-dev build. The only way I'm testing this is by simply letting the video screensaver run until it freezes, or until I give a ton of time.

Like you, @mahoneyt944, I appear stable with non-dev (so good news for RetroPie in general as dev is experimental).

If it turns out to be source-built ES-dev, I'll zero-in on whether or not it's how it's being built, or if a breaking change came in.

UPDATE: Freezes with latest pre-built binary of ES-dev. Convinced issue is indeed with EmulationStation-dev. Will manually build from older commits (looks like May timeframe has some good targets). It also would appear that my issue is something fairly new, as this thread started in Nov 2020...

UPDATE #2: When the video freezes, it appears that vcgencmd is hanging as seen in htop (status D):

D - uninterruptible sleep (usually IO)
Uninterruptable processes are USUALLY waiting for I/O following a page fault. The process/task cannot be interrupted in this state, because it can't handle any signals; if it did, another page fault would happen and it would be back where it was.

mahoneyt944

@roslof my arcade cabinet "retropie" runs 24/7 so I would always find it frozen when returning after a period of time on the screensaver "slideshow". Once I started testing things though I found ES would sometimes randomly freeze when scrolling menus / navigating around. So I'm not sure it's limited to screensavers. In my case it seemed to be anytime the GPU was getting a jolt of action within ES.

roslof

@mahoneyt944 and you were overclocked for GPU?

mahoneyt944

@roslof I was using gpu_freq=500 before I updated. Never an issue. After updating, I could no longer use this setting in config.txt

There are other setting that effect GPU, so there maybe others effected too.

roslof

@mitu since ES stable seems fine for me, I went ahead and created a debug version of EmulationStation-dev leveraging the existing RetroPie script and adding params+=(-DCMAKE_BUILD_TYPE=Debug)

Ran GDB and needed to handle incoming SIG32s for each new video that ran.

It didn't take long before I got a freeze in the video screensaver. No information in GDB. where provided in the tail I captured here:
GDB ES-Dev Screensaver Freeze Tail

Here are my current ES settings

A few things:

The information above is from latest ES-dev, latest kernel and software and an updated RetroPie setup script.
Prior to this test, I downgraded ES-dev to a build from May that I'm confident used to work, but now it doesn't (suggesting some non-ES influence is affecting ES-dev)
Events I recall since May that may be affecting: a) New RetroPie changes b) a sudo apt full-upgrade which updated the Kernel and VLC c) worked with the wine/box86 project which installed some development mesa drivers (apparently shouldn't affect current drivers, but unclear)

I haven't tried rolling back the Kernel yet, but that's next on my list. If that doesn't provide insights, I'll keep playing with older versions of ES-dev and maybe at least figure out what's not playing nicely with my system.

Does GDB "where" from my log provide any insights to you?

Hope this information is helpful.

mitu

@roslof said in Emulation station freezing:

Does GDB "where" from my log provide any insights to you?

The gdb trace shows the application is running somewhere in libvlccore, not sure if it's where the the crash/hang happens.
vcgencmd hanging may indicate a firmware/GPU crash or error.

Leaving aside a problem with ES, if you want to bisect a problem with the firmware you can use rpi-update and selectively update the just RPI firmware to see which revision is at fault (e.g. run it with sudo SKIP_KERNEL=1 rpi-update <git-revision>).

roslof

@mitu I confirmed that the initial freeze reported in November is not the same as the freeze I am experiencing. I also confirmed that this commit in ES-dev from 26-Feb is causing my RPi4B freeze. Only trying to understand why, since I'm certain that when I first received this commit, everything was fine with my system, and had been for quite sometime. I only started noticing random freezes after updating RPi software with sudo apt update and sudo apt full-upgrade on June 12th, 2021.

HYPOTHESIS #1: Something in the commit may be incompatible with the latest vlc-bin: 3.0.12-0+deb10u1+rpt2.

I ruled out overclocking, as I receive the freeze as often overclocking as not overclocking.

Kernel and firmware also may be ruled out. I rolled back to January, well before I noticed issues and freeze still occurs. Possible user-error on my part, but I doubt it.

I also compiled a version of vlc (version 11) but I failed to get it to render with the screensaver (or video snaps)... I may try again to prove or disprove my hypothesis above.

Beyond that, I'm out of ideas. Something changed since that commit. Not sure how many people are running ES-dev. The improvements are remarkable. But to remain stable, I'll have to stick with stable.

Hope this information is helpful.

mitu

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

roslof

@mitu said in Emulation station freezing:

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

That's correct. Standard RetroPie image and using fkms and my tests were with overclocking turned off, although I also tested with overclocking on, and the behavior is constant.

As instructed I added core_freq_min=500. I also insured all overclock-related settings were commented out.

There is no change in behavior. The screensaver crashed after about 20 videos. FWIW: The timing here is seemingly random. Could freeze immediately or could take up to an hour swapping videos every 10 seconds (during testing).

Adding: Every emulator (about 80-90) work perfectly. No known freezes or crashes. Only the screensaver exhibits the freeze.

roslof

Is anybody aware of a way to downgrade the vlc dependencies from latest (eg. downgrade vlc 3.0.12-0+deb10u1+rpt2) to any older version?

These are the three (3) main vlc-related dependancies for EmulationStation:

libvlc-dev libvlccore-dev vlc

For VLC alone, I was able to download vlc 3.0.11.1 and compile it without issue following these guidelines, but that isn't enough.

After running sudo make install files are copied around, but the system still sees VLC as the latest 3.0.12 -- So I'm not doing something right.

Then would also need to handle for libvlc-dev libvlccore-devof which I haven't even scratched the surface.

Seems rolling back VLC dependencies is not easy.

mitu

@roslof Compiling the sources from videolan.org will not get you the same thing as the packages distributed with Raspberry Pi OS. The version distributed by the RPI folks is patched with RPI specific support and is found at https://github.com/rpi-distro/vlc.

If you wish to downgrade your VLC version, you can download the VLC related packages from http://archive.raspberrypi.org/debian/pool/main/v/vlc/ and install them with dpkg -i.

I gave it a brief test and I don't get a crash/freeze with the stable version, but only briefly tested with the -dev version. Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

roslof

@mitu said in Emulation station freezing:

Top notch information here. Thank you!

Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

I could describe the freeze as this: When the problem kicks in, it's usually in-between videos. The screen is usually black and ES becomes unresponsive to any input. Network/SSH continues to work, but vcgencmd can no longer be used, as it will hang until Ctrl-C or it will lock-up the system completely.

At this point, sudo reboot from ssh will not work. You must kill power to the pi in order to restart it properly.

FWIW the issue with vcgencmd was easy to see when the bashwelcometweak is enabled. When starting a new ssh session, the script uses vcgencmd to read the CPU temperature, and gets stuck until Ctrl-C is pressed. htop shows a status of "D" (uninterruptible sleep).

roslof

@mitu said in Emulation station freezing:

The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

The timing of when the issue occurs is variable, and can be accelerated by shortening the time videos swap via ES Video Screensaver options, or by pressing right on the gamepad to advance to the next video. When I do this, I can usually reproduce the issue in less than 5 minutes. If I let it run natually with 35-second swap intervals, it could take an hour or more before the issue is seen.

In short: The faster you can cycle through videos, the more quickly the problem can be replicated.

dankcushions

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

roslof

@dankcushions said in Emulation station freezing:

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.