Emulation station freezing

roslof

@mitu since ES stable seems fine for me, I went ahead and created a debug version of EmulationStation-dev leveraging the existing RetroPie script and adding params+=(-DCMAKE_BUILD_TYPE=Debug)

Ran GDB and needed to handle incoming SIG32s for each new video that ran.

It didn't take long before I got a freeze in the video screensaver. No information in GDB. where provided in the tail I captured here:
GDB ES-Dev Screensaver Freeze Tail

Here are my current ES settings

A few things:

The information above is from latest ES-dev, latest kernel and software and an updated RetroPie setup script.
Prior to this test, I downgraded ES-dev to a build from May that I'm confident used to work, but now it doesn't (suggesting some non-ES influence is affecting ES-dev)
Events I recall since May that may be affecting: a) New RetroPie changes b) a sudo apt full-upgrade which updated the Kernel and VLC c) worked with the wine/box86 project which installed some development mesa drivers (apparently shouldn't affect current drivers, but unclear)

I haven't tried rolling back the Kernel yet, but that's next on my list. If that doesn't provide insights, I'll keep playing with older versions of ES-dev and maybe at least figure out what's not playing nicely with my system.

Does GDB "where" from my log provide any insights to you?

Hope this information is helpful.

mitu

@roslof said in Emulation station freezing:

Does GDB "where" from my log provide any insights to you?

The gdb trace shows the application is running somewhere in libvlccore, not sure if it's where the the crash/hang happens.
vcgencmd hanging may indicate a firmware/GPU crash or error.

Leaving aside a problem with ES, if you want to bisect a problem with the firmware you can use rpi-update and selectively update the just RPI firmware to see which revision is at fault (e.g. run it with sudo SKIP_KERNEL=1 rpi-update <git-revision>).

roslof

@mitu I confirmed that the initial freeze reported in November is not the same as the freeze I am experiencing. I also confirmed that this commit in ES-dev from 26-Feb is causing my RPi4B freeze. Only trying to understand why, since I'm certain that when I first received this commit, everything was fine with my system, and had been for quite sometime. I only started noticing random freezes after updating RPi software with sudo apt update and sudo apt full-upgrade on June 12th, 2021.

HYPOTHESIS #1: Something in the commit may be incompatible with the latest vlc-bin: 3.0.12-0+deb10u1+rpt2.

I ruled out overclocking, as I receive the freeze as often overclocking as not overclocking.

Kernel and firmware also may be ruled out. I rolled back to January, well before I noticed issues and freeze still occurs. Possible user-error on my part, but I doubt it.

I also compiled a version of vlc (version 11) but I failed to get it to render with the screensaver (or video snaps)... I may try again to prove or disprove my hypothesis above.

Beyond that, I'm out of ideas. Something changed since that commit. Not sure how many people are running ES-dev. The improvements are remarkable. But to remain stable, I'll have to stick with stable.

Hope this information is helpful.

mitu

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

roslof

@mitu said in Emulation station freezing:

@roslof Thanks for all the testing.
This is with a standard RetroPie image - i.e. no overclocking or switching on the kms overlay ? As a quick test, can you try adding core_freq_min=500 to config.txt and see if the issue re-occurs ?

That's correct. Standard RetroPie image and using fkms and my tests were with overclocking turned off, although I also tested with overclocking on, and the behavior is constant.

As instructed I added core_freq_min=500. I also insured all overclock-related settings were commented out.

There is no change in behavior. The screensaver crashed after about 20 videos. FWIW: The timing here is seemingly random. Could freeze immediately or could take up to an hour swapping videos every 10 seconds (during testing).

Adding: Every emulator (about 80-90) work perfectly. No known freezes or crashes. Only the screensaver exhibits the freeze.

roslof

Is anybody aware of a way to downgrade the vlc dependencies from latest (eg. downgrade vlc 3.0.12-0+deb10u1+rpt2) to any older version?

These are the three (3) main vlc-related dependancies for EmulationStation:

libvlc-dev libvlccore-dev vlc

For VLC alone, I was able to download vlc 3.0.11.1 and compile it without issue following these guidelines, but that isn't enough.

After running sudo make install files are copied around, but the system still sees VLC as the latest 3.0.12 -- So I'm not doing something right.

Then would also need to handle for libvlc-dev libvlccore-devof which I haven't even scratched the surface.

Seems rolling back VLC dependencies is not easy.

mitu

@roslof Compiling the sources from videolan.org will not get you the same thing as the packages distributed with Raspberry Pi OS. The version distributed by the RPI folks is patched with RPI specific support and is found at https://github.com/rpi-distro/vlc.

If you wish to downgrade your VLC version, you can download the VLC related packages from http://archive.raspberrypi.org/debian/pool/main/v/vlc/ and install them with dpkg -i.

I gave it a brief test and I don't get a crash/freeze with the stable version, but only briefly tested with the -dev version. Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

roslof

@mitu said in Emulation station freezing:

Top notch information here. Thank you!

Just to make sure, by freeze you mean ES is not responsive to any input ? The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

I could describe the freeze as this: When the problem kicks in, it's usually in-between videos. The screen is usually black and ES becomes unresponsive to any input. Network/SSH continues to work, but vcgencmd can no longer be used, as it will hang until Ctrl-C or it will lock-up the system completely.

At this point, sudo reboot from ssh will not work. You must kill power to the pi in order to restart it properly.

FWIW the issue with vcgencmd was easy to see when the bashwelcometweak is enabled. When starting a new ssh session, the script uses vcgencmd to read the CPU temperature, and gets stuck until Ctrl-C is pressed. htop shows a status of "D" (uninterruptible sleep).

roslof

@mitu said in Emulation station freezing:

The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

The timing of when the issue occurs is variable, and can be accelerated by shortening the time videos swap via ES Video Screensaver options, or by pressing right on the gamepad to advance to the next video. When I do this, I can usually reproduce the issue in less than 5 minutes. If I let it run natually with 35-second swap intervals, it could take an hour or more before the issue is seen.

In short: The faster you can cycle through videos, the more quickly the problem can be replicated.

dankcushions

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

roslof

@dankcushions said in Emulation station freezing:

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

roslof

@roslof said in Emulation station freezing:

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

Well that didn't take long. Same 'freeze' symptom in about the same amount of time (roughly 20-30 videos).

Will play around with older VLC builds and such later today and see what I can find. There is a possibility that the newer VLC isn't playing nice with some type of randomly selected video, so I'll see if I can catch something there, too.

mitu

I've been testing by leaving the screensaver for about 1h and then enabling the controls and switching manually between videos. So far I haven't encountered any freezes.
I only have a hand-full of videos, if the issue might be triggered by one of the videos, I may be spared because of the small selection I have available . FWIW I have gpu_mem set to 256, no overclock or other GPU/CPU related settings though.

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

roslof

@mitu said in Emulation station freezing:

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

I'll spend time later today trying this:

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

Assuming this is printing to standard output? When I see (an) affected video(s), I will isolate them in a single system and see if it's 100% reproducable.

mitu

@roslof said in Emulation station freezing:

Assuming this is printing to standard output?

Yes, just run emulationstation --debug from a SSH session (optionally via screen) and it will print the games from where the videos are loaded.

mitu

@roslof Didn't manage to reproduce the freeze so far.
Can you post your ES settings (es_settings.cfg) ?

roslof

@mitu said in Emulation station freezing:

@roslof Didn't manage to reproduce the freeze so far.
Can you post your ES settings (es_settings.cfg) ?

Of course:

Here are my current ES settings

Also, I rebuilt ES-dev with the PR that you provided and gave it a few runs. Here is the [tail] output of three (3) separate runs. Nothing appears out of the ordinary:

https://pastebin.com/bNbWHPTC

I created a system with only the videos (and a custom gamelist.xml + roms) that appear at the end of each run, but they all render fine. I still have more testing to do, but I believe that with a small number of videos, the freeze doesn't occur. Will inform when I wrap with the testing.

FWIW: I'm currently running with screensaver indexing on and with a Max VRAM of 500, but I've also disabled indexing and adjusted VRAM to 250. The freeze still occurs, and the log only shows videos playing without indexing.

mitu

@roslof said in Emulation station freezing:

FWIW: I'm currently running with screensaver indexing on and with a Max VRAM of 500, but I've also disabled indexing and adjusted VRAM to 250. The freeze still occurs, and the log only shows videos playing without indexing.

Hm, never used the indexing option (something new), but I'll use your .cfg file and re-run the tests. There's no point in allocating more than 256 Mb of RAM for the GPU, this only affects the GPU's HW video decoder and - on the Pi4 - 192 Mb should be plenty for that.

mitu

@roslof I've re-run the test with your configuration, but haven't encountered any freezing so far. Doesn't seem like there's any setting that affects the behavior in EmulationStation.

If you can isolate the issue to a particular system, it would be easier to test it - if it's either related to the video file(s) or something else.

Can you run

raspinfo | head -n 60

and post the output ? Ever since the bluetooth issues that appeared only on certain (newer) Pi revisions, I'd like to see if you're using the same board revision.

roslof

@mitu said in Emulation station freezing:

Can you run
raspinfo | head -n 60
and post the output ?

Certainly, and thank you: https://pastebin.com/whWMWmwc

Will continue to test videos. If I can't find an issue with specific videos, I may devote time finding the offending code in the 7 files from the commit. I don't think there is anything inherently wrong with the code because this code used to work on my pi without issue. The issue started around the time that I performed a full upgrade with adb. Will continue to hammer on this.

The only other thing that I can think of, around the time of having these issues, is that I installed Wine + Box from this thread, which installs these dependencies:

binfmt-support cmake gtk2-engines-murrine libncurses5 libncursesw5 libssl1.0.2 libglu1-mesa zenity mesa-utils libinput10 libxkbcommon-x11-0 matchbox-window-manager xorg

I may uninstall each of these and see if it makes a difference.