Emulation station freezing

roslof

@mitu said in Emulation station freezing:

The new version has a new 'sleep after X' min configuration that will 'freeze' the currently running video screensaver.

The timing of when the issue occurs is variable, and can be accelerated by shortening the time videos swap via ES Video Screensaver options, or by pressing right on the gamepad to advance to the next video. When I do this, I can usually reproduce the issue in less than 5 minutes. If I let it run natually with 35-second swap intervals, it could take an hour or more before the issue is seen.

In short: The faster you can cycle through videos, the more quickly the problem can be replicated.

dankcushions

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

roslof

@dankcushions said in Emulation station freezing:

this sounds similar to the memory leak issue, but that was with omx player only i think.. does increasing gpu_mem affect it?

https://retropie.org.uk/forum/topic/29735/emulatio-station-crashes/2?_=1627316238405&lang=en-GB

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

roslof

@roslof said in Emulation station freezing:

Thank you @dankcushions. I've been running gpu_mem=256 and just bumped up to 512. Will let things run for a bit and see what we get.

Well that didn't take long. Same 'freeze' symptom in about the same amount of time (roughly 20-30 videos).

Will play around with older VLC builds and such later today and see what I can find. There is a possibility that the newer VLC isn't playing nice with some type of randomly selected video, so I'll see if I can catch something there, too.

mitu

I've been testing by leaving the screensaver for about 1h and then enabling the controls and switching manually between videos. So far I haven't encountered any freezes.
I only have a hand-full of videos, if the issue might be triggered by one of the videos, I may be spared because of the small selection I have available . FWIW I have gpu_mem set to 256, no overclock or other GPU/CPU related settings though.

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

roslof

@mitu said in Emulation station freezing:

I'll let it run a bit more by itself, but either there's an issue with the video or I'm running with a different config (w.r.t. screensaver) that doesn't trigger the freeze.

I'll spend time later today trying this:

EDIT: I'm running the tests using this PR, which has the advantage of printing the current video played by the screensaver (needs --debug as a parameter).

Assuming this is printing to standard output? When I see (an) affected video(s), I will isolate them in a single system and see if it's 100% reproducable.

mitu

@roslof said in Emulation station freezing:

Assuming this is printing to standard output?

Yes, just run emulationstation --debug from a SSH session (optionally via screen) and it will print the games from where the videos are loaded.

mitu

@roslof Didn't manage to reproduce the freeze so far.
Can you post your ES settings (es_settings.cfg) ?

roslof

@mitu said in Emulation station freezing:

@roslof Didn't manage to reproduce the freeze so far.
Can you post your ES settings (es_settings.cfg) ?

Of course:

Here are my current ES settings

Also, I rebuilt ES-dev with the PR that you provided and gave it a few runs. Here is the [tail] output of three (3) separate runs. Nothing appears out of the ordinary:

https://pastebin.com/bNbWHPTC

I created a system with only the videos (and a custom gamelist.xml + roms) that appear at the end of each run, but they all render fine. I still have more testing to do, but I believe that with a small number of videos, the freeze doesn't occur. Will inform when I wrap with the testing.

FWIW: I'm currently running with screensaver indexing on and with a Max VRAM of 500, but I've also disabled indexing and adjusted VRAM to 250. The freeze still occurs, and the log only shows videos playing without indexing.

mitu

@roslof said in Emulation station freezing:

FWIW: I'm currently running with screensaver indexing on and with a Max VRAM of 500, but I've also disabled indexing and adjusted VRAM to 250. The freeze still occurs, and the log only shows videos playing without indexing.

Hm, never used the indexing option (something new), but I'll use your .cfg file and re-run the tests. There's no point in allocating more than 256 Mb of RAM for the GPU, this only affects the GPU's HW video decoder and - on the Pi4 - 192 Mb should be plenty for that.

mitu

@roslof I've re-run the test with your configuration, but haven't encountered any freezing so far. Doesn't seem like there's any setting that affects the behavior in EmulationStation.

If you can isolate the issue to a particular system, it would be easier to test it - if it's either related to the video file(s) or something else.

Can you run

raspinfo | head -n 60

and post the output ? Ever since the bluetooth issues that appeared only on certain (newer) Pi revisions, I'd like to see if you're using the same board revision.

roslof

@mitu said in Emulation station freezing:

Can you run
raspinfo | head -n 60
and post the output ?

Certainly, and thank you: https://pastebin.com/whWMWmwc

Will continue to test videos. If I can't find an issue with specific videos, I may devote time finding the offending code in the 7 files from the commit. I don't think there is anything inherently wrong with the code because this code used to work on my pi without issue. The issue started around the time that I performed a full upgrade with adb. Will continue to hammer on this.

The only other thing that I can think of, around the time of having these issues, is that I installed Wine + Box from this thread, which installs these dependencies:

binfmt-support cmake gtk2-engines-murrine libncurses5 libncursesw5 libssl1.0.2 libglu1-mesa zenity mesa-utils libinput10 libxkbcommon-x11-0 matchbox-window-manager xorg

I may uninstall each of these and see if it makes a difference.

mitu

@roslof said in Emulation station freezing:

The only other thing that I can think of, around the time of having these issues, is that I installed Wine + Box from this thread, which installs these dependencies:
[...]

That shouldn't cause a problem.

raspinfo | head -n 60
...

Linux retropie 5.10.52-v7l+ #1439 SMP Thu Jul 22 15:41:13 BST 2021 armv7l GNU/Linux

Videocore information
---------------------

Jul 21 2021 16:21:46 
Copyright (c) 2012 Broadcom
version 6a796bb0062a6c75191c57cba1c13f9300076d02 (clean) (release) (start)

however, the above tells me you've used rpi-update ?

roslof

@mitu said in Emulation station freezing:

however, the above tells me you've used rpi-update ?

Yes, but only recently. The issue existed prior.

This past weekend I was running latest stable:
Linux retropie 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux

Leaving aside a problem with ES, if you want to bisect a problem with the firmware you can use rpi-update and selectively update the just RPI firmware to see which revision is at fault (e.g. run it with sudo SKIP_KERNEL=1 rpi-update <git-revision>).

Starting playing with older and newer firmwares and landed on latest when I ran raspinfo.

Here is a pastebin of the output after downgrading back to stable: https://pastebin.com/RLwVHwGL

mitu

@roslof said in Emulation station freezing:

This past weekend I was running latest stable:
Linux retropie 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux

Yes, that's the version shipped as stable by Raspberry Pi OS. I upgraded with rpi-update and re-tested, but it seems the kernel/firmware version is not at fault - still can't get it to freeze.

roslof

@mitu said in Emulation station freezing:

Yes, that's the version shipped as stable by Raspberry Pi OS. I upgraded with rpi-update and re-tested, but it seems the kernel/firmware version is not at fault - still can't get it to freeze.

I appreciate you trying, mitu. I'll keep at it.

roslof

Issue still occurring. Thought it was fixed. Then I got another freeze.

After exhausting all other options, I manually downgraded VLC using RPi patched versions, and in the interest of certainty, went down to 3.0.11rpt3 (where I did most of my early testing with snap videos).

The process to downgrade was fairly simple, ensuring all downloaded deb packages match each other. So for 3.0.11rpt3, these were the files:

libvlc-bin_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-data_3.0.11-0+deb10u1+rpt3_all.deb
vlc-plugin-base_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-qt_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-video-output_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-bin_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlccore9_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlccore-dev_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlc5_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlc-dev_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-l10n_3.0.11-0+deb10u1+rpt3_all.deb
vlc-plugin-notify_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-samba_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-video-splitter_3.0.11-0+deb10u1+rpt3_armhf.deb

Then rebooted.
Verified that VLC running was indeed 3.0.11. Freeze still occurs.

Can rule out A LOT of things:

VLC is fine (verified with rollback)
Kernel/Firmware is fine (verified with rollback and roll-forward)
ES Settings are fine (others don't freeze with my settings
This is not an issue with the .mp4s (freeze is random and videos that were running during a freeze run perfectly fine otherwise).
The ES-dev commit that causes my freeze is not affecting others
Ruled out overclocking (disabled completely, and the config.txt file is the same I've used for years)

shrug

Thank you again @mitu and @mahoneyt944 for your information and valuable time. I'll keep on this. Not sure about next steps yet.

roslof

Actually @mitu I noticed some Debian VLC patched files that contain Debug/Symbols versions. If I install those, would I be able to get any useful data? If so, any advice on how I can get to that information?

mitu

@roslof said in Emulation station freezing:

If I install those, would I be able to get any useful data? If so, any advice on how I can get to that information?

The dbg flavour packages should be compiled with debug symbols, so it's easier to trace the execution through a debugger (i.e. gdb). I'm not sure how easy is to do it when used as library (as libvlc is used in EmulationStation).

If you could reproduce the freeze by just looping vlc over a set of video files, that would be easier - verbose logging in vlc prints a lot (a lot) of data and I reckon that the logging info would be useful as is, even without a debugger. It would be easier to pinpoint where in the code the freeze occurs and perhaps inquire over at the VLC repo or in the RPI forums.

roslof

@mitu said in Emulation station freezing:

If you could reproduce the freeze by just looping vlc over a set of video files, that would be easier - verbose logging in vlc prints a lot (a lot) of data and I reckon that the logging info would be useful as is, even without a debugger. It would be easier to pinpoint where in the code the freeze occurs and perhaps inquire over at the VLC repo or in the RPI forums.

I found the source of the problem and a work-around.

The ES-dev commit I identified earlier pjft/remove-vlc-guard adds a new option called VLC: Screensaver Video Resolution. Options range from "Original" (default) to "Max". The higher resolutions will enhance the sharpness of subtitles generated from the "Show Game Info on Screensaver" option. Desirable and an otherwise welcome addition to ES.

I had set this to "Max" for ultra-crisp subtitles, and it appears that there is some % chance that any scaled-up video will eventually freeze VLC. I lowered this to "High" and this also, eventually freezes VLC.

Setting to "Original" solves my freeze problem. Note: I haven't attempted "Low" or "Medium" yet.

So there it is. I know where the freeze is coming from. It's just not clear why.

Final thoughts for now:

I was stable using "Max" in the past. Still not sure why I'm impacted by this now, but it did seem to correlate with an apt full upgrade run in June
I tried a bunch of isolated Atari 2600 videos (whose default resolution is consistent) with the Max setting and [so far] cannot get a freeze. Granted, I've been fooled before in thinking things are stable only to freeze after a length of time. But if these don't freeze, it would suggest the source video format may influence the likelihood of a VLC freeze
Using this PR I learned which videos were playing during a freeze and also what the prior video was (in case the prior video was triggering the freeze on the next use of VLC). I plopped all the potentially "bad" videos into a single system and disabled all other systems... So about a dozen videos were cycling. Could not reproduce a freeze. Memory caching helping me out here, with such a small # of videos? There was nothing special or interesting about these 12 videos, which otherwise render just fine.

Bottom Line: I'm certain that if ES-dev scales-up resolution on a video there is a percentage chance that VLC will freeze during the screensaver. I am hypothesizing that some source videos may be more or less likely to freeze, but unclear. I've seen stability when running a small number of videos (~100-200) and imminent freeze when using several from multiple systems.

For now, I'm keeping resolution set as "Original", where subtitles will vary in crispness depending on the video quality.