Emulation station freezing

roslof

@mitu said in Emulation station freezing:

however, the above tells me you've used rpi-update ?

Yes, but only recently. The issue existed prior.

This past weekend I was running latest stable:
Linux retropie 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux

Leaving aside a problem with ES, if you want to bisect a problem with the firmware you can use rpi-update and selectively update the just RPI firmware to see which revision is at fault (e.g. run it with sudo SKIP_KERNEL=1 rpi-update <git-revision>).

Starting playing with older and newer firmwares and landed on latest when I ran raspinfo.

Here is a pastebin of the output after downgrading back to stable: https://pastebin.com/RLwVHwGL

mitu

@roslof said in Emulation station freezing:

This past weekend I was running latest stable:
Linux retropie 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux

Yes, that's the version shipped as stable by Raspberry Pi OS. I upgraded with rpi-update and re-tested, but it seems the kernel/firmware version is not at fault - still can't get it to freeze.

roslof

@mitu said in Emulation station freezing:

Yes, that's the version shipped as stable by Raspberry Pi OS. I upgraded with rpi-update and re-tested, but it seems the kernel/firmware version is not at fault - still can't get it to freeze.

I appreciate you trying, mitu. I'll keep at it.

roslof

Issue still occurring. Thought it was fixed. Then I got another freeze.

After exhausting all other options, I manually downgraded VLC using RPi patched versions, and in the interest of certainty, went down to 3.0.11rpt3 (where I did most of my early testing with snap videos).

The process to downgrade was fairly simple, ensuring all downloaded deb packages match each other. So for 3.0.11rpt3, these were the files:

libvlc-bin_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-data_3.0.11-0+deb10u1+rpt3_all.deb
vlc-plugin-base_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-qt_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-video-output_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-bin_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlccore9_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlccore-dev_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlc5_3.0.11-0+deb10u1+rpt3_armhf.deb
libvlc-dev_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-l10n_3.0.11-0+deb10u1+rpt3_all.deb
vlc-plugin-notify_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-samba_3.0.11-0+deb10u1+rpt3_armhf.deb
vlc-plugin-video-splitter_3.0.11-0+deb10u1+rpt3_armhf.deb

Then rebooted.
Verified that VLC running was indeed 3.0.11. Freeze still occurs.

Can rule out A LOT of things:

VLC is fine (verified with rollback)
Kernel/Firmware is fine (verified with rollback and roll-forward)
ES Settings are fine (others don't freeze with my settings
This is not an issue with the .mp4s (freeze is random and videos that were running during a freeze run perfectly fine otherwise).
The ES-dev commit that causes my freeze is not affecting others
Ruled out overclocking (disabled completely, and the config.txt file is the same I've used for years)

shrug

Thank you again @mitu and @mahoneyt944 for your information and valuable time. I'll keep on this. Not sure about next steps yet.

roslof

Actually @mitu I noticed some Debian VLC patched files that contain Debug/Symbols versions. If I install those, would I be able to get any useful data? If so, any advice on how I can get to that information?

mitu

@roslof said in Emulation station freezing:

If I install those, would I be able to get any useful data? If so, any advice on how I can get to that information?

The dbg flavour packages should be compiled with debug symbols, so it's easier to trace the execution through a debugger (i.e. gdb). I'm not sure how easy is to do it when used as library (as libvlc is used in EmulationStation).

If you could reproduce the freeze by just looping vlc over a set of video files, that would be easier - verbose logging in vlc prints a lot (a lot) of data and I reckon that the logging info would be useful as is, even without a debugger. It would be easier to pinpoint where in the code the freeze occurs and perhaps inquire over at the VLC repo or in the RPI forums.

roslof

@mitu said in Emulation station freezing:

If you could reproduce the freeze by just looping vlc over a set of video files, that would be easier - verbose logging in vlc prints a lot (a lot) of data and I reckon that the logging info would be useful as is, even without a debugger. It would be easier to pinpoint where in the code the freeze occurs and perhaps inquire over at the VLC repo or in the RPI forums.

I found the source of the problem and a work-around.

The ES-dev commit I identified earlier pjft/remove-vlc-guard adds a new option called VLC: Screensaver Video Resolution. Options range from "Original" (default) to "Max". The higher resolutions will enhance the sharpness of subtitles generated from the "Show Game Info on Screensaver" option. Desirable and an otherwise welcome addition to ES.

I had set this to "Max" for ultra-crisp subtitles, and it appears that there is some % chance that any scaled-up video will eventually freeze VLC. I lowered this to "High" and this also, eventually freezes VLC.

Setting to "Original" solves my freeze problem. Note: I haven't attempted "Low" or "Medium" yet.

So there it is. I know where the freeze is coming from. It's just not clear why.

Final thoughts for now:

I was stable using "Max" in the past. Still not sure why I'm impacted by this now, but it did seem to correlate with an apt full upgrade run in June
I tried a bunch of isolated Atari 2600 videos (whose default resolution is consistent) with the Max setting and [so far] cannot get a freeze. Granted, I've been fooled before in thinking things are stable only to freeze after a length of time. But if these don't freeze, it would suggest the source video format may influence the likelihood of a VLC freeze
Using this PR I learned which videos were playing during a freeze and also what the prior video was (in case the prior video was triggering the freeze on the next use of VLC). I plopped all the potentially "bad" videos into a single system and disabled all other systems... So about a dozen videos were cycling. Could not reproduce a freeze. Memory caching helping me out here, with such a small # of videos? There was nothing special or interesting about these 12 videos, which otherwise render just fine.

Bottom Line: I'm certain that if ES-dev scales-up resolution on a video there is a percentage chance that VLC will freeze during the screensaver. I am hypothesizing that some source videos may be more or less likely to freeze, but unclear. I've seen stability when running a small number of videos (~100-200) and imminent freeze when using several from multiple systems.

For now, I'm keeping resolution set as "Original", where subtitles will vary in crispness depending on the video quality.

juggalo_love

@roslof I'm running into the same problem that you had originally with emulation station hanging and freezing. I'm trying to find the .cpp file that you noted so I can amend the vlc options to "original" but am having no luck. Can you point me to the directory where I would find the file or the command line prompt to load it