Video Screensaver System Lockup
Starting a new thread for the gurus here to start tacklinging another problem for me...
I have the system running fine, start it up before I go to work, then after an 8 hour day I return home and the video is locked up, as is the whole system. I can't SSH into the system to run something like
topto see if there is a memory problem. Nothing works, so I have to pull the plug and restart the entire system.
Not exactly sure what I can do to reproduce the issue, so the first thing I'll need is a method for tracking. I'd like to see if it's doing it on the same video, or the same system, or after a certain period of time. Can you provide me tool for testing that? Does the current system keep a log? Something with a timestamp as well as a list of videos it's ran?
Here's the standard troubleshooting prerequisites:
- Raspberry Pi 3 not overclocked
- Original/Default Memory Split
- OS exists on 8GB SD Card
- Roms and Videos exist on 1tb External USB Drive
- EmulationStation v2.4.1RP
- RetroPie v ??? (how do you even check that?)
- RetroPie-Setup v4.2.12 (bda4178)
- Video Screensaver Settings:
Use OMX - ON
Screensaver Controls - ON (not sure what this does)
Show Game Info - Always
Stretch Video - Off
- ES VRAM Limit - 80Mb
- ES Use OMX Player - ON
@hansolo77 hopefully someone has a better answer for you, but one thing you can do is install retropie manager. Log into retropie manager right after you start up your pi. Open the logs tab. This will show you a lot of information and show a ton of info if and when the screensaver encounters errors or missingg files. Leave this running when you leave. When you return simply hit the browser refresh button, the log should update to the last error reported...:.i hope.....if the the pi is completely locked, it might not refresh and say the pi is offline, but it is something to try. I use this tool all the time to check for errors in my themes and game lists.
If you have gamelist that are "master" lists and contain a ton of entries for video that don't actually exist in your hardrive this could be an issue as the screensaver doesn't like that.
Also, this could be an issue with your hard drive, do you think it might be stalling out or stopping?
What happens if you turn your screensaver to black and let it run, does the system still lock up? If it does you can rule out the screensaver as the issue.
Good idea about changing the screensaver. I will try that. I used to have it on
DIMbefore the Video Screensaver became an option.
I also considered the problem might be the drive. Like maybe it's firmware is set to put the drive into a standby/sleep state after a certain period of time if a certain threshold of activity isn't met (like 200mb every minute, etc). That's why I kinda wanted to get a log of what it's playing, so I could figure out if maybe it was having trouble with a certain video size (resolution or filesize), bitrate, codec, etc etc.
I'm not familiar with "Retropie Manager"... where do I get that? And does it require my computer to remain on during the day as well My suspicions are "yes" so I can keep a live eye on the system. But if it requires a constant pressing of "refresh" or whatever, that probably won't help me if it locks up when I'm at work.
When I get some free time (should have a vacation coming up) I plan on dumping a file-copy of this drive to my server then formatting the drive and re-adding the roms and re-scraping so I get a good clean system (without all the crap
Last Playedentries from all my shutdown script testing). Hopefully that'll clear out the possibility of it trying to run a video that doesn't exist.
Yes your computer needs to be on the whole time with this open. No you do not need to continually refresh. When you get back to your computer just refresh one time and it will update the log for you.
Just installed RetroPie manager. We'll see if it helps me figure something out. Thanks for the suggestion. :)
Currently running Log:
Looks like it's only logging errors, which probably won't help if the system locks up and I can't get a refresh of the log... but it's also logging from
/opt/retropie/configs/all/emulationstation/es_log.txtwhich should be accessible were it not for getting overwritten when it reboots. I really think nailing this is going to require a more dynamic log though... have it record every video it's trying to play and copy that log to a location that won't get overwritten when the system reboots. Then I can go back in and see what it hung up on. Probably not possible without adding bits to the ES code though.
@hansolo77 It will log every error or missing file that the screensaver encounters. Trust me, when i was working with @pjft on testing the screensaver build i used this to clean up my gamelist, it logged every video that wouldn't play back properly or was missing.
You need to look at the log prior to rebooting. The trick would be to see if the log would refresh when your pi is locked up before you restart it.
I can't write code, so i am of no help there. I would start with things you can do yourself, I think the first thing i would do is turn your screen saver to black and let it run over night, see if it locks up on you. I have never left my pi run with the screensaver on for more than 4 or so hours, but i have never had it freeze up on me. I am guessing if it is freezing it is a bad video file.
I'd definitely start with dim and black screensavers first, to narrow down where the problem may be coming from.
You mentioned in the past that this didn't happen all the time as well, so I wonder if it is either a drive thing, or maybe even a corrupt video or sector that's trying to be read. I wouldn't necessarily expect that, but it could be the case.
In my case I often have it running for days with the screensaver and no troubles afterwards, so I suppose my setup can't be used as a benchmark. I'm using the exact same settings.
I can easily provide you with a binary that can log all the videos it plays, if we end up feeling it helps.
You may also try to choose VLC to play the screensaver videos, but I wouldn't do it without a fan... At least!
Anyway, let us know how this goes. Try the other screensavers for a while.
I know prior to getting the Video Screensaver running, I was using
DIMand never had any issues. It was only after I started using this one that I began having lockups. And from what I can remember, every time it was locked up, the video was in the middle of fading out to the next one. I don't think it was on the same video each time, but it might have been trying to load the same NEXT video.
I did set the screensaver to
DIMlast night and after 10 hours the Retropie Manager was able to refresh. My log doesn't show anything new, except for a bunch more of:
lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow! lvl1: OptionListComponent too narrow!
The system was also able to recover out of the sceensaver just fine. I'm gettingn ready to leave for work, so I have it set to the Video Screensaver now.
@hansolo77 I believe those errors are all related to missing or non-playable video files that are listed in your gamelist but either don't exist in the correct location, or are not playable.
Could be... I"m curious how the Video Screensaver actually works.. does it parse your entire gamelist across all systems, then just randomly pick one as the game it'll play? Or does it pull the gamelist then test with
ifto verify a video exists before trying to play it? If it's just a blind "pick and play" that can explain it not working. The best way to resolve that would be have it test to verify a video exists first...I know there are a LOT of games in my lists that don't have videos. GBA for instance has an entire romset, but only like 4 videos. So if it's trying to play a video from that system, and crashes because none exist.. that explains a lot.
Oh, I'm home from work now. :) I can't stay up too late playing around though, I work early in the morning (uhg). But as soon as I could, I refreshed the Log page on the Retropie Manager and everything is still running. Turned on the TV and sure enough, videos are still chugging a way. I've got an SSH terminal up running
toptoo, so I can see what was running if/when it crashes.
So far, it looks like this:
@hansolo77 the screensaver looks for all
<video>tags in all gamelists from all systems.
It does not look at all games and then try to play videos. so that is not an issue. It looks for the
<video>tags that have content. SO if you have empty video tags like this
<video />or like this
<video></video>they will be ignored. If your game does not have a video tag, they will be ignored.
If however you have a video
<video>does not exist.mp4</video>it will try to read this video and it will fail. After a set number of failures in a row, i think @pjft has this set to 20? he can verify, the screensaver will just turn itself to a black screen. When i was first testing this I was building my library and had a gamelist that had the wrong path to the videos, i experienced freezing when the screen would go black from failures, once i fixed the path, the problem was solved. If you use a pre-made gamelist that has video files listed, but you don't actually have those videos, this could be the cause of your freezing.
The best way to resolve that would be have it test to verify a video exists first
I disagree with this. The screensaver works nicely and I don't think we need to add complexity to it. Again, @pjft did a great job taking what @fieldofcows started and improving and implementing this. He knows the ins and outs of the build, but there have been very few users with issues regarding the screensaver, that is what leads me to believe it is either bad video files, gamelists with video tags that don't exist, or a hard drive issue with your build.
Keep the updates coming so we can get to the bottom of this. do you have an update to the retropie manager log? does it have a lot more of the same error?
I wonder then if there is a quick and dirty script I could use that just goes through and DOES report back
does not exist. Should be fairly easy to do, I'm just not a coder. Have it do essentially this:
- Load gamelist.xml
- Check game in xml to see if it has a path designate for a vider
- Check for existence of
blah blah.mp4in the path identified in step 2
no exist, report the gamelist.xml with the error, as well as the game entry, and the supposed video that should exist to a log file
- If exist or not, move on to the next game in the xml and test again.
- Do this for every game in the gamelist.xml
- Repeat for the next gamelist.xml
- When completed, I can read the log it created and see what it found. I can then replace the videos, or edit the xml to fix this issue.
Again, a simple bash script should be able to do that, I just wouldn't know how to write it. If somebody can do that for me, I can run the script and see what kind of results I turn up.
In the meanwhile, here is the current running ES_LOG:
Had to terminate this little expeiment briefly tonight to continue testing the Mausberry shutdown script tonight. Pretty sure that's taken care of (happy!) so now I can devote another night of having the video screensaver running.
One brief thing I noticed.. The
topI had running in the SSH terminal had locked up when I went into to try changing the script. Pressing CTRL+Z did nothing, as did anything else I pressed. I had to close the SSH terminal and open a new session. The system wasn't locked up though. I could access Retropie Manager, and the screensaver was still chugging away with 2 different videos at least while I tried to restart a session. So I don't know if having
toprunning till the Pi locks up will be much help if it locks up itself after only about 2 hours...
So, there shouldn't be a problem with missing videos in the tags - I mean, the worst that will happen is that it will default to the Black screensaver if it can't find a random video for 20 times in a row or something. The video screensaver will randomly pick a video tag, check if the video exists, and if it doesn't moves on to the next random one. If 20 times in a row it doesn't find a video, it'll default to the black screensaver.
What may be causing lockups may be a specific video or so that may have a weird format or encoding. You may to well in re-encoding your videos with Handbrake or so, see if it helps.
I am not a script guy per se - a simple bash will be able to do it, certainly, though.
The "OptionListComponent too narrow!" messages relate to the popup menu - start or select - on occasion. Don't quite know what triggers them.
You may also want to try out the Black screensaver, and see how that goes.
OK, the Video Screensaver has been running all night last night and all day while I was at work. The forums here say it was 20 hours ago.. so that's a record for me. It's probably one specific file that's causing the lockup. I just wish I knew which one it was so I could fix it. With no way to test that, I"m out of luck. I'd still like to be able to see a log of what it's doing, so I can see if it DOES in fact run into an issue and let's me fix it. Right now, I'm not getting anywhere with the tools I have. When I came home from work, the Video Screensaver is still running. My SSH connection crashed:
If that clock was right, it crashed around 5:50 this morning. I say "IF" because when I run it now, it says it's 22:35, which would be 10:35pm, but it's only 6:35pm. So I think it's 4 hours fast. Takeaway would make it 1:35am last night. I was asleep then, so I don't know what it could have crashed on. Might it be a session time out? Like after 2 hours of no input it disconnects me?
The only other bit of help is the
es_logfrom Retropie Manager:
Heading to bed... I've gone in and changed my settings in Putty to send keepalives every 1 second. I've also got
toprunning with 1 second updates. Probably overkill, but it'll help me identify the exact second the terminal is halting. I also adjusted the system time to be in the
US/Easternso now it's the right time. As of now, I'm still running the video screensaver without any issues. Really kinda pissed about that because it's not doing it now and I can't identify something that's causing it to crash! The
es_loghasn't updated yet either. One thing I have started to notice though is that every now and then there seems to be a large amperage draw on my power supply. I'm getting an occasional lightning bolt icon on my TV whenever it goes to load up a new video. It doesn't ALWAYS do this, so I'm not sure if it's happening on a specific system, or just randomly.
topthough, is it normal for ``top` to be taking up 100% cpu?
Just got home from my early shift. Screensaver is still running. There must be a bad video or path somewhere, and it's just working like it should. I would have liked to have been able to figure out what the problem is, but now it's not doing it. I mean, what if it's crashing when it's trying to find a video for the 20th time and goes to black? I can't test it! But hey, it's working and that's all that matters. I'm not going to keep an eye on logs and
topanymore. My bedroom is really hot now from the computer being on for 3 days keeping a constant watchful eye.
Here are the results of the SSH session. I thought setting up a keep alive would keep it from disconnecting. Must be something else that's causing the disconnect. Maybe it's related to the occasional lightning bolt icon, and the system is crashing but recovering with a reboot. The SSH disconnects, but the screensaver "appears" to be working because after rebooting it starts up again:
Also, my Retropie Manager, while still connected, shows NO new lines added to the
es_log, and those other ones it had before have also stopped showing up. I dunno what's up.
@hansolo77 Yeah. You're probably better off running all videos through Handbrake, re-encoding, seeing if per chance it ends up fixing any odd one.
The connection dying on you is quite normal, depending on the software and the keep-alive settings for the connection. I wouldn't read too much into it. Maybe Windows is going into deep sleep mode and that's when the connection drops. The error message seems to suggest that it's PuTTy who's dropping it.
As for the ES log, this one in particular started at 23:46:24 , so it should help you understand whether that matches the time you booted it up, or if it might have really restarted. Your PuTTy log also suggests that the session had been running for 22h at least, before the connection dropped.
Might also be helpful, if the date doesn't match what you expected, to check whether the date on your Pi is actually correct, as the Pi does not have an internal clock and reads the date from the internet, and adjusts based on the time zone you have set for your Pi. Easiest way to check is, when you go to the terminal or log in via SSH, to check the time it shows you and see if it matches your current time.
I'm spending the day today rebuilding my rom drive. I've got a backup made now so if anything happens I can always restore it. Not sure what's going on.