Collections Causing Lockups/White Backgrounds

hansolo77

Heh even better actually:
VRAM clearing threshold (less is better)

pjft

Wonderful!

I will doublecheck this when I get home, to see if the games presented truly have a "Times Played" entry. Since I'm using the Video Screensaver now, I think I might have started a game or 2 from within the screensaver, without actually playing it. Does the metadata for "Times Played", etc get updated as soon as the the game launches (when the Run Command is still present) or when the game returns to ES?

It will get updated on return to ES, but regardless of whether the emulator is actually launched. So, only if you had somehow rebooted the Pi without returning to ES, the moment you select "launch" and it exists ES, on return it will update it.

As I recall, the games were listed initially in aphabetical order. I've played through the entire game of Doom (port) and have started playing Doom 2 (port). When I opened "Last Played", it showed Doom as the first entry with Doom 2 under it. The most recent game I had played was Sonic 1 (Genesis). This collection only showed I had played like 15 games total, which isn't accurate either. But when I used the "select" menu to change the sort, it didn't save it when I exited. I'll have to test a bit more, as it might have been because I exited with the B button rather than going to the "back" menu button.

Hm. Unfortunately, whether you click "back" or "ok" on that menu, the sorting should still apply (it is quite dumb that there's no "Cancel" option, I know...).

As for it only showing 15 games in total, that would certainly be weird, unless you don't have metadata being saved?

There's a reasonably easy test you can perform: go to the "All games" system and sort by "Times Played, Descending". It should show all the games sorted by the number of times you have played them, as present in the metadata.

If there's a discrepancy in the games listed with more than "zero" times, and the games in the "Last Played" list (they won't necessarily be sorted the same way, mind you, as Last Played sorts by date (according to the metadata, once again), whereas here you're sorting by the actual number), I'd want to look into that.

I will check this when I get home, and send gamelists if you think it'll help.

We'll see if we need to. I know it's a pain to get the gamelists one by one, so I'd want to avoid that if we can :)

It may have been something else that caused the crash. The crash was caused ES to close to terminal, not "lockup". But it makes a lot of sense, that if you don't have the Favorites collection enabled prior to adding favorites it would crash.

Yeah, crashes manifest themselves in different ways, depending on the type. It can crash to the terminal, it can stay with the video (as OMX Player is a separate process)... In principle, it shouldn't have crashed in this scenario, as the collection does exist, it's just not displayed. That being said, I'd not be shocked that there's something I've missed there, so I will look into that. This is an excellent test scenario I had not gone through.

Good stuff. I assumed that the MORE is usally better. So by increasing VRAM to the max would just make the system that much better. But if the way it works is by telling ES how much memory you want it to use so it can clear out the memory before going over the limit, then REDUCING makes sense. I think I better way to handle this would be to change the menu wording a bit. Maybe something like Max VRAM available to ES (less is better!)

Yeah. I'll figure something out. Eventually. I am not that familiar with that code, but we should be able to prevent users from shooting themselves in the foot. For sure.

Yes I am using the OMX Player. It was the latest "fix" I needed to get the video screenshots to work right. Without it, I was getting a lot of green screens, and massive pixelation. Using the OMX Player solved all of that. I will monitor TOP and and see if ES is no longer running the next time that happens.

Agreed, OMX Player is a great help for videos on the Pi! I bet that that's what happening then, which unfortunately doesn't help us much as it doesn't bring us any closer to solving this. :) That being said, you mentioned that you had had some lockups prior to this update? Were they similar - meaning, video stays playing, nothing is responsive?

Thanks!

hansolo77

@pjft said in Collections Causing Lockups/White Backgrounds:

It will get updated on return to ES, but regardless of whether the emulator is actually launched. So, only if you had somehow rebooted the Pi without returning to ES, the moment you select "launch" and it exists ES, on return it will update it.

This sounds good to me. The random game in "Last Played" collection I don't remember playing must have just been one of those times I simply started it from the video screensaver as a test.

Hm. Unfortunately, whether you click "back" or "ok" on that menu, the sorting should still apply (it is quite dumb that there's no "Cancel" option, I know...).

As for it only showing 15 games in total, that would certainly be weird, unless you don't have metadata being saved?

I have it set to Parse gamelists only to "ON" so it doesn't generate new gamelists on bootup. Then I have it set to Save Metadata on Exit also set to "ON". This actually just got me thinking.. If it doesn't save metadata until exit (from ES right?), maybe that's why the "Last Played" collection isn't getting updated? I've been playing around with keeping the screensaver running all day, so nothing gets saved until it crashes. But the "Last Played" or "Times Played" flags in the metadata should be getting added immediately right?

There's a reasonably easy test you can perform: go to the "All games" system and sort by "Times Played, Descending". It should show all the games sorted by the number of times you have played them, as present in the metadata.

If there's a discrepancy in the games listed with more than "zero" times, and the games in the "Last Played" list (they won't necessarily be sorted the same way, mind you, as Last Played sorts by date (according to the metadata, once again), whereas here you're sorting by the actual number), I'd want to look into that.

I actually just did a little experiment when I got home. I re-enabled the Last Played collection and looked. It's now showing 24 games. I know for a fact I've played a LOT more than that. But still, going forward. I noticed it appeared to be sorted by system. That might have just been coincidence. But I remember playing Doom 2 last night because the RetroAchievements site was having issues, so I just played that. It doesn't show up on my list till WAY down at the bottom:
alt text
That "The Battle of Olympus" I don't remember ever playing... It looks like it might be sorted by system, but it's not alphabetical as I originally though. "NES" comes after "GENESIS", and "PINBALL" should come after "METROID", so I guess that's right. To further the experiment, after taking that snapshot, I loaded up Doom 2, then backed out, and re-opened that collection. Doom 2 now shows up at the top:

alt text
So apparently it's working right?

Yeah. I'll figure something out. Eventually. I am not that familiar with that code, but we should be able to prevent users from shooting themselves in the foot. For sure.

I will say, that in the short time I've been playing around with it now with the VRAM set to 80, it definitely "appears" to be more stable. It hasn't had a hard lockup, crash back to terminal, or have white background screens.

Agreed, OMX Player is a great help for videos on the Pi! I bet that that's what happening then, which unfortunately doesn't help us much as it doesn't bring us any closer to solving this. :) That being said, you mentioned that you had had some lockups prior to this update? Were they similar - meaning, video stays playing, nothing is responsive?

Yeah I will try to keep a close eye on this and see if I can get it to trigger. The lockups I was having (video continuing to play with no input response) seem to be cleared out for now. I almost want to try to put my VRAM back up to 100 to see if I can trigger it again, just to see if ES is still running. At this point, without it crashing anymore, I don't really have anything more to try. I've purposely tried to spam the lists, scrolling really fast, etc because it really felt like it was doing it if I didn't give the video time to load up. Nothing is happening now. When I was having the issues before updating, my VRAM was set at the default 100, so maybe lowering does the trick.

Thanks!

Glad to help! I can't code, even though I'd love to learn. So just by providing a report of issues is about all I can do to feel helpful. :)

hansolo77

One other quick observation...
If you have the Video Screensaver enabled, but are in the "Start" or "Select" menus, the screensaver doesn't work. It's just blanking the screen to black. Not related, just thought I'd mention it. I noticed it when I was typing up my reply about having "Save Metadata on Exit" enabled.

pjft

I have it set to Parse gamelists only to "ON" so it doesn't generate new gamelists on bootup. Then I have it set to Save Metadata on Exit also set to "ON". This actually just got me thinking.. If it doesn't save metadata until exit (from ES right?), maybe that's why the "Last Played" collection isn't getting updated? I've been playing around with keeping the screensaver running all day, so nothing gets saved until it crashes. But the "Last Played" or "Times Played" flags in the metadata should be getting added immediately right?

No, only on ES exit are the gamelists saved to the disk.

What do you mean "I've been playing around with keeping the screensaver running all day, so nothing gets saved until it crashes."? Does it crash when you leave the screensaver running all day? That should hopefully not be the case, it's not the best user experience :P

I actually just did a little experiment when I got home. I re-enabled the Last Played collection and looked. It's now showing 24 games. I know for a fact I've played a LOT more than that. But still, going forward. I noticed it appeared to be sorted by system. That might have just been coincidence. But I remember playing Doom 2 last night because the RetroAchievements site was having issues, so I just played that. It doesn't show up on my list till WAY down at the bottom:
That "The Battle of Olympus" I don't remember ever playing... It looks like it might be sorted by system, but it's not alphabetical as I originally though. "NES" comes after "GENESIS", and "PINBALL" should come after "METROID", so I guess that's right. To further the experiment, after taking that snapshot, I loaded up Doom 2, then backed out, and re-opened that collection. Doom 2 now shows up at the top:
So apparently it's working right?

Hm. I'm not sure. We do have a new sorting option for "sorting per console", etc. It would be a big coincidence that the games you had played would have been sorted by system, and by name, descending... it can be, but I'm not ready to believe it yet. I'm assuming you did not enter the Select menu, nor did you try to sort it.

Would you want to share your Genesis, NES and Ports gamelists via PasteBin or something? Also, I'm assuming the time/date on your Pi is correct (i.e. is it connected to the internet?).

Yeah I will try to keep a close eye on this and see if I can get it to trigger. The lockups I was having (video continuing to play with no input response) seem to be cleared out for now. I almost want to try to put my VRAM back up to 100 to see if I can trigger it again, just to see if ES is still running. At this point, without it crashing anymore, I don't really have anything more to try. I've purposely tried to spam the lists, scrolling really fast, etc because it really felt like it was doing it if I didn't give the video time to load up. Nothing is happening now. When I was having the issues before updating, my VRAM was set at the default 100, so maybe lowering does the trick.

No need to increase VRAM and try to force crash it - I mean, if it crashes because it runs out of memory, there's not much we can do other than try to get it to handle things gracefully. :) But, once again, it's early days, so keep using it with 80 and we'll see how it goes.

Glad to help! I can't code, even though I'd love to learn. So just by providing a report of issues is about all I can do to feel helpful. :)

By all means, thank you for the thorough explanation and investigation here.

As for the note on video screensaver not triggering when you're on a menu, I had run into that as well. I suppose I would need to look into that one day, I'll add it to the list of things to check out. :)

Thanks!

hansolo77

@pjft said in Collections Causing Lockups/White Backgrounds:

No, only on ES exit are the gamelists saved to the disk.

What do you mean "I've been playing around with keeping the screensaver running all day, so nothing gets saved until it crashes."? Does it crash when you leave the screensaver running all day? That should hopefully not be the case, it's not the best user experience :P

No it doesn't crash when I exit the screensaver. I was merely saying that the screensaver is running fine, then I try to play some games and experience the crash. So after a full day of the screensaver working fine, the system crashes once I start to navigate. I can have it working fine, play a few games or whatever, screensaver loads, come back home to test, and it crashes. So it never saves the xml's. It's not related to the screensaver, just ignore that bit. :)

Hm. I'm not sure. We do have a new sorting option for "sorting per console", etc. It would be a big coincidence that the games you had played would have been sorted by system, and by name, descending... it can be, but I'm not ready to believe it yet. I'm assuming you did not enter the Select menu, nor did you try to sort it.

Would you want to share your Genesis, NES and Ports gamelists via PasteBin or something? Also, I'm assuming the time/date on your Pi is correct (i.e. is it connected to the internet?).

Because my gamelists are so large (complete sets) I had to upload them to my Dropbox.
https://www.dropbox.com/sh/gnjsjj0ge5tx6kz/AAC9JdOqquIxWUUKUOtK0iu8a?dl=0

No need to increase VRAM and try to force crash it - I mean, if it crashes because it runs out of memory, there's not much we can do other than try to get it to handle things gracefully. :) But, once again, it's early days, so keep using it with 80 and we'll see how it goes.

I only said "almost". ^_^ I don't want to purposely cause it to lock up. With all the hard crashes and lockups, I'm probably going to corrupt my SD card. So I'd like to minimize it where I can.

By all means, thank you for the thorough explanation and investigation here.

As for the note on video screensaver not triggering when you're on a menu, I had run into that as well. I suppose I would need to look into that one day, I'll add it to the list of things to check out. :)

Yeah I thought it was something I did. But I tested the theory and was able to duplicate it perfectly by just keeping those menus up. Not a biggie, I"m sure it's a rare issue.

pjft

@hansolo77 said in Collections Causing Lockups/White Backgrounds:

So after a full day of the screensaver working fine, the system crashes once I start to navigate. I can have it working fine, play a few games or whatever, screensaver loads, come back home to test, and it crashes. So it never saves the xml's. It's not related to the screensaver, just ignore that bit. :)

Oh my. Does this happen all the time?

I'd be curious to learn whether this also happens if you're not using the video screensaver.

I have left it on overnight a few times and didn't experience such crashes, but that being said, I haven't done so that many times.

As for the sorting, this actually clarifies that it's working as intended:

https://docs.google.com/spreadsheets/d/129M3inADmelPb4jR3dL31B8FsT1yYAj1J9g7Fg58tOk/edit#gid=0

Those games seem to have been opened in sequence, probably as part of you setting up an Achievements subfolder of sorts? Unsure about that, but at least the data on your gamelists matches the exact sorting order shown in your screens.

One less mystery to solve :)

Thanks!

hansolo77

No no no!! It has nothing to do with the screensaver! Before I adjusted the VRAM to 80, I was having things lockup and crash all the time. Randomly. All I was saying was that I had the system working, played a game, left for work, came back with the screensaver running, browsed around to find another game and then it would (or would not) crash. Had I not had the screensaver running, or the DIM screensaver, it would have crashed anyway. The only reason I mentioned it was to reiterate that the system was stable enough to run for HOURS until I started moving around in the menus. Now that's I've dropped back the VRAM, I've not seen it happen. The issue is that the gamelist xml's aren't saving when the system crashes. I've played MANY games and saw that the "Last Played" list had not been properly updated. THAT was the issue I was reporting.

Now that I think about it, it might have something to do with the Mausberry Power Circuit I'm using. As I recall, pushing the power button causes it send a command to the Pi to shutdown. BUT, I don't think it's sending a signal to ES first to close it down. So the Pi shuts off gracefully, but not ES. So the lists aren't saving. They only save if I tell it to shut down ES (or restart it, etc..). Not knowing anything about how all that works, I wouldn't know where to being. But I'm a smart fellow, I could probably figure it out. Just need to add a line to the shutdown script to first exit ES. Is there a terminal command I should be looking for to try? I was thinking sudo killall emulationstation, but will that initiate the "save on exit"?

And yep.. I created subfolders in each of my rom folders for games that have Achievements. I then copied the roms that have them into it. When I did the scrape, it added that folder path to the gamelist.xml. I didn't know if it would work, but it does! I added a "_" to the folder name, hoping to have it show show up at the TOP of the gamelist, but at the bottom works just as well.

pjft

@hansolo77 Hm.

It might certainly be related, though I'm not sure. I mean, you clearly have some metadata being saved at times.

I'd probably test playing a game, or adding a favorite, then shutting it down with the Mausberry, and then seeing if it sticks.

If not, you can then test sending out that signal to ES and seeing if it sticks.

Trial and error, I suppose.

hansolo77

I did actually test it. I think that's the problem. Before shutting down via the Mausberry, I checked the "Last Played" collection and it had 2 or 3 more games on the list after my screenshot. The I shut down. Waited, powered on, and checked. The added games weren't there. So I went and played a game or 2, then checked again. The games were on the list. Did a quick Mausberry shutdown again and once again the games are gone.

I tried adding the sudo killall emulationstation to the script on the line before sudo poweroff but it didn't make any difference in the following test. I wonder if that's not the best command to use? Either that, or should I add some kind of delay to have the script pause until it confirms the task has finished before initiating the poweroff? I'm not sure how to do that.

I can confirm however that the sudo killall emulationstation command DOES work via SSH. I started a game, exited, verified it was in "Last Played", then SSH'd and sent the command. It existed to terminal. I then ran emulationstation to get it back up, and the game was there in "Last Played" like it should.

So I just need to figure out how to make shut down ES properlly before sending out the poweroff command.

pjft

@hansolo77 maybe a post in the power block section may help, I'm sure others would have had the same problem in the past. I'm curious about it though :)

hansolo77

Updated the OP to indicate this problem is solved, and the solution.