Solution for " White Screen of Death "
-
Right, so it works. I've tried a couple of things out and it seems that dynamically loading and unloading the images rather than keeping them cached in RAM is the way to go. It does cause a bit of a delay when scrolling the carousel when using very large images (TronkyFran) but for smaller themes it is not noticable.
I need to finish a few things off before this is complete:
- Tidy the code - it's still a bit experimental
- Implement a texture cache where you can set a maximum VRAM size and it will only purge the oldest textures when this limit is reached - this should prevent the jerkyness on configurations where the WSOD was not previously happening
- More testing
I've added the memory usage to the FPS display and get the following numbers:
Zoid Theme:
Font VRAM 25.17, Tex VRAM 8.90 Tex RAM: 0.00 Tex Total: 381.20Oldroom (1080p) Theme:
Font VRAM 20.97, Tex VRAM 8.51 Tex RAM: 0.00 Tex Total: 157.91TronkyFran Theme:
Font VRAM 25.17, Tex VRAM 17.31 Tex RAM: 0.00 Tex Total: 495.21All the above had ~65 configured systems but each theme does not support all of them. An explanation of the fields:
Font VRAM: VRAM used in Mb to store font textures
Tex VRAM: VRAM in use for images and icons
Tex RAM: This is the one that I was using a lot of in my previous version - amount of system RAM used by textures. Now this is generally zero
Tex Total: The amount of texture memory that would have been used without my fix in place -
@fieldofcows those are awesome results! what an achievement to quash one of the big ES bugs. sounds like an interesting bit of coding, also!
it's times like this that it's a shame that aloshi abandoned (temporarily? forever?) ES. i can imagine a popular project like that quickly becomes overwhelming (as i'm sure retropie can be!), but slowly but surely, people will take up the slack.
if it's not a pain, it might be nice to submit the finished PR to the main ES repo also. likely nothing happens, but he might at least see it :)
-
@fieldofcows When you're ready for people to test it, let me know and I can pull your code. With the official upgrade I did a day or two ago, I'm now seeing white screens on several systems, and jerkiness while switching. The one system that I put video previews into also is not displaying correctly, but that might be a theme issue and not a memory issue.
I'm impressed by the numbers you've posted! I'm glad it appears this was a relatively easy fix and did not require a total rewrite of the rendering system.
-
Right, it's all done :)
I've completed the three tasks I gave myself above, having implemented a cache of active textures, purging the oldest ones to make room for newly loading textures when necessary.
I would appreciate some people picking up my experimental build and giving it a good thrashing. Release is here. I've only built it for Raspberry Pi 3 because that's all I've got at the moment. The change also works on Linux and Windows (although I've not yet tested the latter) but is not really necessary.
What I've done:
I had a bit of a u-turn on my first experiment where I loaded textures into memory then only populated them into VRAM when used as this was consuming a lot of system RAM which could prevent some other things (read emulators) from working.
Instead, I experimented with actually purging the texture completely from any memory and reloading from SD card when needed. The Pi performed remarkably well with this although there is some jerkiness. So I implemented a fixed VRAM limit using a LRU. Each time a texture is used it is loaded from the disk (causing a slight delay) but it remains in VRAM until the limit is reached. When this happens the least used texture is purged from VRAM to make space for the new one.
This cache is configurable in es_settings.cfg and via the UI in "Other Settings" -> "VRAM LIMIT". The default is 100Mb and I can push my RPi3 to about 120Mb before it starts dropping textures. Note that this limit does not include fonts which load on top of this limit.
The beauty of this approach is that for themes that fit nicely within the VRAM limit (basically any theme that does not WSOD on current ES) will not experience any jerkiness at all because no textures will be purged. Larger themes will only experience the jerkiness when shuffling textures in the cache - which I think is a better result than just getting a white screen :)
Anyway, please try it out and give feedback. It's all prepared ready to submit as a PR to RetroPie if there are no issues with it.
Wow - I write alot don't I?!
-
Oh, I forgot to say. If you turn on the "Show Framerate" option then you get something like this:
Font VRAM: 33.55 Tex VRAM: 120.31 Tex Max: 381.28This shows the VRAM used by the fonts, the VRAM used by all currently loaded textures (which should not be much greater than the limit you set), then the memory that is required to load all textures without caching. The last value is what would have been loaded in one go in the current ES.
-
I had to go into the file manager without allowing ES to load. I went in and uninstalled all themes except Carbon. This allowed me to go back in to ES and do what I needed to do. I eliminated unnecessary consoles (I don't use INFOCOM or SCUMMVM or AppleII). Once I brought it down to <10, I went in and reinstalled Zoid.
-
@fieldofcows said in Solution for " White Screen of Death ":
Wow - I write alot don't I?!
Wait until you see the design proposal doc I'm putting together in response to your Metadata Improvements in EmulationStation post.
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
-
@mattrixk said in Solution for " White Screen of Death ":
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
Thanks :) I'm really enjoying this. If it wasn't for this forum, and the excellent work done by everyone else I would have lost motivation a long time ago. Now, where can I find more hours in the day...
-
@fieldofcows said in Solution for " White Screen of Death ":
@mattrixk said in Solution for " White Screen of Death ":
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
Thanks :) I'm really enjoying this. If it wasn't for this forum, and the excellent work done by everyone else I would have lost motivation a long time ago. Now, where can I find more hours in the day...
If you figure that one out, please share with the rest of us! :)
Do you happen to have installation steps for your latest patch handy? I'll pull my SD card and make a backup now and can report back after I get your patch up and running.
-
@fieldofcows said in Solution for " White Screen of Death ":
Now, where can I find more hours in the day...
Insomnia?(please don't)
-
@mattrixk I've also written up a design doc that I shared with @fieldofcows regarding the metadata and database redesign. Still a work in progress, but I'm hoping it's heading in the right direction.
-
@MWGemini Hi, yes I've read it and it looks good and aligned to what I'm currently working on. I've just been too consumed with this WSOD fix to work on the metadata. If this fix works OK then I'll be back to it shortly.
-
@fieldofcows Sounds good! I've made some further edits, but I know you've got plenty on your plate. Keep up the good work! I'll install your patch and do some testing.
-
@MWGemini said in Solution for " White Screen of Death ":
Do you happen to have installation steps for your latest patch handy?
Borrowed from my previous instructions on the Video Preview thread:
Installation instructions:
- Download a fresh copy of retropie from https://github.com/RetroPie/RetroPie-Setup/releases/download/4.1/retropie-4.1-rpi2_rpi3.img.gz and install it on an SD card (or install it on an existing card at your own risk)
- Connect the raspberry pi to the network
- Boot the raspberry pi and exit ES back to the terminal
- Download or copy the new ES with WSOD fix from https://github.com/fieldofcows/EmulationStation/releases/download/v0.3-RPi-1/emulationstationinto the home directory, e.g.
wget https://github.com/fieldofcows/EmulationStation/releases/download/v0.3-RPi-1/emulationstation- Make executable and replace the original ES (maybe backing up the original first)
chmod +x emulationstation sudo cp emulationstation /opt/retropie/supplementary/emulationstation/-
Add the biggest theme you can think of and as many systems as you can :)
-
Restart
-
@fieldofcows I was just about to post that link. You beat me to it!
-
I believe the build should work on the RPi2 as well if anyone wants to try it but doesn't have a RPi3 - although I cannot confirm this as I don't have a RPi2 myself.
-
@fieldofcows Initial reports: huge success! It fixed the dropped textures I was seeing, although there is that slight but noticeable hitch as I cycle through the carousel and force new textures to be loaded. The only bug I have found so far is that if you are pressing right or left while the load is happening, it reacts as though the input key is being held down. This only happened once, and I have not been able to replicate it yet. Edit: It looks like if you move in one direction (either left or right) until textures start to load, then start moving in the opposite direction, as soon as you hit the end of the cache, where new textures have to be loaded, the "onKeyRelease" event (or whatever it is in ES) doesn't register, causing the input to loop. This is easily fixed by the user by simply pressing and releasing that key.
I can also confirm that 120mb is the upper limit before textures start getting dropped (at least on my system).
Will investigate further and try various ROMs as well.
-
@fieldofcows I've tried a variety of ROMs, on a variety of systems (including PSX, N64, and Dreamcast), and have not noticed any performance change from the prior official RP build of ES. The closest I came to seeing any gameplay change was graphical glitches in one of the N64 ROMs I tried, but that emulator has always been glitchy for me, so it's very possible that nothing has changed in that regard.
Overall, I'm extremely happy with the change. The only thing I might suggest is that instead of a least-recently-used cache, to keep textures in the cache based on proximity to the currently selected system. In my case, I have 18 or so systems loaded, and the cache can only contain about 10 of the images (modified tronkyfran theme with 100mb VRAM limit). To go right one index is more likely than going left 11, so it might reduce the loading hiccup if the farthest systems are unloaded and the closest systems are kept in memory. Edit: even better would be to do this on a background thread if possible, which could prevent any noticeable hiccups as textures load and unload.
At this point, that's really a nitpicky request, though. You've just fixed one of the biggest (and longest-lasting?) bugs in ES, and I'm sure the community will be very happy about that!
-
@MWGemini Brilliant. Thanks for that. Yes, I'm already looking at prioritising the systems around the currently selected system - great minds think alike.
I have considered a background thread because this would completely cure the jerkiness at the expense of seeing images appear over time. I'll take a look but I'm not sure how easy it will be to make the texture code thread safe.
-
@fieldofcows said in Solution for " White Screen of Death ":
This cache is configurable in es_settings.cfg and via the UI in "Other Settings" -> "VRAM LIMIT". The default is 100Mb and I can push my RPi3 to about 120Mb before it starts dropping textures. Note that this limit does not include fonts which load on top of this limit.
i don't know if it's useful, but
sudo vcdbg relocgives you the free vram (in a bit of an unwieldy format)
sudo vcdbg reloc |grep "\[" | awk '{split($0,a," "); print a[12]}' | awk '{split($0,a,","); print a[1]}' | awk '{sum+=$1}END{print sum}'this gives it in bytes (stole this from somewhere.. not sure if it's properly working)
vcgencmd get_mem arm vcgencmd get_mem gpugets the memory split for system and gpu respectively.
obviously you need some sort of fallback for none-pi systems, but maybe this could be used to pick the cache? not sure.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.