Solution for " White Screen of Death "
-
@dankcushions, @MWGemini
Yes, I too am concerned about memory usage in this situation so I'm looking at ways to mitigate this. There are two things I plan on looking at:- Compressing the images when stored in RAM
- Freeing RAM for unused textures and reloading from disk where necessary
Compression is easy and will save a large amount of space on most themes. For example, the zoid theme is 36Mb on disk but takes >350Mb of texture memory when it is uncompressed. I'm not sure how much of a delay this will cause on the pi when decompressing just in time for rendering.
The second option may cause significant delay problems due to the speed of SD card storage.
Also, I may try to implement a priority cache where we can set a maximum VRAM usage and purge only the oldest entries to free room for new textures - this in conjunction with 1) or 2) above should hopefully sort us out.
Anyway, I'll try both these and see what happens.
-
Nice work.
I released a bandaid for this as well that involved a nice picture resizing algorithm that reduced picture sizes without a notcible difference - but your solution seems more appropriate.
The TronkyFran theme is the best to test with this since the images are so crisp, any resolution is immediately noticible.
Also, a few of the screens on the tronky theme revert to manually blurred images. Ignore those, and just focus on the system console photos.
Email me at mattborkin@gmail.com if you need me to test anything out. I've been working on this for weeks.
Thanks.
-
Hi @KillerQ,
Yes the tronky theme is definitely the one to test with. The images are huge. They render perfectly with the exception of a few (e.g. Virtual Boy, Wii, WonderSwan/color, Dragon32, etc.) so I think they are just incomplete images in the theme.
Thanks for the offer of testing. I'm nearly ready to produce a good version of the fix - just a few things to iron out which I'll detail in a minute. The more testing the better :)
-
Right, so it works. I've tried a couple of things out and it seems that dynamically loading and unloading the images rather than keeping them cached in RAM is the way to go. It does cause a bit of a delay when scrolling the carousel when using very large images (TronkyFran) but for smaller themes it is not noticable.
I need to finish a few things off before this is complete:
- Tidy the code - it's still a bit experimental
- Implement a texture cache where you can set a maximum VRAM size and it will only purge the oldest textures when this limit is reached - this should prevent the jerkyness on configurations where the WSOD was not previously happening
- More testing
I've added the memory usage to the FPS display and get the following numbers:
Zoid Theme:
Font VRAM 25.17, Tex VRAM 8.90 Tex RAM: 0.00 Tex Total: 381.20Oldroom (1080p) Theme:
Font VRAM 20.97, Tex VRAM 8.51 Tex RAM: 0.00 Tex Total: 157.91TronkyFran Theme:
Font VRAM 25.17, Tex VRAM 17.31 Tex RAM: 0.00 Tex Total: 495.21All the above had ~65 configured systems but each theme does not support all of them. An explanation of the fields:
Font VRAM: VRAM used in Mb to store font textures
Tex VRAM: VRAM in use for images and icons
Tex RAM: This is the one that I was using a lot of in my previous version - amount of system RAM used by textures. Now this is generally zero
Tex Total: The amount of texture memory that would have been used without my fix in place -
@fieldofcows those are awesome results! what an achievement to quash one of the big ES bugs. sounds like an interesting bit of coding, also!
it's times like this that it's a shame that aloshi abandoned (temporarily? forever?) ES. i can imagine a popular project like that quickly becomes overwhelming (as i'm sure retropie can be!), but slowly but surely, people will take up the slack.
if it's not a pain, it might be nice to submit the finished PR to the main ES repo also. likely nothing happens, but he might at least see it :)
-
@fieldofcows When you're ready for people to test it, let me know and I can pull your code. With the official upgrade I did a day or two ago, I'm now seeing white screens on several systems, and jerkiness while switching. The one system that I put video previews into also is not displaying correctly, but that might be a theme issue and not a memory issue.
I'm impressed by the numbers you've posted! I'm glad it appears this was a relatively easy fix and did not require a total rewrite of the rendering system.
-
Right, it's all done :)
I've completed the three tasks I gave myself above, having implemented a cache of active textures, purging the oldest ones to make room for newly loading textures when necessary.
I would appreciate some people picking up my experimental build and giving it a good thrashing. Release is here. I've only built it for Raspberry Pi 3 because that's all I've got at the moment. The change also works on Linux and Windows (although I've not yet tested the latter) but is not really necessary.
What I've done:
I had a bit of a u-turn on my first experiment where I loaded textures into memory then only populated them into VRAM when used as this was consuming a lot of system RAM which could prevent some other things (read emulators) from working.
Instead, I experimented with actually purging the texture completely from any memory and reloading from SD card when needed. The Pi performed remarkably well with this although there is some jerkiness. So I implemented a fixed VRAM limit using a LRU. Each time a texture is used it is loaded from the disk (causing a slight delay) but it remains in VRAM until the limit is reached. When this happens the least used texture is purged from VRAM to make space for the new one.
This cache is configurable in es_settings.cfg and via the UI in "Other Settings" -> "VRAM LIMIT". The default is 100Mb and I can push my RPi3 to about 120Mb before it starts dropping textures. Note that this limit does not include fonts which load on top of this limit.
The beauty of this approach is that for themes that fit nicely within the VRAM limit (basically any theme that does not WSOD on current ES) will not experience any jerkiness at all because no textures will be purged. Larger themes will only experience the jerkiness when shuffling textures in the cache - which I think is a better result than just getting a white screen :)
Anyway, please try it out and give feedback. It's all prepared ready to submit as a PR to RetroPie if there are no issues with it.
Wow - I write alot don't I?!
-
Oh, I forgot to say. If you turn on the "Show Framerate" option then you get something like this:
Font VRAM: 33.55 Tex VRAM: 120.31 Tex Max: 381.28This shows the VRAM used by the fonts, the VRAM used by all currently loaded textures (which should not be much greater than the limit you set), then the memory that is required to load all textures without caching. The last value is what would have been loaded in one go in the current ES.
-
I had to go into the file manager without allowing ES to load. I went in and uninstalled all themes except Carbon. This allowed me to go back in to ES and do what I needed to do. I eliminated unnecessary consoles (I don't use INFOCOM or SCUMMVM or AppleII). Once I brought it down to <10, I went in and reinstalled Zoid.
-
@fieldofcows said in Solution for " White Screen of Death ":
Wow - I write alot don't I?!
Wait until you see the design proposal doc I'm putting together in response to your Metadata Improvements in EmulationStation post.
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
-
@mattrixk said in Solution for " White Screen of Death ":
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
Thanks :) I'm really enjoying this. If it wasn't for this forum, and the excellent work done by everyone else I would have lost motivation a long time ago. Now, where can I find more hours in the day...
-
@fieldofcows said in Solution for " White Screen of Death ":
@mattrixk said in Solution for " White Screen of Death ":
On a serious note, I'm blown away by the work you are doing to improve all parts of ES.
Thanks :) I'm really enjoying this. If it wasn't for this forum, and the excellent work done by everyone else I would have lost motivation a long time ago. Now, where can I find more hours in the day...
If you figure that one out, please share with the rest of us! :)
Do you happen to have installation steps for your latest patch handy? I'll pull my SD card and make a backup now and can report back after I get your patch up and running.
-
@fieldofcows said in Solution for " White Screen of Death ":
Now, where can I find more hours in the day...
Insomnia?(please don't)
-
@mattrixk I've also written up a design doc that I shared with @fieldofcows regarding the metadata and database redesign. Still a work in progress, but I'm hoping it's heading in the right direction.
-
@MWGemini Hi, yes I've read it and it looks good and aligned to what I'm currently working on. I've just been too consumed with this WSOD fix to work on the metadata. If this fix works OK then I'll be back to it shortly.
-
@fieldofcows Sounds good! I've made some further edits, but I know you've got plenty on your plate. Keep up the good work! I'll install your patch and do some testing.
-
@MWGemini said in Solution for " White Screen of Death ":
Do you happen to have installation steps for your latest patch handy?
Borrowed from my previous instructions on the Video Preview thread:
Installation instructions:
- Download a fresh copy of retropie from https://github.com/RetroPie/RetroPie-Setup/releases/download/4.1/retropie-4.1-rpi2_rpi3.img.gz and install it on an SD card (or install it on an existing card at your own risk)
- Connect the raspberry pi to the network
- Boot the raspberry pi and exit ES back to the terminal
- Download or copy the new ES with WSOD fix from https://github.com/fieldofcows/EmulationStation/releases/download/v0.3-RPi-1/emulationstationinto the home directory, e.g.
wget https://github.com/fieldofcows/EmulationStation/releases/download/v0.3-RPi-1/emulationstation- Make executable and replace the original ES (maybe backing up the original first)
chmod +x emulationstation sudo cp emulationstation /opt/retropie/supplementary/emulationstation/-
Add the biggest theme you can think of and as many systems as you can :)
-
Restart
-
@fieldofcows I was just about to post that link. You beat me to it!
-
I believe the build should work on the RPi2 as well if anyone wants to try it but doesn't have a RPi3 - although I cannot confirm this as I don't have a RPi2 myself.
-
@fieldofcows Initial reports: huge success! It fixed the dropped textures I was seeing, although there is that slight but noticeable hitch as I cycle through the carousel and force new textures to be loaded. The only bug I have found so far is that if you are pressing right or left while the load is happening, it reacts as though the input key is being held down. This only happened once, and I have not been able to replicate it yet. Edit: It looks like if you move in one direction (either left or right) until textures start to load, then start moving in the opposite direction, as soon as you hit the end of the cache, where new textures have to be loaded, the "onKeyRelease" event (or whatever it is in ES) doesn't register, causing the input to loop. This is easily fixed by the user by simply pressing and releasing that key.
I can also confirm that 120mb is the upper limit before textures start getting dropped (at least on my system).
Will investigate further and try various ROMs as well.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.