Solution for " White Screen of Death "

KillerQ

Thanks for the quick reply. There MUST be some way to crack this problem. There ls always a way - lol.

I'm tempted to try split images or something else out of the box.

I'll report back.

I'm also gonna figure out how to have it load a background once the screen is stabilized for a particular system.

fieldofcows

Ok, I may have a simple-ish fix for this. I've just tried modifying the TextureResource component in EmulationStation to keep textures in conventional RAM instead of texture memory and only use texture memory when bound during rendering, freeing it once the render operation is complete.

On my Linux system this works perfectly and there is no noticable delay when rendering the images. I have yet to try this on the RPi to see if it works ok. I'll give it a go and report back.

Just as an illustration, I've logged the texture memory that would have been used with the current ES compared with the texture memory used in my modified ES for textures - note that this does not include textures used by fonts at the moment so doesn't give the total texture memory that is used.

Theme used: Nismo's excellent OldRoom 720p theme (maybe not the latest version)
Number of systems: 13
Texture memory used by TextureResource in current ES: 97Mb
Texture memory used by TextureResource in new ES: 3.5Mb

Fingers crossed this may be a solution for the WSOD problem :D

mattrixk

@fieldofcows said in Solution for " White Screen of Death ":

Texture memory used by TextureResource in current ES: 97Mb
Texture memory used by TextureResource in new ES: 3.5Mb

I don't know much about what you are doing, but that number difference is none-the-less very impressive.

fieldofcows

Well my change definitely seems to solve the problem but it still needs a little bit more work. Regenerating the texture on every render is a bit too much for the poor RPi to handle so it makes the rendering sluggish. It shouldn't be too hard to fix.

I tried 'zoid' and 'oldroom 1080p' themes with the maximum amount of systems. Without my changes both failed to render. With my changes they rendered fine, albeit slowly.

fieldofcows

Well, that's sorted it. I've put a bit of caching in there so I track on every render when a texture has been used or not and purge any texture that was not used in the last render frame. It is automatically recreated on the next render where it is used. This sorts out the performance issue.

So, large 1080p themes on RPi are a go :D

It's not the most elegant fix but short of reengineering the whole texture code I think it's adequate. I don't suppose the community will complain as long as it fixes the problem...

I need to do a bit more testing before I submit this fix. Can anybody suggest some good themes to try that you would normally expect to cause the white screen of death?

MWGemini

@fieldofcows I'm currently using a modified version of the tronkyfran theme, with 18 systems on the main carousel, and videos in one of the systems. At version 2.1.0RP of ES, I did not have a WSOD issue (with an increase memory split of 384- default mem split did cause WSOD). With the latest update (2.1.2RP), I'm getting partial white screens on several systems.

mattrixk

@fieldofcows I know lots of people complained about the original "Simple" theme, but I haven't tried it myself (by the time I came along, Carbon was the default for RetroPie on the Raspberry Pi).

fieldofcows

@MWGemini That's a big theme! I've just tried it with 62 systems if I counted that correctly on my RPi3. I haven't changed the memory split - everything is in it's default configuration. It works!

A couple of the backgrounds are rendered a bit blurred. I'm not sure if that's my change or something else but I'll investigate.

The other issue is that my caching will consume a fair amount of RAM. I'm not sure if that is going to cause any problems.

fieldofcows

Commit is here if anybody fancies reviewing the code:
https://github.com/fieldofcows/EmulationStation/commit/b4ad5eeda104b166667b8d4bc5262c925f6f86a3

MWGemini

@fieldofcows 62 systems without a WSOD? Wow! As for blurred backgrounds, I know that at least a few are blurred on mine. Off the top of my head, Sega 32x and Sega CD are both blurred game images, and PC is a slightly blurred PC image. All of the other systems I have in my collection have pretty crisp backgrounds.

With regards to the caching, will that be cleared or garbage collected when a game is launched? If not, I can try to pull down your changes and run a few of the more demanding games in my collection and see if I notice any significant performance changes. I'm guessing that PSX, N64 and Dreamcast will be the heaviest, but I don't have any sort of memory profiling system set up currently, so the best I can do is provide anecdotal evidence.

herb_fargus

@fieldofcows if you're talking about some blurred images on the tronkyfran theme, some I think were reduced to 720p which may have cause some blur on downscaling perhaps, also the default simple images were left for systems tronkyfran didn't create a wallpaper for. might have to check against other images you know are full HD just to see if it's reproducible but I'm just guessing it's the incompleteness of the theme

dankcushions

@MWGemini said in Solution for " White Screen of Death ":

@fieldofcows 62 systems without a WSOD? Wow! As for blurred backgrounds, I know that at least a few are blurred on mine. Off the top of my head, Sega 32x and Sega CD are both blurred game images, and PC is a slightly blurred PC image. All of the other systems I have in my collection have pretty crisp backgrounds.

With regards to the caching, will that be cleared or garbage collected when a game is launched?

i second this question. if it remains in system memory, if i use a lite theme (pixel) i'd personally want it to use my VRAM over my system ram. VRAM is barely used in retropie outside of emulationstation, but system ram can be used a lot in mame games with large samples, or system compilation.

fieldofcows

@dankcushions, @MWGemini
Yes, I too am concerned about memory usage in this situation so I'm looking at ways to mitigate this. There are two things I plan on looking at:

Compressing the images when stored in RAM
Freeing RAM for unused textures and reloading from disk where necessary

Compression is easy and will save a large amount of space on most themes. For example, the zoid theme is 36Mb on disk but takes >350Mb of texture memory when it is uncompressed. I'm not sure how much of a delay this will cause on the pi when decompressing just in time for rendering.

The second option may cause significant delay problems due to the speed of SD card storage.

Also, I may try to implement a priority cache where we can set a maximum VRAM usage and purge only the oldest entries to free room for new textures - this in conjunction with 1) or 2) above should hopefully sort us out.

Anyway, I'll try both these and see what happens.

KillerQ

Nice work.

I released a bandaid for this as well that involved a nice picture resizing algorithm that reduced picture sizes without a notcible difference - but your solution seems more appropriate.

The TronkyFran theme is the best to test with this since the images are so crisp, any resolution is immediately noticible.

Also, a few of the screens on the tronky theme revert to manually blurred images. Ignore those, and just focus on the system console photos.

Email me at mattborkin@gmail.com if you need me to test anything out. I've been working on this for weeks.

Thanks.

fieldofcows

Hi @KillerQ,

Yes the tronky theme is definitely the one to test with. The images are huge. They render perfectly with the exception of a few (e.g. Virtual Boy, Wii, WonderSwan/color, Dragon32, etc.) so I think they are just incomplete images in the theme.

Thanks for the offer of testing. I'm nearly ready to produce a good version of the fix - just a few things to iron out which I'll detail in a minute. The more testing the better :)

fieldofcows

Right, so it works. I've tried a couple of things out and it seems that dynamically loading and unloading the images rather than keeping them cached in RAM is the way to go. It does cause a bit of a delay when scrolling the carousel when using very large images (TronkyFran) but for smaller themes it is not noticable.

I need to finish a few things off before this is complete:

Tidy the code - it's still a bit experimental
Implement a texture cache where you can set a maximum VRAM size and it will only purge the oldest textures when this limit is reached - this should prevent the jerkyness on configurations where the WSOD was not previously happening
More testing

I've added the memory usage to the FPS display and get the following numbers:

Zoid Theme:
Font VRAM 25.17, Tex VRAM 8.90 Tex RAM: 0.00 Tex Total: 381.20

Oldroom (1080p) Theme:
Font VRAM 20.97, Tex VRAM 8.51 Tex RAM: 0.00 Tex Total: 157.91

TronkyFran Theme:
Font VRAM 25.17, Tex VRAM 17.31 Tex RAM: 0.00 Tex Total: 495.21

All the above had ~65 configured systems but each theme does not support all of them. An explanation of the fields:

Font VRAM: VRAM used in Mb to store font textures
Tex VRAM: VRAM in use for images and icons
Tex RAM: This is the one that I was using a lot of in my previous version - amount of system RAM used by textures. Now this is generally zero
Tex Total: The amount of texture memory that would have been used without my fix in place

dankcushions

@fieldofcows those are awesome results! what an achievement to quash one of the big ES bugs. sounds like an interesting bit of coding, also!

it's times like this that it's a shame that aloshi abandoned (temporarily? forever?) ES. i can imagine a popular project like that quickly becomes overwhelming (as i'm sure retropie can be!), but slowly but surely, people will take up the slack.

if it's not a pain, it might be nice to submit the finished PR to the main ES repo also. likely nothing happens, but he might at least see it :)

MWGemini

@fieldofcows When you're ready for people to test it, let me know and I can pull your code. With the official upgrade I did a day or two ago, I'm now seeing white screens on several systems, and jerkiness while switching. The one system that I put video previews into also is not displaying correctly, but that might be a theme issue and not a memory issue.

I'm impressed by the numbers you've posted! I'm glad it appears this was a relatively easy fix and did not require a total rewrite of the rendering system.

fieldofcows

Right, it's all done :)

I've completed the three tasks I gave myself above, having implemented a cache of active textures, purging the oldest ones to make room for newly loading textures when necessary.

I would appreciate some people picking up my experimental build and giving it a good thrashing. Release is here. I've only built it for Raspberry Pi 3 because that's all I've got at the moment. The change also works on Linux and Windows (although I've not yet tested the latter) but is not really necessary.

What I've done:

I had a bit of a u-turn on my first experiment where I loaded textures into memory then only populated them into VRAM when used as this was consuming a lot of system RAM which could prevent some other things (read emulators) from working.

Instead, I experimented with actually purging the texture completely from any memory and reloading from SD card when needed. The Pi performed remarkably well with this although there is some jerkiness. So I implemented a fixed VRAM limit using a LRU. Each time a texture is used it is loaded from the disk (causing a slight delay) but it remains in VRAM until the limit is reached. When this happens the least used texture is purged from VRAM to make space for the new one.

This cache is configurable in es_settings.cfg and via the UI in "Other Settings" -> "VRAM LIMIT". The default is 100Mb and I can push my RPi3 to about 120Mb before it starts dropping textures. Note that this limit does not include fonts which load on top of this limit.

The beauty of this approach is that for themes that fit nicely within the VRAM limit (basically any theme that does not WSOD on current ES) will not experience any jerkiness at all because no textures will be purged. Larger themes will only experience the jerkiness when shuffling textures in the cache - which I think is a better result than just getting a white screen :)

Anyway, please try it out and give feedback. It's all prepared ready to submit as a PR to RetroPie if there are no issues with it.

Wow - I write alot don't I?!

fieldofcows

Oh, I forgot to say. If you turn on the "Show Framerate" option then you get something like this:

Font VRAM: 33.55 Tex VRAM: 120.31 Tex Max: 381.28

This shows the VRAM used by the fonts, the VRAM used by all currently loaded textures (which should not be much greater than the limit you set), then the memory that is required to load all textures without caching. The last value is what would have been loaded in one go in the current ES.