Increased "White Flash" in newest updates???

TMNTturtlguy

@pjft and @jdrassa I decided to start a new thread after seeing a lot of posts over the last 2 or 3 days regarding people updating and seeing increased amount of flashing. I am guilty of not have played around to much with my system over the last few days as i had a work deadline and spent my free time creating more art for my theme.

today i decided to look at a few of my builds to see if i noticed anything. I can't give you scientific testing, but what i can tell you is that i agree with the increase in posts. It appears that when i run my theme on one of the builds from a few weeks ago with es V2.1.5RP or the Apr17-OMX I saw vast improvements over retropie 4.1 and the ES prior I had very little if no white flash and was running a ton of sytems very smoothly. The only issue is the black boxes that appear on the MD data fields. When I run my system that I updated this weekend where i updated everything, the black box is fixed, but i feel that there is a lot more flashing. I am not loosing anything to glget errors, but the flashing and my VRAM settings take me back to where we were Pre - "White Screen of Death" fix.

Again, this is not very scientific and it is running on 2 different raspberry pi's and 2 different sd cards, but with the other posts noticing this after this weekend, I think there may be something in the new updates that is using up more memory? Any ideas or thoughts? @pfjt - did you notice anything on your build using my theme?

pjft

@TMNTturtlguy yeah, I mean, it is certainly related to memory management and how the current textures are loaded as well as how VRAM is managed. Certainly your theme causes more white flashes than, let's say, Carbon. For a benchmark, each new system browsed in System View in your theme currently takes 20mb of memory. So, when you move, it will have to quickly load those 20mb. Since we can't fit the entire theme in memory, unless you're constantly navigating through the same systems or have just a handful, the white currently comes in as the textures are loaded.

I'm not sure of how to address it within the current system's limitations (memory and such), but I do want to look into @fieldofcows 's code to at least have it handle actual out of memory situations gracefully. That being said, it will take some time so no promises in any way for the short term, as well as the fact that that code is a little opaque to me and I don't have the best development/debug environment as I'm on a Mac. Maybe when I go back to Windows later in the year things will be easier/better for that.

On a side note, I'm very much open to suggestions or recommendations from anyone developing ES on a Mac on how to live debug EmulationStation - either remotely on a Pi or, assuming it does run on a Mac, locally :)

Edit: now, let's also be fair and be factual. Before the WSOD fix you wouldn't be able to load the theme, and you'd just get a white screen period. That's why it was called white screen of death. :) The flashes exist precisely because the fix was implemented, so that the theme is loaded in chunks, as the textures are needed, replacing unnecessary textures. It was a tremendous amount of work by @fieldofcows that is effectively working as intended, make no mistake. Surely there may be one or two things to iron out or perhaps optimize, but I've seen a few people referring to WSOD not having been fixed these days that are making incorrect and unfair statements. :) Most people probably don't remember how it was before, that's more likely. ES would just be white, and sometimes even freeze. The only way out was to manually edit the settings file to change the theme back to Carbon. We are no longer living in that era, thanks to that fix.

pjft

@TMNTturtlguy Added a few more relevant comments to that post likely after you read it, as we were doing it at the same time. Have a great day!

TMNTturtlguy

@pjft Thanks! I am on board with the fact that WSOD is a major improvement to what it was just a few months/weeks ago. I think my question is more that I saw a major improvement after that update, and now after the latest update they white symptoms seem to be increasing. Definitely not shutting down or locking up at all. There have also been a lot of others, who i assume are using different themes that have noticed this increase. Here is another thread, link, I also know that you and I were both on a couple of threads the other day defending the WSOD fix as it has been a major improvement. I guess I was just wondering if something in the code in the latest update might be triggering the increase in white flashing. My theme hasn't changed but its performance seems to be different is all.

Anyways, thanks for the detailed reply! back to work on my theme updates....and one day I will figure out how to improve its performance without loosing quality! I am trying to figure out how to get rid of the .svg line work per @Nismo suggestions, but just can't get a good performing black line in another format! Best-

pjft

@TMNTturtlguy I suppose a good way to troubleshoot is to ask these new cases:

what ES were they on before the update;
did they change theme recently;

I'll be fair, though. My bet (just gut feeling, not an educated guess) is that most of these complaints are either people who are just now installing RetroPie for the first time, or were in an older ES version prior to the WSOD fix and have now installed the build with the fix. The latter case will explain whereas prior to the update their entire theme was loaded into memory and luckily it fit, and now with the VRAM limit it doesn't. A way to mitigate that is, for those cases only, definitely to increase VRAM significantly, as they will never (hopefully) hit the limit. If it turns fully white with the glget errors, that's when we need to decrease it in order to force the textures to onload and recycle.

I might look into seeing if there's a smarter way to cache textures, but my understanding is that it won't. A texture, I think, is stored as an in memory bitmap. So, rough calculations, if that is indeed the case, a single 720p background image could be taking up something like

~~1280 * 720 pixels * 4 color values (RGBA) * 32-bit color depth (4 bytes) = 14MB of GPU memory~~

1280 * 720 pixels * 4 bytes (32-bit color depth - RGBA) = 3.7MB of GPU memory

Now, add to that the logos, the gamelist view, any other system-related artwork and you can see this easily scaling up.

Of course, this may have some optimizations in the pipeline (16 bit color depth may halve it, if it's not transparent (24-bit) it may be reduced to 75%, etc...). It can be worse and be stored at full rendering resolution as well, which would mean 1920 * 1080... I don't know, will need to read up on that and investigate further. But it'll all depend on the underlying data structures and SDL GL capabilities.

So yeah, this is kind of what we're up against. Hope this clarifies.

I might try to create a build with a few profiling features just to output logs of all the memory usage by the textures, for each texture. Maybe that can inform theme decisions. But I think one key aspect is that, as far as actual runtime memory goes, resolution seems to be the key factor - not file format, nor color depth (unfortunately).

Edit: fixed the formula I pasted - was double counting the color depth. Also, checked in code that all non-svg images are loaded as 32-bit images, and that svgs go through a different pipeline which, at this stage, I don't know if it's more or less efficient than the other one.

pjft

Just a small update here, @TMNTturtlguy .

I spent a considerable amount of my free time the past few days looking into the whole texture management and memory consumption issues. I put together a few attempts at a profiling build (i.e. a build that provided logs on the texture loading, memory used, etc) but the results weren't really satisfactory given the extremely limited available resources to debug and get information from the Pi's GPU. The only way available that I was able to gather is via sudo vcdbg reloc, which unfortunately is one of the very few closed source components of the Pi software, so there's that.

I'll keep digging into this a bit more, though I suspect I won't get a lot farther in this in the short term, so I'll need to balance this with investing time in other ES developments that may be more relevant to the wider community (specifically bringing over OMX support as an option, and screensaver to the main branch).

If you want to get a bit more insights into this yourself - or at least see what may be happening - I'd recommend installing the script at https://github.com/MilhouseVH/bcmstatand running it as ./bcmstat.sh Dd2 (the "2" standing for the refresh period in seconds - you can change the number) in a SSH session, while you have ES running.

Also, running sudo vcdbg relocwill display the loaded textures and memory taken, though unfortunately in a somewhat useless manner for us as we don't know which texture block actually corresponds to in our themes. Still, interesting for a one off exploration thingie.

Anyways, that's where we're at, and likely where we'll be heading shortly in the coming days. If nothing comes out of it, I'll move on for now. Overall, I think your time is best spent with your theme rather than trying to troubleshoot this for now :D

Hope you're doing well and have a great weekend!

TMNTturtlguy

@pjft WOW! thanks for the information and spending all that time looking into this! I really appreciate it and the follow through. I am going to bookmark this to come back to after I finish my upcoming theme update release. I am always interested in trying to track things and see if I can make sense of how my changes affect the overall performance.

Per our previous dialogue above, ES runs with a ton of systems loaded. With my newest updates I have been testing the systems and I have had 22 systems all loaded and scrolling without getting any errors or white screens, just the flashing. To me this is excellent. To have a theme as robust as mine and 22 themes scrolling smoothly is great, especially when I plan to run 12-16 systems.

Hope you have a great weekend as well. The family and I are heading out of town for a few days so I will be without my pi :( but at least i will have forum access to feed my addiction.

pjft

@TMNTturtlguy Haha - well done! I believe you mentioned you had been traveling for work - hope it all went well, and enjoy your time with the family, which is always precious.

For the sake of your fun, here are some excerpts of:

Logs from the bcmstat.sh GPU stats tool during one ES session as well.
The logs for a ES session and navigating the system view - they're somewhat superficial, though it's hard to get better granularity at the moment, I need to understand the code better. I also removed several logs from parts of the code where I thought they'd be using up GPU memory but weren't in the end. Still, I may be missing quite a few as well, so we'll see where this takes us. You'll see some floating - in particular when releasingVRAM - as that's the code from the Texture Manager to prevent the WSOD. I have my VRAM limit set to 70MB. I assume if I increase it significantly, we'll see this going down until it hits zero.
A sample from the vcdbg tool.

As I said, nothing particularly helpful without context, though it allowed me to observe a few interesting things in regards to the texture memory management - namely which ones are released and when. It seems ES currently tries to always buffer the system to the right and to the left of the currently selected one, in the SystemView, likely to avoid the white screens. So I'll need to see if the white screens are related to low memory, which in turn leads to slowdown in loading caused perhaps by the VRAM release process or some texture management things that take place.

So yeah, it's been useful, though still unclear whether anything tangible will come out of this from my end.

pjft

@TMNTturtlguy Ok, I'm onto something in regards to memory consumption, from observing the logs and thinking about it.

Putting it simply: get rid of the border SVGs and integrate them in the background PNGs.

Putting it in more detail:

Each of your background images takes up 6MB in memory.
Each of the SVGs you load take up 13MB.

So, for each new system you load, it takes up the absurd amount of 19-20MB - only in the system view!

I tried to edit the theme to load the SVG border from the same path (rather than from each of the individual system folders) but ES isn't especially smart, and will create different textures in memory for each of the systems, even if it's the exact same file. So that didn't work.

I'll try to look into that code and see if I can key the textures by path (which would make some sense...), and that could certainly hope quite a bit with redundancy and such.

Until that's available, though, (or, if it does not affect your theme's aspect in any way) my recommendation is to get rid of the border svgs.

You can perform a very simple test by renaming the border and menu SVG files to a different name, temporarily, and then just loading ES and browsing through the theme. You'll see the difference it'll make.

I'm happy with this outcome/discovery, as at least it gives me some path forward (i.e. attempt to not load duplicates of the exact same textures). Whether that a) is possible, b) solves it, and c) does not bring new problems with it remains to be seen, but fingers crossed.

Have a great family weekend now, though, don't think too much about this :)

pjft

Ok, and asides from the memory thing, I will add an option to not show white screens.

They show because we're loading textures in separate threads, and as they're heavy to render they'll sometimes choke the GPU and take slightly longer to load. I'll add an option to not load images in separate threads, should the user choose to.

In reality, that'll probably be the preferred option, though it might not feel as smooth to navigate in heavier themes.

Omnija

@TMNTturtlguy from what @pift mentioned, it seems your theme + systems have been draining you of the mb that you have. I'm pretty sure that it's not just the theme and systems that drain but also all the box arts as well (boxarts can be very large and slow down scrolling and such). As a means of attempting to fix that sort of issue could probably be done in a few ways. You could replace svgs to png if that would shrink size (not sure haven;t tested) then compress the pngs afterwards using "pngoptimizer". Id assuming you have also tested with raising the gpu_mem=, but figured id mention the obvious anyways. This could help with reducing the entire theme as well as any box art that may not have been resized.

TMNTturtlguy

@Omnija thanks, the .png files are already heavily compressed as is, any further compression and we loose quality and the dithering of the images is increased. As for box art, that will depend on what each individual has on there system, my theme cannot control that.

pjft

@Omnija Thanks.

Just to clarify, the image size on the HD does not have any reflection in the actual GPU memory usage, but rather its size/resolution. Indeed merging the backgrounds with the SVGs could help.

In other news, I've submitted a PR to mitigate this.

Let's see. It's not really fixed per se, but I'm adding two options for users with heavier themes to exchange smooth navigation for little to no white screens. Well, except for VRAM related ones :)

@TMNTturtlguy - as usual, you may find it in my repository as well.

Best.

TMNTturtlguy

@pjft if we select this option will we just get a blank(black) screen while the next image loads?

Edit: and I am gone for the weekend! Shoot! I will be testing this late Sunday night for sure! Thanks for your hard work!

pjft

@TMNTturtlguy It depends on the transition you have. If using "Fade", then yes, it'll just stay in the black screen for slightly longer.

If using slide, it may have a slight stutter.

I strongly recommend merging the SVG frames with the background PNGs, though. After you test your theme without the SVGs (by renaming them), you'll see the huge difference. Seriously.

Enjoy the weekend.

TMNTturtlguy

@pjft the problem is the theme backgrounds will no longer look nice and will not adjust to larger screen sizes. The .svg allows the line work to resize to large HDTV screens and 4K resolutions while staying crisp. The line work as a .png will be very jagged at all angles lines. If you look at the video view game selection view and the angles lines by the box art closely, you will see the jaggy effect on a larger screen. Those are short lines so they are not supper noticeable and I chose to merge the line work. One option would be to create a separate .png with the black lines, but that file size will need to be larger so that when upscale it stays clear. Any idea how using a secon .png would affect it?

pjft

@TMNTturtlguy I suspect that'll be a non-factor for the Pi, as its max output resolution is 1920x1080 via the HDMI port. However, for PC that may certainly be a factor.

As I mentioned, a second image will take up as much memory as its resolution - refer to my earlier post with the formula.

If you want to, my best recommendation would be to make a Pi theme with the backgrounds merged, and - if you then feel, after testing at higher resolution, that it loses a lot - an HD version where you'd have the separate SVG files. It's a bigger burden for maintenance, though, so I wouldn't take that decision lightly. That being said, if you still have everything in Photoshop format, together with the borders, maybe you can just batch process all and generate both versions easily though?

Anyway, you'll be able to judge by yourself later, but the Pi is an impressive, yet severely constrained device in terms of hardware, let's not forget :)

We can't have our Pie and eat it too. (Bad pun intended).

And I'll take a moment and say, after so much time going through the code, what an amazing job @fieldofcows did with the texture manager. Really, asides from not failing gracefully if it does run out of memory, it is truly an excellent, fantastic piece of work.

TMNTturtlguy

@pjft as always your knowledge and service is absolutely wonderful and so very much appreciated. I have re-read your posts and have a few questions, or confirmations. Per your formula you suggest that 720x1280 should take up 4mb. My backgrounds are all 720x1280, but your testing shows the .png is using 6mb. Where are the extra 2mb coming from? The system logos?

Also, from what I now understand, the file size does not matter? Would a 512kb .png perform the same as a 5mb .png if both are 720x1280? Not that I want my files that large but just asking. @Nismo has previously indicated that as the file sizes increase the amount of memory increases. Under 512 uses 256mb and above uses 1024 I think? Can't remember the exacts and I am on my phone now!

Edit: found the great posts from @Nismo which related to everything @pjft is talking about and also references @fieldofcows work as well. Yes, I know.....@Nismo suggested the .svg are bad as well! 10 days ago!!! But it looks so much better! As @pjft suggested, I want to eat my pie too! Here is the link to the post link to Nismo post

I will try to merge and see what happens. If I merge the black lines over the already compressed background and flatten, it might be ok.

TMNTturtlguy

@pjft Ok, so i don't have my pi with me, but i do have my computer and photoshop. Family went to bed and i have spent a few hours trying different ways to get the black boarder to appear clean and crisp and still compress down below 512kb. I think I have done it, but I can't test it from here. If you have time this weekend. I have uploaded a few of the backgrounds i have created. If you would copy these into the theme and name them comic in their corresponding folders. Also rename the .svg file so it does not show up anymore.

If you would want to, you could put these into the theme and then scroll back and forth between these and the other systems that would still have the .svg and let me know what you think? You have a great eye and a great screen to see little details, so I am confident in your opinion. If you don't have time to test, that is ok as well. You have pushed me to keep improving so I will definitely test on Sunday evening. Thank you - Link to github test backgrounds: TempThemeBackground

pjft

@TMNTturtlguy Thanks.

I tested them, and effectively they look more jagged than ideal.

I tested creating 1080p images, and they looked crisper, but the navigation appears to have become as bad as like the version with SVGs. It appears the bottleneck is not as much the memory (even though it does affect performance when we need to start unloading and loading them from memory), but rendering speed.

I got logs from them, but haven't really spent time looking at them in detail. Here's the link, just for curious people:

https://pastebin.com/ureadR50

I suspect that, if the PR I submitted goes through, you may be able just to keep the theme as is for the time being, and then we'll see if it does need to be optimized further or not. I'd still recommend getting all the individual system themes to point to the shared SVG for the borders, though, in order to save memory (i.e. in individual systems, instead of pointing to ./art/border_4-3Outline.svgpoint to ./../art/border_4-3Outline.svg.

Other than that, you do want the theme to look crisp so let's not worry about that for now :)

I'm uploading the themes I used for testing - a full 1080p-noSVG version, a full 720p-noSVG version, and your regular SVG one, pointing to the same SVG border files (I think). You can test these 3, just for a feel, both with the current ES you have, as well as with the one from my repository if you want to update. There'll be two options in the "Other Settings" menu - you'll want to turn off Multithreaded images, to get rid of the white screens, and then test the 4 themes to see the difference.

Here's the link.

If you choose to select the "Re-Use SVG images" (I think that's the name), you may need to re-load ES/the theme, so that it only loads one copy in memory.

That's it!

Have a great weekend, and do enjoy your time off - it's precious as well. Plenty of time for tinkering when you're back :)