White flicker on moving through UI after update to 4.2.3

pjft

@josete2k I should have fixed it now. I'm trying to force the SVG to be kept in memory to always be the highest resolution used in the theme.

Hopefully this fixes things. Changes have been made both to this build as well as to the OMX branch.

Download and test using the exact same instructions as before - it's the same link. You'll know you're in the right one as it will no longer have the Multithreaded and Re-Use SVGs options.

To ensure you're using it right, though, you may want to remove the two new option values from your ES-Settings (multithreaded images, and re-use SVGs), or set them to "true, true" in your current build before reinstalling as they are now gone from the menu, and defaulted to "true, true".

All of you testing, let me know how it goes. And thanks a lot for catching that one yesterday, @josete2k !

Edit: I've been testing a bit more, and while it seems to be fairly improved, it's still not eradicated. I'll keep digging into this.

josete2k

@pjft yes flickering is still there.

TMNTturtlguy

@pjft I will be on late tonight, probably while you are getting some rest. I plan to test on my comic book theme and carbon theme. I will test with my 12 systems and then load up 20 or so and see what happens.

If there are any specific tests or things you want time to look into let me know here or on the PR comments. I will check both and report back. Thanks!

pjft

@josete2k hopefully the issue with the system logos has disappeared, though? And the flickering should hopefully not be as bad as it was before?

@TMNTturtlguy thanks. I'm trying one final thing - hoping it solves it, but compilation takes quite a bit, so still waiting.

josete2k

Another point...

With the "old" emulationstation, if I set ON Re-Use theme images and Multithreaded in OFF, the white stripe (where the system.svg is loaded) is thinner than normal.

josete2k

@pjft yes, the flickering is... a little "shorter" now.

It happens with the logo in the right side on the screen.

Before it was until it arrived to the center.

Controllers are shown OK.

pjft

@josete2k said in White flicker on moving through UI after update to 4.2.3:

Another point...

With the "old" emulationstation, if I set ON Re-Use theme images and Multithreaded in OFF, the white stripe (where the system.svg is loaded) is thinner than normal.

Can you test with the new one only and check whether it is normal? That combination should not be useful anymore.

I'm still looking into this - thanks for the details!

Update:

Yeah, this is still somewhat borked. I'm calling it a night, tomorrow is a work day.

I'll keep looking into this in the coming days and see if there's anything that I can think of further.

TMNTturtlguy

@pjft I have done extensive testing of your update, and first off, let me just say, you are a miracle worker!

Now to some quick notes - I will also leave feedback on the github PR.

Test 1: Comic Book Theme (heavy theme, 17 systems loaded, uses .svg files) VRAM at 100

After boot up it looks amazing, i can slide with little to no stutter at all, the slide is very smooth, logos and backgrounds load very fast and clean. Every now and then there is a mili-second of a black flash if you want to call it that, but compared to the very large white flashing, this looks so smooth and clean.
After entering a game and exiting: .png background reload time can vary, sometimes fast, sometimes a slight stutter. Exiting back to system select screen. The first time scrolling to the right, there is a large black flash, and the .png system logos now have a white box flash for a half second before the .png logo loads. After one full cycle, everything goes back to completely smooth. I also noted this does not happen after every time i exit a game. Sometimes, it works smoothly.

Video of this test:

Test 2: Carbon Theme 17 systems loaded VRAM 100

This reacts the exact same as the comic book theme did. I notice the same stutter after exiting roms and the same white box on the logos when sliding right. These logos are .svg in carbon. From my testing, it seems like your current solution as fixed the use of .svg and .png, not it is just figuring out how to get the white boxes to disappear after exiting a game.

Test 3: Comic Book Theme (heavy theme, 17 systems loaded, uses .svg files) VRAM at 120

This is absolutely amazing! I can scroll through as fast as I can hit my arrow buttons and never flash or loose the background! After existing a game, the first time though there is a black flash and the logos are still white, after the first time through, it is back to lightning fast speeds.

Video of this test:

Overall - amazing work! I have not seen any negative affects, only major positives. I would suggest this is an upgrade to the current White Screen of Death fix. Even if not perfect yet. I plan to keep this running as my main ES and will report anything else i see back.

EDIT: After more testing with the comic book theme I can create glget errors. More information on this at the Github PR.

pjft

@TMNTturtlguy Thanks for the testing.

As usual, I'd be hoping that my changes would not have caused any particular increase of glGet errors, at least nothing that wouldn't have happened before, but that's yet another thing to chase.

Yes, thanks to @josete2k 's feedback I have indeed narrowed it down to mostly how to fix things after launching a game and coming back.

ES is unloading all textures when it launches a game, so that we have enough memory available to play the games. However, when we get back I am so far struggling to make it work or understanding why that is happening. I have a few theories I'll be chasing down, but it's a puzzle for me. Nothing short of killing ES and actually restarting it seems to fix it for good. Even fully reloading a theme doesn't seem to make a difference.

So yeah, that's where we're at.

I'll also be testing it with a very old ES version, to understand whether this is a regression or whether this used to happen in the past, as that might yield some clues.

We'll see.

Thanks all!

TMNTturtlguy

@pjft I am not sure that your new changes have caused a new way to get glget errors. Take a look at my more detailed testing and report on the PR comments. I have additional notes, testing and another video there. I did or want to duplicate my posts on both sites, so I kept the forum post to a synopsis of my larger post.

As you and I both found, we had been able to reproduce glget errors with several systems running on the comic book theme prior to your modification. I had found that it ran fine at VRAM 100 but in order to operate as smooth as possible and avoid Glget errors it was beneficial to run at a different VRAM, slightly lower or higher than 100. I was also never able to scroll at extremely high speeds as I can now. Lastly, as noted on GitHub I cannot create glget errors on other themes such as carbon or futura.

pjft

@TMNTturtlguy Thanks for the reply. I'll make sure to dutifully read both posts, but thanks for the extensive posting. Hopefully you had a great family weekend! :)

paffley

Hi,

Don't know if this helps but I managed to get rid of the White Flash altogether on my created theme.

Rpi3
Latest RetopiePie build
Video & Marquee support enabled
18 Systems and adding
Scraped boxart within retropie
Game shot images dl'd from web
different backgrounds for system view and gamelist view (PNG 30kb, 6kb)
10 extra image elements added to both system view and gamelist view* (PNG 85kb, 4kb, 2kb, 8kb, 6kb, 10kb, 4kb, 4kb, 31kb, 8kb)

What I did...
I changed my 'background png' from 1920x1080 to 1280x720 and batch shrank my scraped and dl'd images (boxart/game shot) down from approx. 300kb to 50kb - This seemed to stop the white flash completely on my theme and runs smooth on all parts.

My thoughts
I believe it was the weight of my theme (pretty light) and the weight of the scraped boxart/game shot images together causing the White Flash in my opinion, only happened since RP4.2.

Hope this helps...

Cheers,

pjft

Thanks @paffley for sharing. Indeed, there's plenty of things that can - and should! - be done at a theme level, no doubt about that, as well as for the artwork people use in their own libraries.

The main thing that has come up recently, as well, is that since a few ES builds ago there have been an increase of odd white flashes, not fully related to the themes themselves - as even Carbon was now suffering from them - and that's kind of the issue we had been trying to tackle here.

@josete2k - I put together a new build. Same place, same instructions. It should have reverted the changes that caused the white flashes to start showing up. Please try it. It is not the latest - the final one will have the option to Re-Use SVGs as it helps saving memory in larger themes. That is not the case in Carbon, and it should work well.
@TMNTturtlguy - You can also find a new build in my repository if it helps. It will have the option to Re-Use SVGs. Don't use the build I'm sharing with @josete2k as it will fail miserably on your theme. Heavy themes still have problems with the limited memory on the Pi, even after reverting the changes that caused the white screens to become more prevalent. The Re-Use SVG option is the one you want to use for your theme.

In general, and after testing and profiling your theme fairly extensively recently, a few recommendations:

Definitely get all systems to point to the same SVG border file.
Select the "Re-Use SVG" option to reduce memory consumption.
Use "Fade" transition rather than "Slide", especially on the Pi. Slide works best when it can load every single system in memory, so that it doesn't have to load things on demand. It's not that it won't work, but if you hold your finger on the right or left button to scroll a lot of systems, you'll see a lot of black screens as it frantically tries to load the textures from memory. This also causes white flashes on the logos after returning from a game, as it tries to load all the textures at once again. This also happened in the old build :(
Don't set your VRAM to more than 100, unless you have increased your memory split further. Indeed 120MB makes it run as smooth as butter, but that's because it thinks it can load everything at once into memory :) You'll run out of memory after a few systems (10-15, I think) and then you get glGet errors :) I still haven't fixed ES to recover gracefully from that, so it just results in blank screens. I actually run your theme with 70MB of VRAM, I find it a good compromise, as well as using fade.

You may test the last known good build here. You will need to remove the background color from the theme's systemInfo text, as it wasn't supported at the time.

I'll still tinker with this in the coming days, as I'd rather not have the re-use SVG option as an option, but rather as a default. The problem is that it causes the flashing in Carbon, as its logos are SVGs. I have some thoughts on how to adjust that, but will see.

I'm close to calling it a day for these developments. We'll see.

Sorry the outcome isn't extremely better, but at least it should not be worse than where we were a few months ago - hopefully slightly better even.

Thanks.

EDIT: @josete2k - I have just updated the build, it should now have the option as well. You'll want to set it to false (so, do NOT re-use SVGs).

TMNTturtlguy

@pjft Thanks! You are working so hard on this, we all owe you big time.

For your information regarding my Comic Book Theme, Part of the optimization i have been working on for both the 4:3 and 16:9 versions has been reducing files sizes and deleting non used information and pointing all artwork and sound clips that are shared to one location in the art folder, so yes, all systems point to the border .svg in one location now. The testing i rand yesterday and today all incorporated those changes already.

I will run some extensive testing tonight to see what i can come up with and re-create. I think you have done an amazing job of improving to this point, I don't think any of us are expecting you to be able to make the demanding themes work perfectly, there are limits to what we can do! Get some rest!

Oh. by the way, do you want me to post my testing results to the PR comments? I can do a short summary here and a more detailed report on github? Or i can just comment here if you prefer. Thanks

pjft

@TMNTturtlguy Either way works - I suppose here is probably good!

Thanks for the encouraging words - I'm actually thankful you insisted on this being a regression, so much that I started noticing it more and it annoyed me enough to get me to spend time on this.

Unfortunately, though, it seems I won't take it a lot further, but at least it's back to where it was - with warts and all :)

No need for extensive testing, really - you're investing a lot of time in this, which we all certainly appreciate, but at this stage the changes I'm proposing are so few now that, unless you notice something really breaking, it shouldn't be a cause for significant concern.

You may test if things work acceptably for you without enabling the re-use SVG option, though, as I'd love to be able to remove it. But it will be a memory hog without it, so it's kind of a mixed feeling for me. I'f love to at least sort that one out.

But no, just enjoy your RetroPie setup and have fun - otherwise you remind me of me enjoying spending so much time reading about old games that I never find the time to play them :)

TMNTturtlguy

@pjft Reporting back on further testing!

Comic Book Theme - VRAM at 100

RE-USE Theme SVGS ON - this runs super smooth with 17 systems in slide transition I have no black flashes at all. When i enter a game and then exit the game and go back to system selection screen, when i scroll right i still get a whit box over my .png system logos. I don't get any black flashing though. This is an improvement over the previous build.
RE-USE THEME SVGS OFF - Now i get black flashes at every slide transition. It is better looking than the white flash, but not ideal. The SVG ON is definitely a very good setting to have! I still get white flashes over the .png logos the first time through after exiting a game.

Carbon Theme VRAM 100

RE-USE THEME SVGS ON - Runs very smooth. I do get a white box over the .svg logos after exting a game.
RE USE THEME SVGS OFF - I get a slight black flash moving screen to screen in slid transition. With 17 systems loaded, having SVGS ON is an improvement. With SVGS off - there is no white box over the .svg Logos after exiting a game that i can tell.

Menu - my ps4 controller hooked up over usb wireless dongle does not like the Other settings menu. It wants to skip over VRAM Limit and RE-USE THEME SVGS. It does not do this on any other menus or selection screens, but consistently on the Other Setting Menu. This just started with the introduction of the SVG ON/OFF. My keyboard performs fine with no issues. This is a minor detail, I wouldn't worry about it to much.

My thoughts. Honestly in both carbon theme and comic book theme i think that this is an improvement over where we started from. I think that the SVG ON option is a much smoother running system than with it off. The only downside is that the .svg logos get the white box. It appears that .png logos always get the white box after exiting a game in both the ON and OFF options. I think you did an exception job of making this a much better running system. To be able to run a heavy theme like comic book with 17 systems and get smooth transitions at 100 VRAM is unheard of! Nice work and thank you!

pjft

@TMNTturtlguy Thanks for the report, and as usual for the extensive testing.

I tend to agree that it is marginally better than where we started from, though I am still slightly unhappy about the status of things as a whole. I don't want to keep Re-Use SVGs ON as a default as it breaks previously functioning themes like Carbon, unless I can figure out how to prevent that from happening. So I'll keep looking into that, even if for a bit more. :|

Hopefully we'll be able to take this somewhere, but now it's more a matter of persistence and a bit of luck. Oh, and a tiny bit of masochism, I suppose! :)

I'll keep you posted. Thanks.

TMNTturtlguy

@pjft So i was laying in bed trying to get some rest last night and I thought of another test to run. As I noted above, the .png system logos flash after exiting a game with the RE-USE SVG in both the on and off setting. The .SVG system logos in Carbon only do so with SVG ON.

To test out my theory on .png logos flashing all the time I completely removed the .SVG outline boarder from the comic book theme. As you would expect the theme now runs smooth in with .SVG On and Off - no difference. However after exiting a game, the .PNG logos flash when scrolling through the first time with both .SVG ON and .SVG OFF. At this point I don't think it is a memory issue with the comic book theme that causes the .png to flash in both instances as I have removed the .SVG border. The .PNG logos are also very small and lite weight, so they shouldn't be causing any memory issues on their own. The comic theme slides with no "black" screens or flash, but that logos are an issue.

Second thought to this. Not sure that i know if the .png logos have always flashed or not? The entire screen flashed white in previous builds, so it would have been impossible to see. I don't have time before work to revert to the older build to test. But thought I would share some insight with you.

Sorry if this adds more complexity to the plate? I am unsure how many old themes use .png for system logos, but I think new ones in development will be. It appears that the theme ruckage is building has graphics for logos, so those will certainly be .jpg or .png. (i did not tag ruckage as there is no need to bother him with this post)

Again, thanks for all your hard work......Take some time to rest, no need to fret about this to much!

TMNTturtlguy

@pjft OK - so I had one more thing to test before work today, and I have done it! I have successfully found a way to create a crisp black border that is not an SVG file! I have successfully created a crisp clear PNG file to use as a border. It is still a separate file, but it is not an SVG. I can now run super smooth at VRAM 100 and the REUSE SVG OFF.

BUT...we still have issues with flashing after exiting a game. So i can run super smooth prior to entering a game. VRAM 100 SVG OFF. I enter a game, exit and when I scroll Right i get a slight black flash and a white box for the PNG system logo. The black flash and White box happen at the same time. This happens the first time through all 17 systems, once through the systems 1 time, they run smooth again. I can create a video to post later in the day if you would like to see this in action.

@pjft your continued efforts on this and other initiatives have kept pushing me to figure out how to fix my theme, and it appears we have done it! One small success today, let me know what i can do to assist you as you continue to trouble shoot. Thanks!

pjft

@TMNTturtlguy Hah. The moment you start thinking about this at night it's when you know you're in too deep :)

Yes, at this stage it's a problem with the logos in particular - in fact, I'll be more explicit: with the fact that we load two copies of the logos in memory - one for the selected version, the other for the unselected one. But it's all a bit convoluted.

I'm intrigued by your experiments with PNGs, though. Will you send that version of the theme my way for me to test? I'd love to see if it takes up less memory than the SVG (I doubt it, if it's 1920x1080), but in particular if it's being re-used in memory.

The Re-Use SVG option is not ideal, and I'm not happy with it as it brings all these other smaller side effects into play, which I'm not that keen on having to deal with, as it feels like the end result might be much worse than the original one.

A test for you, then, if you're keen on testing:

If your PNG overlay is 1920x1080 (which I imagine it may be), why not just having a single image, without overlay? i.e. having a single full res image for the system, with the border in it? I imagine (though I'd love to check that myself afterwards) that that overlay - if full res - will take up exactly the same amount of memory as any other full-res image. So, in effect, you're loading, for every system, a full-res overlay (that may perhaps be reused), and a HD image with the background.

Just a thought, if it's easy for you to test - though I think I had sent you earlier one such theme, with a single 1080p image as background.

Thanks for the tests here.

I don't think a video is necessary - I know exactly what you mean.

I am thinking that, if we get rid of the re-use SVG option (which I'd be in favor off), I might try to implement an option to "force reload all images" when coming back from a game - if, of course, that actually solves it. :)

It'll cause the return to ES after a game to take a bit more time, but it could perhaps make things work in heavier themes? What transition are you using these days - slide or fade?

Thanks.