Please test: Using OMXPlayer as video renderer

Nismo

@pjft As far I know, the problem is that the video it's rendered on top of all, and you can't move the video to the background.

pjft

@Nismo oh. I completely misunderstood what we didn't have control over then.

fieldofcows

@pjft said in Please test: Using OMXPlayer as video renderer:

So, here's a suggestion which was what I tried to describe earlier - I'm unsure how feasible it is, so I wonder if it's doable:

move the video to the background
when rendering the theme on top of it, force a fully transparent area to be rendered on the video area, creating effectively a "hole" in the theme's background. You may need to process the background image on load to turn it into a transparent background with that area unfilled.

Good idea! But I've already tried it. The problem is that the way the layer is setup by SDL completely covers any layers below it. I set the video to render below the SDL layer then actually stopped ES from rendering anything and I still didn't see the video.

If I could get hold of the dispmanx handles from SDL then I could experiment a bit further but I would have to use internal SDL structures to do this which would make it fragile when the system is updated.

pjft

@fieldofcows Got it, that makes sense. Thanks for elaborating.

And I assume that it would be extremely hacky to separate the rendering of the video overlay layers from the rest of the theme, so that you'd render everything from the generic theme using the standard 10000 SDL definition, then you'd render the video on 10001 or something, and then force render the overlay elements only separately on top of the video?

I was looking through the code but couldn't find much. Still, I like the way you used omxplayer to render the video - it's smart and effective.

A suggestion, after searching for the code, and then doing a couple of Google searches. Read the following thread for inspiration:

https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=101383&p=702598

Would it be feasible to just not render the overlay on the current ES rendering pipeline, but just use something like

https://github.com/AndrewFromMelbourne/raspidmx/tree/master/pngview

and execve it to render the image on a layer on top of the video, at the designated x and y coordinates?

I know it's hack-ish, but hey. Just trying to help out and think about options.

If I do have the chance (which I doubt, unfortunately, as everybody is sick at home...) I'll try to get your ES build with OMXPlayer to run a video, and then via SSH attempt to run this PNGview thing on a layer further on top, hoping that we can take over the rendering pipeline.

Well, just a thought. Thank you so much for thinking through this, though.

BuZz

Some more solutions

statically link with upstream vlc with h/w acceleration enabled (Will add significant build time etc).
not worry for now until h/w acceleration is included in Raspbian by default (the next version I should think).

I don't think it's a major problem anyway for now.

I may at some point include upstream ffmpeg builds as a module, which could be used by attractmode / retroarch etc. Perhaps ES could utilise them (would obviously need to use the probably lower level ffmpeg api rather than vlc's), but you would have the increased buildtime of emulationstation. Waiting seems a good option :-)

pjft

Agreed, it's certainly not a blocker nor a big deal. I suppose, if people are using 480p videos (I believe the HQ videos from EmuMovies are in that resolution) it can certainly degrade a bit the performance, or not render without artifacts (480p@60fps) but there are workarounds, for sure. It's certainly a good perspective.

A question on your second bullet, though, just to clarify. What would that entail/how would that ultimately influence these developments? Would that mean that "out of the blue" the same build would certainly start working, as the VLC being called would suddenly default to HW acceleration?

Or how do you envision its applicability in ES and its development?

Apologies if it sounds naive, but want to understand better how these intricacies work, if you do have a moment to spare. :)

BuZz

@pjft It might need a code change to enable the h/w decoding - I don't know as I've never used the vlc library api.

fieldofcows

@pjft I guess the PNG thing would work but it would be less than ideal and imagine would result in other rendering bugs (such as rendering on top of the menus, etc.) that would all have to be sorted out.

Currently I'm approaching this from a different angle. I don't see any reason why we can't pull the source in for omxplayer, patch it a little bit then build it as a library. This gives us some flexibility then with the rendering.

What I would do is provide a patch to modify the rendering pipeline to render to an EGL texture rather than directly to the screen as omxplayer does at the moment. I could then use the existing VLC-based video code to render in ES maintaining overlay support, etc.

This is all theoretical at the moment because I don't know a great deal about the HW acceleration beyond what I read at the weekend. I might try giving it a go later but I need to extract my pi from my bartop cabinet because I've destroyed my development pi :(

BuZz

@fieldofcows This sounds like too much work vs reward - relying on a patched omxplayer etc - and more to maintain.

If going that route it might be better to just build against upstream vlc/ffmpeg with acceleration.

(Although as I said, I think it's better to wait).

BuZz

I'm happy to buy you another rpi3 from the project donations btw (if you want).

pjft

@fieldofcows Got it. I can see how that could be a pain, monitoring all input events to display and hide the texture depending on navigation and menus. Well, it was an interesting exercise in theory :)

If you're considering pulling the source from omxplayer, would it make it less cumbersome just to follow @BuZz 's suggestion of statically linking with upstream vlc with hardware acceleration enabled, and using your VLC code, seeing if there's an improvement?

After having compiled mame a few times this last weekend during the 4.1 upgrade, I can't think it'll become as painful as that. ;)

@BuZz Thanks for the clarification.

@fieldofcows @BuZz Happy to contribute to that as well if you want. I can make another contribution to RetroPie for my part. Seems fair to say the least.

MoebiuZ

Why not using OpenMAX libraries to decode and render to texture instead interfacing with omxplayer? That would solve the overlay issue.

https://www.raspberrypi.org/forums/viewtopic.php?t=6577

fieldofcows

@BuZz said in Please test: Using OMXPlayer as video renderer:

I'm happy to buy you another rpi3 from the project donations btw (if you want)

Thanks BuZz but I've just ordered one - should arrive tomorrow. I don't need donation money for it. I've bought a case for it as well so hopefully won't short this one out :)

fieldofcows

@MoebiuZ said in Please test: Using OMXPlayer as video renderer:

Why not using OpenMAX libraries to decode and render to texture instead interfacing with omxplayer?

I think going this route you would just end up performing exactly the same as omxplayer itself. There is a lot of boilerplate required to get this working.

fieldofcows

@BuZz said in Please test: Using OMXPlayer as video renderer:

This sounds like too much work vs reward - relying on a patched omxplayer etc - and more to maintain.

The voice of reason. I just don't like being beaten by a problem!

I'm not sure I'm happy with the current omxplayer solution though because out of the few themes that support video, most seem to require an overlay. I don't want to see people reporting problems saying "box art is displayed under video" and things like that.

I don't think the accelerated VLC will work either because I believe the accelerated pipeline only includes full-screen rendering output. I haven't checked that for certain though.

Maybe there isn't really a problem here to fix? Is the temperature thing really a problem or should people be fitting adequate cooling to their pi's if they run into over-temperature issues? Gargh! I don't know!

How about we leave the omxplayer integration as it stands now but set VLC to the default? Therefore if anyone complains of overheating they can switch over?

Nismo

@fieldofcows +1 for vlc by default, theme developers can provide information if omx it's supported by the theme or if it requires vlc.

Or better suggestion, maybe adding a tag on main.xml ... Can we make ES autoselect default renderer based on theme?

<view name="video">
    <video name="md_video">
      <renderer>VLC</renderer>
    </video>
</view>

Of course we need also a tag on es_settings.cfg...

<bool name="OverrideThemeBasedRenderer" value="false" />

This one is for people who don't want the renderer be changed because of theme. Also maybe could be possible to add a ES menu entry for this one...

If doing the above commits we need to change this entry:

<bool name="VideoOmxPlayer" value="true" />

For this one:

<bool name="VideoPlayer" value="VLC" />

Where you can select VLC or OMX. This only will work in case you override the theme based renderer. If wrong value inserted here ES can override this entry with the default renderer (VLC).

What do you think?

pjft

@fieldofcows I wanted to have replied yesterday, but only had access to my phone and I didn't want to type it all there.

First of all, a big thank you for all the effort you're putting into this project. It has certainly helped revitalize ES, which was - up until a few months ago - a front-end that a lot of people in the RetroPie community were starting to give up on and even consider scrapping altogether in favor of others like AttractMode. You deserve a ton of credit for that, and I am positive that it can only get better from here!

A few questions you asked:

a) Maybe there isn't really a problem to fix?
b) Should people be fitting adequate cooling to their Pis if they run into over-temperature?
c) Leaving OMXplayer integration but set VLC to default?

So, in regards to a), I believe I was one of the first to escalate this issue. Truth be told, I've re-encoded most of my videos to 360p@30fps, and since then I've had no more problems. So I'm really involved in this at the moment because I feel it could be better for everyone, especially since with Kodi and other applications users are used to having videos of higher quality being rendered without overheating. That being said, it is perfectly functional as it stands, except for a few cases:

480p videos with official Pi case (or a case without a fan) will overheat. See here and here (even though @abodi claims these are the low quality from EmuMovies, which are 240p?).
480p videos at 60fps will have glitches when rendering.

This would not be a problem except that the EmuMovies HQ videos are 480p, and several are 60fps (I believe the SNES ones at least, and a few other systems).

So I can certainly envision some users attempting to get videos and running into these problems out of the box - as you can see even from the limited sample here in the forums. Certainly, people can re-encode videos or add cooling to it, but that's an additional step that not everyone may be equipped to do.

As for b), I've always seen mentions that the Raspberry Pi shouldn't need a heatsink or any cooling. After using mine for a while without any overclocking, I find that this is not always the case, but still I believe the recommendation stands in the sense that "it shouldn't be needed for the majority of situations". And certainly, if people can run videos of higher quality in other applications, there can certainly be the reasonable expectation that running videos in ES should not require (or have the recommendation) of adding cooling to the unit.

So, it all leads to c). I believe leaving the OMXplayer integration as it stands, as an option (you can mark it "experimental" in the option), provided that the video stops rendering when we open a menu (so that it doesn't overlay on top of menus and such), is probably a good compromise for the time being. Of course, all of this is assuming that @BuZz would be supportive of that, from the main ES branch perspective's, given that I expect that ultimately the goal is to have this merged back to it.

And, of course, assuming that it doesn't add a significant maintenance overhead for the project. If it does, I'm supportive of just halting this development at the moment and either try to integrate with HW accelerated VLC upstream, or wait for HW acceleration be the default in Raspbian, and then go from there. I certainly don't like to be defeated by a problem that feels solvable, but ultimately I'm not really the one facing these challenges hands-on, and it's comparatively easy to stand in the sidelines commenting, chiming in on things and supporting an idea.

As I said, I am currently fine with my particular setup, but would like to help solving the problem for all users - in particular new users out of the box. That is my key investment in this thread.

@Nismo : I think your ideas from a theme perspective have merit, and would provide some flexibility with the two players. From a long-term perspective, however, my personal take is that I fear they may add too many unnecessary changes and maintenance hooks for little benefit.

I believe we all agree at this stage that overlays are a key component of themes, and that the expectation is that in the future there will only be one single video player that has both good performance and allows for overlays. As such, temporarily changing the themes' schema just to accommodate a temporary situation with two players is likely bound to not add for a lot in the grand scheme of things.

If VLC is the default player, I expect that themes will work just fine for everyone, and if anyone needs to turn to OMXplayer, it'll be because they need the better performance/lower temperature, but accept the consequences of lack of overlays. But, truth be told, it might be that OMXplayer will not make it to the main branch as it stands at the moment - in that sense, @BuZz would be the main person to comment on what he'd consider the right take for the RetroPie ES project, given that ultimately he'll be the one approving any PR. Would having the two players as options be acceptable, even if it doesn't have overlays? Or is that something you'd not encourage in the RetroPie main ES branch?

My [extremely long] 2 cents here, and a big thank you to all of you for joining together and trying to tackle this challenge. I'd certainly love to hear about what the next steps are here.

Thanks.

Nismo

@pjft For videos without temp problems and glitches, I'm uploading optimized art and videos for Rpi here: https://retropie.org.uk/forum/topic/8019/oldroom-theme-w-i-p-media-packs?page=1

Videos are at max 360p 25fps, with very good compromise between quality and filesize. I don't have a rpi to test them, but I expect all will be ok with the videos now under VLC.

Sorry for the offtopic.

pjft

@Nismo Thank you sir! I had seen it earlier but I see you have added a few more now - I'll look into those and certainly update my OldRoom theme :)

Nismo

@pjft Yes I'm uploading art regularly, so keep tuned. Also a big update is coming for the theme, with some bugfixes and more systems.

Please if you try my videos, can you tell my if you have some temps or glitches problems?

Thanks.