Please test: Using OMXPlayer as video renderer
-
@pjft said in Please test: Using OMXPlayer as video renderer:
If we're going in the VLC route then, why now attempt to use a HW accelerated VLC, even if we have to compile it ourselves?
I did look at that. The problem, if I remember correctly, is that the VLC acceleration would have the same problem as omxplayer - it would not allow you to overlay on top of video. In order to make this work with overlays in ES we would need to modify VLC to use the acceleration to render into an OpenGL texture - something that is possible but may not fit with the architecture of VLC. Maybe we could do it with a plugin or something but I don't know enough about VLC at this stage to say how hard it would be.
Anyway, let's see how much my shader changes improve the performance (if at all) and make a decision from there.
-
@fieldofcows Huh. That's disappointing, to say the least.
I assume that's because we'd be calling VLC as an external executable, rather than via the library we currently have in place?
I'm definitely not supportive of "modifying" VLC, as ultimately that project is outside our scope of management. If we'd need to have some hardcore architecture changes, we might look into ES then, which is something we kind of have control over.
I've done quite a bit of work on ES during the weekend, to the point where I'm rather satisfied with it. I might open a separate thread to call for people to test out the changes, and keep this one for the specifics of the Video performance.
Summary of my changes:
- Fixed the legacy screensavers (dim and black) that had stopped working with the recent screensaver refactoring. Hopefully I haven't brought about new problems :D
- Pressing "select" in the System Select menu launches screensaver. It's an option in the menu, so it can be disabled, and only works for if "Random Video" is the selected screensaver. Help menu updates accordingly.
- Added "right" during video screensaver to select new random video, and "start" to launch it (or just navigate to the gamelist position where it currently is). These are options in the menu, and can be disabled.
- Added the OMX Player option in the menu (it's called "Use Experimental Player" - the default is VLC, which means it defaults to OFF) so we can change it on the fly.
- Added option to display the video game name and system during the video screensaver. If enabled, it will show the game title and system as a subtitle for the first 5 seconds of the video screensaver, and the last 5 seconds as well.
This way we can have a simple "surprise me"/"random game selector" by waiting for the video screensaver to start (or by pressing "select" at the System select screen), and then launching the game directly with "start", or just skipping to a next suggestion/random pick by pressing "right" while it's playing. I find it an elegant and user-friendly approach to @Zigurana 's "Surprise Me" option in the Kids Mode fork, not to mention that if someone uses this as an actual Kiosk, if players see a game on the screensaver that they'd like to play, they can just press start to launch it, kind of like one of those old multi-game arcade machines.
I'll create a separate thread when I have a binary later today, for people to install it and test it out. I have a binary right now, but doesn't have the final option for the video game name (it's always on - forgot about that!), so might as well just wait for that.
Thanks!
-
@fieldofcows Sigh.
Ok, I should know better, but I really don't, so I decided to dig into everything that you probably already had looked into in the past.
So, I've been digging into this whole layer thing, especially as you mention that VLC HW acceleration suffers the same problem, which I suspect may be a common cause to OMXPlayer HW acceleration.
I tried looking into how to create a separate layer for the extra objects. No luck. The 10000 value is hardcoded in the SDL library, which means we'd need to customize it to take on different values for the different layers. That may be an option, but I am not particularly keen on that. We could also compile the custom version and just statically link to it, but... urgh. Not to mention that I really only knew how to do that back in Visual Studio, a while back.
My next best bet, then, will be to investigate creating a separate layer directly via dispmanx, for the extra elements in the themes.
I'll take a stab at that and see if I can make it as seamless as possible for rendering purposes. I'm not keen on reinventing the whole rendering system, as that's - for lack of a better word - rather foolish.
I'll see what I can make of that and report back. Is there any particular hierarchy that jumps to mind in regards to the theme's elements and sorting order? I take it that on top of the video you'd usually only get the extra elements, but correct me if I'm wrong (@Nismo as well).
If/when you do have a RPi build you'd like me to look into in for the performance bit, send it over.
Best.
-
@pjft Yes, at this moment, only elements tagged with "extra" tag can be displayed on top of video and the elements without "extra" tag are behind the video.
-
@Nismo Thanks. I'll focus on those then.
-
@pjft that is incorrect. Everything except the background can be placed above the video.
-
@jdrassa usual error for some theme developers about video is " I can hear the sound but I can't see the video", just because they use "extra" tag in their background, so yes, you can place the background on top of the video too, every image with "extra" tag will be on top of the video, even if it's the background.
You can make a png background and let a "transparent box", some space for the video.
-
@Nismo said in Please test: Using OMXPlayer as video renderer:
only elements tagged with "extra" tag can be displayed on top of video and the elements without "extra" tag are behind the video.
This is an incorrect statement. While it is true that any "extra" elements are displayed above the video, it is not true that the elements without the "extra" tag are behind the video. Below is the order in which elements are currently rendered. Only the background element is rendered below the video.
- Background
- Video
- Extras
- Game List
- Marquee Image
- Image
- Game Metadata fields
- Header Text/Image
@Nismo said in Please test: Using OMXPlayer as video renderer:
@jdrassa usual error for some theme developers about video is " I can hear the sound but I can't see the video", just because they use "extra" tag in their background, so yes, you can place the background on top of the video too, every image with "extra" tag will be on top of the video, even if it's the background.
You can make a png background and let a "transparent box", some space for the video.
The reason that removing the "extra" attribute from the background works, is because the gamelist views specifically support background as a theme-able element. If you were to remove "extra" attribute from any other image, that image would simply not be displayed.
-
@jdrassa Thanks for the replies, and also for all the proactive work on z-indexes for the theme elements.
I'm unsure of whether you've been following this - and prequel threads - from the start, but I'll try to summarize here what we're trying to solve here. Maybe your perspective can also shed some light here.
Summary:
- VLC is not an optimal player on the Raspberry Pi. It does not have hardware acceleration, which means that videos with higher fps (above 30), or anything of 480p or above will usually lead to overheating the main board. Active cooling may help, but shouldn't be taken for granted. Still, it's quite flexible and allows for easy rendering into a texture, which means that in ES you can decide what z-index you want it in, as it is in practical terms just another element of the UI. This is very helpful for themers as it allows for other elements to be placed on top of the video (overlays, boxart, etc).
- @fieldofcows put together a proof of concept using OMXPlayer, which is a hardware accelerated video player on the Pi. It works wonders in terms of performance, being able to render high-fps and high-resolution videos with minimal effort. The current implementation, however, renders the video in an independent layer (which we determine). This means that at the moment it is either always on top of ES, or below ES, but not "in between" ES-rendered elements. It's a separate window, for the rendering pipeline.
We've been trying to understand what could be done to improve the situation for Pi users overall. @BuZz chimed in on a few options, namely either static linking with existing hardware accelerated libraries (VLC or ffmpeg), or waiting for VLC hardware acceleration to come out of the box in a future Raspbian release. @fieldofcows has done a plethora of research, namely on hardware accelerated VLC, or reducing the CPU load on the current VLC pipeline. His results seem to suggest that using hardware accelerated VLC will lead to the exact same results we currently have with OMXPlayer, meaning that for some reason hardware accelerated VLC seems to render on top of everything.
So that got me thinking, for the long term of the video add-on for ES. This is kind of what I found, and the options I considered:
- ES is stuck to layer 10000 thanks to the SDL hardcoded definition of the video layer for the Pi.
- I was considering having ES render a transparent background, so that we could render a video underneath it, and then the actual ES background being rendered separately by a direct dispmanx call in an even lower layer. @fieldofcows seemed to have tried this earlier with no success, and the answer is clear why in the same SDL definition from before. You'll see that dispman is initialized with alpha opacity at 100, fixed for all pixels, to block anything underneath. Others who have run into this problem have suggested patches to SDL, but while customizing SDL, recompiling and statically linking it is an option, it's just too painful - not to mention having to manage rendering two separate windows for the main elements of ES and another one for the background, sending the right events to each. So after some exploration and consideration, this is not really an option I would recommend.
@fieldofcows suggested customizing OMXPlayer to render into a texture. Initially I dismissed this option, because customizing an external component is asking for trouble in the long run, as other fixes and tweaks in the main project would need to be manually kept up in ours. Furthermore, the question of performance also came up: would it render all the performance improvements useless? I did some research on this and came up with the following:
- There's a component on OpenFrameworks - ofxOMXPlayer - see link for sample code - which allows for rendering OMXPlayer into a texture. Comments seem to suggest it can even render 1080p video full screen, and 720p to a texture with good performance. This could be an option, but I believe it means we'd need to include OpenFrameworks in our list of dependencies. Not to mention I can't find a mention to there existing a pre-compiled version of it for download, so it seems we would have to compile it which could add significant time on our end?
- I also found a "simpler" (i.e. non-dependency-heavy) example that directly rendered OMX hardware accelerated video output to a texture. Here's a topic on it on the Pi forums. Seems to suggest "Everything up to and including 720p encoded in h.264 and Mpeg4" was rendering well on a Raspberry Pi 2 (by the date of the topic).
- There's a separate developed who did quite a bit of work on something similar, but I think he relies on other dependencies (Qt Multimedia) and may have customized his OMX Player code.
So, there seems to be some hope in using OMXPlayer to render to a texture, which would give us both the performance we need while keeping it within the ES rendering pipeline, allowing for overlays and such.
@fieldofcows - I hope these help to some extent, at least to shed some (potential) new light on the developments. This is way out of my area of comfort or expertise, but do you feel any of this is worth exploring?
Best.
-
@pjft Attract Mode can use hardware accelerated FFmpeg and it seems to work ok (for my purposes anyway), presumably you've looked into that already? Why isn't it a viable alternative to VLC or OMXPlayer?
-
@robertybob said in Please test: Using OMXPlayer as video renderer:
@pjft Attract Mode can use hardware accelerated FFmpeg and it seems to work ok (for my purposes anyway), presumably you've looked into that already? Why isn't it a viable alternative to VLC or OMXPlayer?
You're absolutely right and I've been looking through the source code over the last few days. It's definitely something we could look into trying to port to ES.
-
@robertybob That would have just been to easy.
:P
I joke. It is certainly a viable alternative - thanks for bringing it up again. I believe I mentioned it in passing earlier, following @BuZz 's suggestion but truth is we never followed up on it and it was forgotten.
I'm happy to know that AttractMode uses it, though, and especially that @fieldofcows is looking into it already. If it works well for AttractMode, it is probably a good option for us as well.
@fieldofcows : if/when you end up managing to put together a small test build (really, don't worry about polish - just get it to render the video) I'll put it running here for a while and check the performance and temperatures. And if you need any help with the development, feel free to share the code. Happy to look into it as well.
Thanks!
-
After doing this, the carbon theme is broken where the videos are not the right aspect and are over the rom list. Is there something I can do to fix it? I was DESPERATELY needing this omxplayer because none of my videos played using vlc without hardware acceleration. OMXPlayer is playing them FLAWLESSLY but yeah... here's what it looks like: imgur
You can see the video, even the marquee is in the incorrect position.
I'm new to this so don't mind me if it's dumb to ask. I'm just so close to having things work right and I want to continue this project.
THANKS!
-
Hm. Interesting.
I don't quite know why the Carbon theme is broken, as that's the one I'm using. Maybe there's a revamped Carbon theme, based on the new ES theme changes that aren't part of what this build supports?
Anyways, feel free to install this particular version of Carbon, which is the one I'm using. I think it should be good, though I don't quite recall if I edited all that was needed.
https://mega.nz/#!MJEXUZpT!kEBemgjgE_LIkXDHPmPoPu-GhB9pMAUlPWGPoYSs_so
Just unzip this into the themes folder (/etc/emulationstation/themes, I believe?).
Just to confirm, in case it isn't clear: this build is not up to date with the recent changes to ES, namely changes to themes. Use this only if you're fine with not updating your ES for a while, as it's not clear when these changes will make it to the main ES.
Best.
-
@pjft OMG you rock. I'm so glad I asked :) It's working flawlessly now. I'm just using the theme that came with ES 4.2 (fresh image installed a couple of days ago). I don't know why it was broken... I even went and redownloaded it and they all have problems for me. The one you provided works great.
-
@alaindesjardins My suspicion is that it's using features that were developed after this one was developed.
Glad it's working out correctly. You need to select "enable experimental player". Hope you also enjoy the screensaver, and random game launch abilities :)
-
@alaindesjardins Actually, wait!
If you're planning on using OMXPlayer, use this one instead:
It's an update version of it, with the features I described. Slightly more stable, I believe, and with a few critical updates.
-
@alaindesjardins @pjft I know that @jdrassa made some changes to carbon theme to add new features and maybe he removed video support.
Anyway as far I know OMX player was experimental and will be deprecated so you need better videos adapted to your rpi.
You can try the ones I'm uploading here: https://retropie.org.uk/forum/topic/8019/oldroom-theme-w-i-p-new-1-9-beta-media-packs
I'm still need to upload more systems, but that videos should play without problems.
Regards.
-
-
Was there any further movement on this? Is there a new thread regarding it with the updates over the last couple months to ES? I am curious because I am working on adding video snaps to my Pi Zero W and trying to figure out optimization for temps on that board and performance hits. Thanks in advance and all the hard work being done! Kudos.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.