Please test: Using OMXPlayer as video renderer

backstander

@pjft
I believe QEmu doesn't emulate the PRi GPU hardware and EmulationStation needs that to load:
https://retropie.org.uk/forum/topic/448/help-retropie-virtual-emulation-with-qemu/7

pjft

@backstander thanks for the heads up, I wasn't aware of that. I got in trouble even just compiling all things. I based it on a Wheezy image as I struggled with getting Jessie to work, but then even after getting all the libraries installed, from Boost to FreeType and in between, it seems to struggle with VLC. I suspect the required version for the current build isn't available for Wheezy. Which is fine.

I'll see if in Ubuntu the experience is smoother.

Thanks!

pjft

Yay! Release day, finally after doing this on a few minutes a day.

Very minor release for the moment, but thought I'd share. Also, my first release and overall experience with GitHub, so I'm quite happy. Most of the changes are minor/quality of life, but should help people on the RPi and PC have better temperatures while also getting screensavers and the few updates (@Dominus, @abodi, this will help your temperature issues based on @fieldofcows 's changes).

Just to make it clear: the extreme majority of this code is from @fieldofcows - I just added the following changes.

A few things in the release:

Keep aspect ratio for videos in OMXPlayer. Vertical games are no longer stretched. Haven't fixed it for VLC yet, but may look into it in the future.
Added game name and system on screensaver (@mattrixk , @CourierSS , @Dominus) . Very hacky, done via subtitles, but for the time being it's better than nothing (I hope). I will make these an option on the ES menu in the future. Please test on PC - the could should be fine, but VLC rendering performance on the Pi wasn't good in full screen resolution (the default video resolution made subtitles be very blocky, and stretched, and full screen just halted to a stop), which leads me to the next item.
Made OMXPlayer the default screensaver player for the RPi. No reason for that not to be the case, as there are no overlays for screensavers (yet, at least). There's still the option for having either VLC or OMXPlayer for the Video Gamelist View, of course, for overlay purposes. This way we can get a nice full-screen video scaling via the player rather than the texture stretching from VLC. PC should still be fully VLC-based, and have name and system as subtitle. If one day we get the game name as an overlay during the screensaver, this will be reverted - the code is all there, just commented out.
Pressing Select on the System Select screen will launch video screensaver. Why? Well... I have some plans that I'm sure @Zigurana will like :) I'm thinking of trying to use this as the random game selector, and if we exit the video screensaver with START, it will just select/launch that game. I will update if I end up succeeding, or just failing miserably. My several attempts at playing with the inputs so far resulted in the latter, so I'm not extremely hopeful, but I think it is a useful way to have a random game selector for the time being :P

I also did a few minor fixes on the screensaver via OMXPlayer, and when choosing a random video would select a path that does not exist (from @Nismo 's batch gamelist script, it happened on some BIOS roms in some folders for me, or ROMs I deleted but did not update the gamelist for).

To install this release, I'd follow the usual instructions from @fieldofcows :

wget https://github.com/pjft/EmulationStation/releases/download/v0.1-ss_title/emulationstation

Make executable and replace the original ES

chmod +x emulationstation
sudo cp emulationstation /opt/retropie/supplementary/emulationstation/

You may need to install dependencies, if they're not yet installed (I suspect they may be if you're on the latest RetroPie):

sudo apt-get install libvlc5 libvlccore8 vlc-nox

@fieldofcows if you feel any of these contributions are useful to your repository, I'm more than happy to send a pull request at this stage - I had sent out a smaller one earlier, but I certainly do not want to impose on your work. Thank you for taking ES this far!

Jolly good fun.

Nismo

@pjft I think that keep aspect ratio by default it's an error, what I mean it's that keep aspect ratio must be optional for the themer like we do actually with the images by using "size" or "maxSize" tags.

Example 1, not keeping the AR, just using a fixed size:

<video name="md_video">
			<pos>0 0</pos>
			<origin>0 0</origin>
			<size>0.35 0.40</size>
		</video>

Example 2, keeping AR, using maxSize instead:

<video name="md_video">
			<pos>0 0</pos>
			<origin>0 0</origin>
			<maxSize>0.30 0.40</maxSize>
		</video>

The problem it's maxsize it's not implemented for videos yet. Some themes like mine need a specific size for the video because it's "inside a box" and keeping AR will broken the theme or make it look "ugly".

Regards.

pjft

@Nismo thanks for the feedback! I believe the main system where this ends up being a problem is in the arcade, correct? Just out of curiosity, it doesn't change the fact that this is important for themes.

It's a fairly trivial thing to change on the player side - for OMX Player it's just a flag starting whether to stretch or letterbox it, though for VLC it'll be a bit more complex.

Given your previous feedback, though, I won't look into that on VLC for the time being, especially as your theme depends on VLC. When themes have that capability to distinguish max size vs size on video we can investigate that for both players.

The main reason I investigated it is because I think someone else had mentioned this earlier, and for the screensaver it made little sense to stretch the image unnecessarily as it ends up not looking like the actual game. :)

Is there any way I can provide you with a Windows build? I don't have a Windows machine to compile it on, but would love your feedback :/

jdrassa

@Nismo @pjft I have been working on adding maxSize support for video. I am planning on creating a pull request soon.

Nismo

@pjft Of course in arcades it's the main system with differents AR for games.

The main reason I investigated it is because I think someone else had mentioned this earlier

Yes it was mentioned before, and I know some people would love this, but i think that the right way to do this, it's by the same "size" and "maxSize" tags we are already using for images.

Another problem it's the not keeping the AR for images if you select to show image if no video in your theme, very noticiable if you use 3d boxes, that needs to be fixed:

<showSnapshotNoVideo>true</showSnapshotNoVideo>

Anyway, thank you very much to both @pjft and @jdrassa to help make this project better and better. Great work.

pjft

@jdrassa Thanks for the heads up.

I don't know where you forked it from (RetroPie vs fieldofcows) but anyway if you want to send it over to submit a PR for my repository as well, I'm more than happy to merge it so that it makes it easier in the end to converge everything.

Send it over to pjft/EmulationStation, the screensaver-control branch which is where I'm working on at the moment.

I'll try to merge things to my master soon, but can't promise.

Thanks!

Nismo

@pjft Oh! i'm not a programmer. I'm using the fieldofcows windows fork.

Thanks anyway.

Edit: would be nice to have a windows version from your fork to test.

pjft

Yeah. I don't have a windows machine to compile it to, but would love to send it your way.

If anyone uses windows and decides to build it from my repository, if you want to share the file it'd be perfect!

fieldofcows

@pjft - It looks like you've been doing some fantastic work here. I have had even less time than usual to devote to ES recently but I've been trying to keep up with what has been going on.

One thing that is worth noting is that the OMX player integration is still experimental. I'm not sure if that is the way forward for the retropie fork - do we really want the confusion of two different video renderers with different capabilities? I would much rather just pick one and stick with it and make it work the best we can.

I was all willing to go with embedded omxplayer as the only option for rpi but I think the overlay problem is a killer that is not fixable without some unacceptable hacks at the moment.

What does everyone else think? If the consensus is to keep the omxplayer work then I'm happy to collate the change/fixes into my dev branch before submitting a PR for consideration into retropie.

pjft

@fieldofcows Not at all, thanks for keeping up to date here.

My thoughts still stand as earlier - I agree that keeping two separate players makes no sense in the long run, particularly in the RetroPie main branch, and that it will be VLC. I don't have any expectation that this will ever be committed to the main branch - though I'd be more than happy to do whatever work is required to have it committed, should the project owners so decide to. The code I did works (hopefully) equally well on VLC and OMXPlayer (truth be told, the vast majority is yours - I just tweaked some minor things).

But truth is, the effort to have it running now is quite minimal, so that's what I've been doing, even if to get something running now for my own setup. If in 3-6 months the proper VLC HW support comes through, then surely we'll make that the main choice - but in the interim, everyone will be able to make the most of your work during these 90-180 days, and start developing their own things, be it their video setups or themes, while enjoying screensavers and video support in a way that doesn't fully tax the Pi processor. :)

I agree that overlays are critical for themers, and I don't like the idea of keeping both players simultaneously, so I'm fully behind VLC for the long run. But if this is the way it has to work now, I'm more than happy to live with that for the time being :) Who knows if in 3-6 months time you - or anyone - will have the time to make this work? Life takes several turns, so no point in delaying at least getting a minimum-viable-product out there for the community :)

Happy to add it to the experimental branch, if that would be the proper way to get it to reach more people until proper support comes.

Thanks!

fieldofcows

@pjft - That all makes sense, although if enough testing has gone into the omxplayer version then I'm coming around to the idea of keeping it and just using VLC by default. Tinkerers can then switch if they want to.

One of the things I've been playing with over the last couple of weeks which is just about working now is making a change to the VLC rendering pipeline to try to reduce CPU usage.

At the moment, everything is done in the CPU - the VLC library reads the video, prepares the frames and puts them through a colour conversion before calling a callback in ES which then puts this in an OpenGL texture. The colour conversion is to RGBA which actually seems particularly expensive in terms of CPU time and, in fact, in my testing it seems BGRA was less CPU intensive but I haven't proven that.

What I have been playing with though is getting VLC to skip the colour conversion and give ES the frames in their native format which is normally I420. I then put this straight into a texture and use an OpenGL ES 2 shader to perform the colour conversion in the GPU.

I got all this working on Linux but faltered when I tried it on the pi because ES is using OpenGL ES 1. So, of course for fun I went and switched all the ES code from fixed pipeline OpenGL to using shaders :)

This now works but I haven't performed any metrics on CPU performance, video playback performance or temperature yet. That is still todo.

Anyway I'll keep you all posted.

pjft

@fieldofcows I'm happy to run performance (i.e. temperature) tests on a proper Pi.

I can say that I have exclusively been using OMXPlayer, really - especially after long sessions using VLC, the temperature would always rise to a level that's higher than the ideal one. Also, the screensaver at full screen using VLC, being stretched via the texture, seems to look slightly off (but may be because of lack of aspect ratio, rather than anything else, as mathematically "it makes sense"). When I do stretch it via VLC, having VLC render the full screen resolution (in my case, 1080p), just renders a single frame or so and then halts, while the audio keeps playing. Can't blame it :)

If we're going in the VLC route then, why now attempt to use a HW accelerated VLC, even if we have to compile it ourselves? That could then be reused in the future, when the system gets its own version of HW-accelerated VLC, and it's easy to plug it into that, unless the versions are so different the API has changed significantly.

It could also serve to test properly whether HW-accelerated VLC would effectively improve the state of the nation.

What's the right step for us to do that? I imagine there'd be versions of it floating around - how do we link to it, rather than the current library?

Nismo

@pjft I vote for VLC as default renderer... people can use optimized videos for better temps.

@pjft I recommend to you to try my videos, re-encoded with rpi in mind, for better performance and temps: https://retropie.org.uk/forum/topic/8019/oldroom-theme-w-i-p-new-1-9-beta-media-packs?page=1

Of course I still need to upload a lot of systems , but i'm still working and uploading more optimized media for rpi (even when I don't have Rpi) just to contribute to the great work from @fieldofcows, @pjft, @jdrassa, @Zigurana and everybody who spend some of his free time to make this project going better.

fieldofcows

@pjft said in Please test: Using OMXPlayer as video renderer:

If we're going in the VLC route then, why now attempt to use a HW accelerated VLC, even if we have to compile it ourselves?

I did look at that. The problem, if I remember correctly, is that the VLC acceleration would have the same problem as omxplayer - it would not allow you to overlay on top of video. In order to make this work with overlays in ES we would need to modify VLC to use the acceleration to render into an OpenGL texture - something that is possible but may not fit with the architecture of VLC. Maybe we could do it with a plugin or something but I don't know enough about VLC at this stage to say how hard it would be.

Anyway, let's see how much my shader changes improve the performance (if at all) and make a decision from there.

pjft

@fieldofcows Huh. That's disappointing, to say the least.

I assume that's because we'd be calling VLC as an external executable, rather than via the library we currently have in place?

I'm definitely not supportive of "modifying" VLC, as ultimately that project is outside our scope of management. If we'd need to have some hardcore architecture changes, we might look into ES then, which is something we kind of have control over.

I've done quite a bit of work on ES during the weekend, to the point where I'm rather satisfied with it. I might open a separate thread to call for people to test out the changes, and keep this one for the specifics of the Video performance.

Summary of my changes:

Fixed the legacy screensavers (dim and black) that had stopped working with the recent screensaver refactoring. Hopefully I haven't brought about new problems :D
Pressing "select" in the System Select menu launches screensaver. It's an option in the menu, so it can be disabled, and only works for if "Random Video" is the selected screensaver. Help menu updates accordingly.
Added "right" during video screensaver to select new random video, and "start" to launch it (or just navigate to the gamelist position where it currently is). These are options in the menu, and can be disabled.
Added the OMX Player option in the menu (it's called "Use Experimental Player" - the default is VLC, which means it defaults to OFF) so we can change it on the fly.
Added option to display the video game name and system during the video screensaver. If enabled, it will show the game title and system as a subtitle for the first 5 seconds of the video screensaver, and the last 5 seconds as well.

This way we can have a simple "surprise me"/"random game selector" by waiting for the video screensaver to start (or by pressing "select" at the System select screen), and then launching the game directly with "start", or just skipping to a next suggestion/random pick by pressing "right" while it's playing. I find it an elegant and user-friendly approach to @Zigurana 's "Surprise Me" option in the Kids Mode fork, not to mention that if someone uses this as an actual Kiosk, if players see a game on the screensaver that they'd like to play, they can just press start to launch it, kind of like one of those old multi-game arcade machines.

I'll create a separate thread when I have a binary later today, for people to install it and test it out. I have a binary right now, but doesn't have the final option for the video game name (it's always on - forgot about that!), so might as well just wait for that.

Thanks!

pjft

@fieldofcows Sigh.

Ok, I should know better, but I really don't, so I decided to dig into everything that you probably already had looked into in the past.

So, I've been digging into this whole layer thing, especially as you mention that VLC HW acceleration suffers the same problem, which I suspect may be a common cause to OMXPlayer HW acceleration.

I tried looking into how to create a separate layer for the extra objects. No luck. The 10000 value is hardcoded in the SDL library, which means we'd need to customize it to take on different values for the different layers. That may be an option, but I am not particularly keen on that. We could also compile the custom version and just statically link to it, but... urgh. Not to mention that I really only knew how to do that back in Visual Studio, a while back.

My next best bet, then, will be to investigate creating a separate layer directly via dispmanx, for the extra elements in the themes.

I'll take a stab at that and see if I can make it as seamless as possible for rendering purposes. I'm not keen on reinventing the whole rendering system, as that's - for lack of a better word - rather foolish.

I'll see what I can make of that and report back. Is there any particular hierarchy that jumps to mind in regards to the theme's elements and sorting order? I take it that on top of the video you'd usually only get the extra elements, but correct me if I'm wrong (@Nismo as well).

If/when you do have a RPi build you'd like me to look into in for the performance bit, send it over.

Best.

Nismo

@pjft Yes, at this moment, only elements tagged with "extra" tag can be displayed on top of video and the elements without "extra" tag are behind the video.

pjft

@Nismo Thanks. I'll focus on those then.