New emulation method to eliminate input lag

PiBoy

The Libretro team is working on a way to have "lagless input" in emulation. As I understand it, the method involves running the emulation ahead of what the use sees. The game is emulated X number of frames ahead behind the scenes, allowing the software to stay in sync with input, even at 60 frames per second. This means emulated games can run with less input lag than the original cartridges+system.

From ArsTechnica article:

An experimental Input Lag Compensation mode being rolled out in new versions of RetroArch fixes this issue by basically fast-forwarding a few hidden frames behind the scenes before displaying that first "reaction" frame in the expected spot. So in a game like Sonic the Hedgehog, which has two frames of input lag, the game will quickly emulate two additional, hidden frames after every new input. Then, the emulator actually shows the third post-input frame (where Sonic first shows a visible reaction) timed for when the first post-input frame would naturally appear, cutting out the delay a player would usually see.*

https://arstechnica.com/gaming/2018/04/better-than-reality-new-emulation-tech-lags-less-than-original-consoles/

https://forums.libretro.com/t/input-lag-compensation-to-compensate-for-games-internal-lag/15075

Before anybody gets too excited, this functionality may never come to the RPi, in it's current form. In order to perform without input lag the emulator needs a beefy processor, beyond what the RPi can provide.

MajorDangerNine

Yeah, it's pretty cool. I downloaded a Nightly build of RetroArch on my PC to try it out and the responsiveness was great.

There is now incentive to run faster yet less accurate emulators in RetroArch for those that generally have the processing power to spare, though even my modestly recent hardware can't run PSX or N64 with that turned on to the lowest setting of one (I didn't try out lr-pcsx-rearmed as I only have a CHD PSX library on my PC: I have a feeling it might be possible with that).

I do see many that play the 8-bit or 16-bit systems enjoying the games more with this feature. I know I certainly enjoyed having it set to two for the NES and SNES games I loaded up. Setting the feature to anything higher than two is when I begin to notice the constant rewinding, which makes the games much less visually appealing to me.

markwkidd

I don't know that there is reason to automatically rule out that 8bit and 16bit console emulators could use run ahead to cut 1 or 2 frames of lag on the faster rpi systems.

It'll be interesting to see testing here once retroarch 1.7.2 is released

grant2258

the thing that concerns me is how retro arch handles things that are not 60fps when a system doesnt run 60fps(ntsc) 50fps(pal) and some arcades with 30 or 40 fps. If the system doesnt fit in that profile how should i handled on the retroarch end properly. The only effective way i found it to overide the video timing is with with the sound timing seems to work well

Darksavior

I've tried it. You can use it in fceumm. I previously tried run-ahead of 4 but I noticed it felt differently. Scrolling looked jittery. Felt ok on 1. I personally don't notice the better input lag from using any number so I reverted back. Waste of time, really. I'm beginning to think most people complaining just have horrible gamemodes on their tv's or just don't enable it at all.

mediamogul

@darksavior said in New emulation method to eliminate input lag:

I personally don't notice the better input lag from using any number so I reverted back.

Most times it should be practically imperceptible. A good test would be to observe it on and off while fighting Tyson (007 373 5963). The 'Clinger Winger' stage from 'Battletoads' would be an even better test, as you have no more than four frames at each corner to get the speed boosts necessary to complete it.

tekn0

I just tested this on a Pi3B with the newest Retropie 4.4 and retroarch 1.7.3 compiled from source. It did make a nice difference. I was able to increase the fceumm core 2 frames, and picodrive 3-4 frames. It's awesome actually.

mediamogul

@tekn0 said in New emulation method to eliminate input lag:

I was able to increase the fceumm core 2 frames

That's great news. Two frames are all that's needed in most games. For anyone who doesn't know, you can test how many frames of lag a game has by using the RetroArch pause feature, then hold down a button that will invoke an action in game. Then, use the frame advance feature to move forward one frame at a time until you start to see the action take place in game. However many frames it took to begin the action animation, minus one is how many frames of lag exist and it can vary from game to game. Two frames will do wonders for certain games and I'm glad to hear it's possible in lr-fceumm.

tekn0

@mediamogul I was also able to improve performance further on my setup by changing the audio driver to sdl2. Alsa seems to have some cracking issues on my setup.

All snes cores did not work or were too slow. Also mame2003 would error in all kinds of ways, but things are very stable and smooth with 2 frames on fceumm and picodrive.

I am not sure if this applies, but I was reading stretch has some performance issues in general with OpenGL (https://github.com/RPi-Distro/repo/issues/79) I did try with other video drivers and performance was about the same.

dankcushions

@tekn0

I am not sure if this applies, but I was reading stretch has some performance issues in general with OpenGL (https://github.com/RPi-Distro/repo/issues/79) I did try with other video drivers and performance was about the same.

you should read the full comments - that’s about opengl software rendering. retropie uses gles (not opengl) hardware (not software) rendering, and there’s not performance issues with that as far as i’m aware.

tekn0

@dankcushions Thank you for clarifying, I was not sure.

markwkidd

@tekn0 said in New emulation method to eliminate input lag:

mame2003 would error in all kinds of ways

I think this would only work for mame2003 drivers which have savestate support. Do you know if you were trying a game with savestates?

I've read reports of people using runahead with mame2003 but I don't know which games they were playing. I assume anything that works with netplay in mame2003 is a good candidate, but I'm not aware of a list of those games either

tekn0

@markwkidd I only tested 2 games, Aliens, and Aliens vs Predator. They both support save states.

markwkidd

@tekn0 said in New emulation method to eliminate input lag:

@markwkidd I only tested 2 games, Aliens, and Aliens vs Predator. They both support save states.

well dang, there goes that theory

tekn0

@markwkidd I also just tested the same games with the appropriate rom set versions for FBA. I was only able to get 1 fame on Aliens vs predator. Frame-rate was to slow for even 1 frame on aliens.

The most important thing to note was that for FBA, run-ahead has to be enabled after the game is running. If you make it a core option at boot the rom will error. I has second instance enabled during all tests. It seems to not impact performance.

Performance was the same on fba2012 and fba current.

Other games tested on FBA
Metal Slug = 1 Frame
Street Fighter Alpha = 1 Frame
Aliens vs predator = 1 Frame
Street Fighter 3 = Slow
Street Fighter 2 = Slow
Aliens = Slow

Other emus
Fceumm = 2 Frames (all games tested) (core option worked fine)
picodrive = 2 Frames (all games tested) (core option worked fine)
Genesis-GX = 1 Frame
Snes9x (all versions) = Slow
mame2003 = crashes and errors (as core option and enabled after)

mediamogul

I finally got around to testing RetroArch's run-ahead feature tonight and at first blush, I can say that it's freaking fantastic. As an initial test, I was able to reclaim three frames of input lag on 'Battletoads' for the NES. A quick Game Genie code got me right to the infamous 'Clinger Winger' stage and with very minimal effort I was able to get about two thirds through it. After that the level starts to get really insane and it would take some good old fashioned practice on my part to proceed. However, for the very first time this stage feels beatable under emulation. Time is something I don't have a lot of at the moment, but I can't wait to sit down with this and knock a few "impossible" retro games off my bucket list.

Edit: Moved this to a more appropriate thread.

After experimenting with this a little bit more in lr-fceumm, I found I was able to successfully set the run-ahead frames as high as '5' without affecting performance. Previously I saw stuttering issues past '2' on some titles, but after reading more about the feature, it was indicated that audio stuttering can occur without setting a second instance of RetroArch. It's been assumed that the Pi couldn't handle this, but once enabled, the stuttering went away completely and allowed for a much higher run-ahead frame count. Granted, most titles don't need anything more than a setting of '2' and giving those games a setting higher than they need results in some interesting character behavior, but it shows that a run-ahead of 1-5 frames is not out of the question when needed for this core at the very least.

mediamogul

Man, I just can't stay away from this. I've learned that the method being spread online for determining how many frames of input lag a game has, is usually missing one step. I, myself have posted:

For anyone who doesn't know, you can test how many frames of lag a game has by using the RetroArch pause feature, then hold down a button that will invoke an action in game. Then, use the frame advance feature to move forward one frame at a time until you start to see the action take place in game. However many frames it took to begin the action animation is how many frames of lag exist and it can vary from game to game.

However, it should read:

For anyone who doesn't know, you can test how many frames of lag a game has by using the RetroArch pause feature, then hold down a button that will invoke an action in game. Then, use the frame advance feature to move forward one frame at a time until you start to see the action take place in game. However many frames it took to begin the action animation , minus one is how many frames of lag exist and it can vary from game to game.

This test has yielded a surprising result on a few games. Most notably is 'Battletoads'. Both it and 'Mike Tyson's Punch-Out!!' are always the main offenders when discussing input lag. After running the test, 'Mike Tyson's Punch-Out!!' had a whopping, yet unsurprising five frames of lag. However, 'Battletoads' only had one, like almost any other NES game. This means that the margin for error under ideal circumstances on stage 11 was simply programmed intentionally to offer one of the most unforgiving windows of user input in video game history, allowing no more than a few frames on each corner before your speed boost will fail and bring the spinning wheel a little closer each time.

As noted above, before testing the actual existing amount of lag, I was able to do very well on this stage by setting the run-ahead frames to '3'. However, since this is two more frames than actually exist, a setting of three would be tantamount to cheating, in the same way as using anything similar to alter the inherent game play difficulty, such as slow motion. Anyway, I've spent more time on this than I should have today, so I'll stop here, but I have to say that this is truly an amazing feature that's bound to dispel many more preconceived notions of input lag before it's all said and done, as well as dramatically improve how we control our games when used correctly.

pjft

@mediamogul so, now that you have managed to get me some work for my very little free time, tell me what exactly do I need to do to get this running:

update RetroArch from source
configure fceumm and picodrive (any others?) with 5 and 2 run ahead frames.

What's that with the 2 RetroArch instances?

mediamogul

@pjft said in New emulation method to eliminate input lag:

update RetroArch from source

Actually, the latest binary (1.7.3) already has this feature.

configure fceumm and picodrive (any others?) with 5 and 2 run ahead frames.

First, you should test to see how much lag the game in question has. It seems as though one frame is the most common for NES if you were to perhaps want to make that a baseline setting for the system. The test noted above is simple in it's proceedure, but my own input settings conflicted in such a way that made it way more complicated and the same will likely prove true for others.

I ended up creating a preference file that I could load separately with default keyboard input settings that also ignore automatic controller mappings. This was still somewhat inconvenient, so I believe I'll be doing further testing on my desktop computer and applying any changes to the Pi afterwards, as the amount of lag did not vary between the two. To keep it as maintenance-free as possible, I've chosen a baseline setting for a whole system and configured the outliers on a game-by-game basis as I feel it's necessary. This is a good opportunity to note this feature's biggest obvious drawback, in that there is no truly universal setting. This being the case, some games will always need their own custom configuration, like 'Mike Tyson's Punch-Out!!'.

What's that with the 2 RetroArch instances?

Setting this seems to ensure a greater stability and appears to be necessary past a few frames with the cores that can handle it. It runs a second instance of RetroArch to run-ahead. This in some way prevents audio stuttering in cores where audio is affected after a save state is restored. It also allows the rewind feature to operate without glitching the graphics in at least lr-fceumm. In my testing of lr-fceumm with a stock Pi3b, I was able to maintain a steady 60 fps with two instances, rewind enabled and five frames of run-ahead, which is really a step beyond where most of us thought performance would be with this new feature.

tekn0

I am really glad this is working out for you! I agree on the new level of smoothness. I felt great zipping through mega man games and contra on the NES. Also sonic felt wonderful on the Genesis.

I personally just keep it at 2 frames. That seems to be a sweet spot for all the games I tested. Hopefully one day a snes9x patch could allow this feature to run on the pi3.