@Ziffel since no one else has chimed in I'll give my two cents
I use several wireless Xbox 360 controllers with a Generic Xbox 360 Wireless Gaming Receiver that plugs in USB and I was very surprised that I didn't have any noticeable input lag. I also have an ASUS Nexus Player Gamepad that connects using Bluetooth and it seams pretty responsive as well (but I only tried it once and I use it on my laptop more). I bet there's probably a few emulators that are slower than others at input but for the most part I feel like my Raspberry Pi 3 handles all this pretty well.
What kind of controller are you using?
Now you did mention TV lag which is a real thing. I had already optimized my TV a long time ago but you might try some of these and see if it helps with your input lag:
First, if your TV has Game Mode then turn that on!
Second, disable any power saving settings or ambient screen dimming.
Third, turn off all post-processing effects on your TV like dynamic contrast, etc.
Forth, make sure you're not daisy chaining the HDMI through an Xbox or another device (sometimes even amplifiers can delay signals).
Fifth, also test with each HDMI port in the back of the TV. Sometimes 1 or 2 HDMI ports are faster than the rest.
Last, maybe try other TVs and monitors to see if you still have input lag
Also make sure you have the newest version of RetroPie (I think there is some fixes to the emulator input latency):
https://github.com/retropie/retropie-setup/wiki/updating-retropie