Bluetooth limited to 2 xbox one controllers (4 desired)

[Razziel]

Pi Model or other hardware: Pi 4 B
Power Supply used: ksa-15E-051300he
RetroPie Version Used: 4.7.3
Built From: Raspberry pi imager v1.5 - also updated / retropie version from inside the Pi (after booting up from the downloaded image)

All 4 controllers are xbox one wireless.
A Steam link device can connect all 4 of them. I've tested and played with a Steam link device lots of times.
I am trying to simulate a steam link device, with moonlight instead of steam-streaming.

I am unable to connect more than 2 bluetooth controllers to the pi using the retropie setup or via bluetoothctl.
As soon as I pair with 2 of them, the next controller just isn't found by the pi's bluetooth.
It doens't matter the order of the controllers, I can achieve any 2 of them being paired at the same time, but when trying to pair the 3rd, the bluetooth doesn't find any controller (it finds headphones or anything else).

My xboxdrv config (also configured using the retropie setup script) is the following:
picture of the command.

I assumed that the pi's bluetooth will rock because it has bluetooth 5.0 (Steam link has 4.0). I don't know if this matters.

Help?

[mitu]

xboxdrv is not a driver (despite its obvious name) - you don't need to install it for Xbox controllers. Instead, install xpadneo and try pairing your controllers one by one.

[Razziel]

@mitu Thank you for your quick response !

I forgot to mention before, when first posting i had installed xboxdrv, xpadneo and xpad.

I have uninstalled xboxdrv, restarted, and tried to connect with xpadneo and xpad installed. Only 2 of them worked (same result as before)

After this i had uninstalled xpadneo and xpad -> then installed xpadneo only -> restarted -> try connection -> still same result.