@brandflake11 oh nice! that's sounds like a useful gadget, thanks a lot I'll look into that.
@IanDaemon yes that must have been the issue, both the pi and the switch would have been within Bluetooth range, so like you said they were fighting with each other to connect with the controller. I didn't really know much about Bluetooth before but I understand better now.
I'd say that 8bitdo dongle would be a permanent solution but even for the time being, I could try shutting down the switch entirely, rather then putting it in sleep mode so then the switch should not be fighting for connectivity with the controller.
Thanks a lot for your help guys!