@pikey
first off, its worth noting that you need to install a driver for the ps3 controller to connect, so if you havent done that yet, i sugest you try that first, via the retropie setup.
also linux is not windows, so the fact that it works in windows, dosnt automaticly mean it also works in linux.
does the bluetooth adapter work as expected with other bluetooth devices in linux?
edit:
also, are you using an original controller?