@redmarsbluemoon Yes, stock Raspbian. No need for userland ds4drv :) Imperceptible latency atlhough I've never measured it, I can easily play even games like Street Fighter II.
Try updating your packages/kernel? apt dist-upgrade and rpi-update.
Also be careful about interference from other wireless sources (Wi-Fi, other 2.4 GHz devices)... Maybe try moving things around and seeing if that helps. Best of luck.