Some mouse-only games are still setup to use just controllers and not the attached mouse/keyboard when you use the WHDLoad version. Populous, for example. I don't know what i need to do to unset that, so I can't make any suggestion for you on this either (if it's the case).
My only other suggestion would be to disconnect your controllers before firing up an amiga game and/or switch USB ports so that your mouse and keyboard are on the lower port numbers (0 and 1).
And the fact that most amiga roms are of the cracked games, not the originals!!!
The originals are out there. I have a friend with an amazing collection of original boxed Amiga games. He's imaged many of them to run on his more modern devices like the Raspberry Pi and they perform very well.