Ignore that - I misunderstood (And the PS3 controller does work ok in Ubuntu 16.04 unlike I said above - I just had to configure it by holding the playstation button it seemed).
You mean it's fine in ES and games, just no good in the retropie-setup menu ?
It works - it's just it doesn't use the same mapping. Left analog moves, and select will choose an option. This will be resolved at some point in the future, but the menus were never originally designed for navigating via joypad - it was added later via joystick to keyboard mapping. Since it;s mostly configuration and you are on a PC, it's not a big deal to use a keyboard to configure if you don't like the ps3 mapping.