I would also be quite interested to hear people's opinions and experience with this. The advanced controller mapping guide is excellent and very comprehensive for xboxdrv however the retropie wiki does seem to suggest xpad is the preferred and is even the default.
I would especially be interested if anyone has any suggestions for xpad tweaks to improve ScummVM analog control.
After a few tries I had a working base configuration, but the joystick output was still too low. After mediamogul's suggestion and some trial and error I got a configuration which works really nice for me (real N64 Controller and HuiJia USB GamePad adapter). What's nice about xboxdrv is, that you are able to rename your controller, which makes keeping track of multiple controllers with the same name quite easy.