I got this fixed. First I set up the PS3 controller "manually" using this post as a guide:
https://www.reddit.com/r/RetroPie/comments/3ye3f0/cant_manage_to_configure_dreamcast_controls/
Then, I did the first two things on this post (1 - disabled emulationstation's automatic configuration of reicast and 2 - modified reicast's config so it attempts to "read" the proper joysticks from /dev/input)
https://retropie.org.uk/forum/topic/2869/reicast-not-detecting-xbox-360-controller-partially-solved
I noticed that when the keyboard was hooked up it was controller id 0, and the PS3 was Id1. With no keyboard the PS3 controller was Id0 and worked. So I told it to look for id1. But then it stopped configuring the controller in reicast, so I had to do it manually.
Now the PS3 controller works - would be nice if I didn't have to tell it which controller in case the numbers change, but it works and that's what really matters.