I will guess that you must be using the latest retropie, and since the way reicast now handles joysticks, the reicast-joyconfig method commonly found on youtube that everyone is trying, no longer works. The docs need updating, but here is where I found the solution for my similar problem after first installing retropie on my pi3b+:
https://retropie.org.uk/forum/topic/24544/reicast-mapping-controllers-2020-version-stuck
I think that thread should be stickied until the documentation is updated. Note that the bottom line for me was the reicast autoconfig (which uses es joypads) chose the wrong device id#'s in emu.cfg. The reicast-joyconfig did what it was supposed to for my 4 controllers, and created working evdev device files in reicast/mappings/. So all I had to do was change the bottom[input] part so the "players" lined up with the right "device id's". In my case I had keyboard=0, mouse=1, joyou dual arcade=2, joyou dual arcade=3, and xbox one s=4. I initially -1'd the keyboard and mouse, and had 3 players configured, but that didn't work at first (+start reicast wouldn't respond). I went back to emu.cfg and -1'd my player 3 (the second half of the joyou), and reicast responded fine showing both player controls... and games also worked 2 player. I haven't gone back and tested re-enabling player 3, but I wonder if having the dual arcade device may not work out with that.
So bottom line, if you ran the joystick config script, you probably already know which device id #'s need a player # to go with them in emu.config [input] section (the rest get -1'd out). More than likely the proper working evdev files are already in your mappings folder, so after getting your 0 and 1 behind the proper = sign in emu.config you should be good to go.
