@Bluescreen2001 this can be done. I did it but it was many months ago now so I am somewhat rusty.
First to back into the way you normally configure a controller in retropie (like when you boot up the first time) and reconfigure your controller to how you want it to operate in-game.
Second, go to the command line, and use the js_test utility to figure out what button numbers your raspberry thinks your buttons are: type into the command line jstest /dev/input/js0
Then press your buttons and it should tell you which button number is assigned because when you it the button the appropriate number will say “on.” Write down a reference for yourself.
Third, exit the jstest by pressing CTRL + C on your keyboard. Now you are going to edit the emulation station input config file to make new controls for navigating the emulation station menus. At the command line type sudo nano ~/.emulationstation/es_input.cfg that will allow you to change which buttons do what in the emulation station menus by using the button numbers you found and recorded in jstest. You will make button “a” the number button that you found was your start key, you will make button “b” the number button you found was your “stop” key and you will make the “start” button (which usually brings up the menu) the number button you found was your “coin” key. For the sake of still having use of the other buttons In concept you may want to then assign the “select” key to some other button since it will likely be assigned by default to your “coin” button. After you make your changes save the file and go back to emulation station by typing emulationstation in the command line and see if it works.