The quickest way I have found to setup 5200 controls in RetroPi is

open a 5200 game then enter the retro arch quick menu.

From there you want to go down to options. Then set the Atari System to 5200. After that go back to the quick menu.

Now select controls.

Go down to Port 1 Controls and click it.

Change Device Type from RetroPad to Atari Joystick.

Back out to the controls menu.

you may want to set the Device Type to Atari Joystick for
Port 2 controls if you're playing 2 player games.

From the Controls menu Save Core Remap File.

Go back to the Quick Menu.

Back out to the main menu.

Open Configuration File and Save Current Configuration.

Back out to the Main Menu and Quit RetroArch.

Now your controller should work with 5200 games