Tried it. Didn't work. Just too unstable and some of the keys just didn't match up.
Next solution is to dismantle the old Atari joystick and replace the insides with joystick and controller. Need something quite small though, as the existing arcade ones seem to be a bit too big.
So, the plan is. Dismantle the joystick, have the standard single button and directional stick in the joystick, but on the controller board I would add some extra "buttons" connected to the Reset, etc. switches on the Atari machine to map with Start and Select and Back.
First, when configuring the joystick through emulationstation which set of configuration options should I use to configure the joystick? The first set of directional inputs, or those farther down in the configuration menu?
You can use both to navigate EmulationStation, but most games expect something similar to a D-Pad (the first), so I'd use that.
I am currently using the first set of directional inputs but the joystick is very sensitive. If using the other directional inputs don't make a difference is there a way to adjust the sensitivity of the joystick?
You can try to increase the deadzone for the joystick using evdev-joystick , a command line utility.
I found out that the problem is caused by shader I applied to all liberto emulators,
I guess Atari800 emulator doesn't like shaders.
Disabled it for this specific emu and it's working now!
Thank you mitu for answer.