what is the resolution are you running it in.
i ask this due to when i changed mine to a lower res. when i tried to get to the menu to exit i would just get a black screen.
as for your button problem i can not say. have you just to map it to a working key just to see if you can get it to work in general?
im using a 8bitdo nes30 pro and my mapping works like a charm.
i did come across something odd.
when i would start the game once my controller would be JS0, and if i would exit and then start the game again it would be JS1. so i would have to map the key twice. once as JS0 and then JS1. dont know why but it works so i don't care. don't know if you are getting something like that