@mitu thank you, excellent advice
After running the above I noticed that indeed those axis don't work. So I tried changing the position of the connector (this one towards the joystick knob didn't have wiring guidance) and tadaa, they work now. Also this fixed somehow the skipping part, now with a long press of a button I can properly skip it.
Now I'm just trying to figure what is the minimum amount of buttons that I need (directions, start, select, hotkey and ...?) , I don't want the arcade machine to have too much buttons lying around plus I don't have the space, I want to build a double control one this time. Just hope there won't be no more trouble ahead.
Anyway, to get back to the subject, for other folks, if the joystick misbehaves, don't forget to check the wiring also. In my case this inverse polarity allowed it to work partially.