You can check using the jstest utility if the buttons are registering correctly in Linux, outside or RetroPie. Attach a keyboard and exit Emulationstation (F4), then run from the command line:

It will show you the gamepad buttons/joysticks and you can press them one by one to see if the input values change on the screen. You can exit the test by pressing Ctrl + C.

If there is no reaction when pressing a button, either the wiring is faulty or the controller.
You can also try connecting the encoder to a PC and test it there also.