Solved it.
I opened the config TAB menu of the MAME emulator.
The mapping was messed up for some reason.
Here is how it looked like for the first 4 buttons of Player 1
P1 button 1 = LCTRL or RetroPad1 B (This one is correct)
P1 button 2 = LALT or RetroPad1 Y (This one is also fine)
P1 button 3 = SPACE or RetroPad1 X (This one is supposed to be mapped to LSHIFT)
P1 button 4 = LSHIFT or RetroPad1 A (This one is supposed to be mapped to SPACE)
Buttons 3 and 4 are rightfully mapped as RetroPad X and A, but for some reason, the first values (SPACE and LSHIFT) were switched.
What I did is to edit the value by hand with the keyboard.
If you compare the mapping for the X and A buttons with the MAME retroarch.cfg you will notice the difference.
input_player1_a = "space"
input_player1_x = "shift"
Have no clue why this happened, but it works now. :)