@jonnykesh Thanks for your answers and tips!
I re-download the Retropie image and ApplePi-Baker, formatted the SD card and installed the image all over again. The result was the same, the controllers would not respond.
I then got my hands on another Raspberry Pi Zero W, inserted the SD card with the same image, booted up and everything works perfectly now. Both controllers and the keyboard.
So I guess you were right, that it was an issue with the Pi.
Thanks! :)