@mitu I was running a normal XOrg session before (which is the default on PopOS), but running emulationstation in a Wayland session worked! Unfortunately, Wayland is pretty unstable in PopOS, but maybe it will work just for running retropie.
In case anyone runs into the same issue, you have to enable Wayland as described here and it will work.