@mitu I was running a normal XOrg session before (which is the default on PopOS), but running emulationstation in a Wayland session worked! Unfortunately, Wayland is pretty unstable in PopOS, but maybe it will work just for running retropie.
In case anyone runs into the same issue, you have to enable Wayland as described here and it will work.
Have you tried overclocking your GPU ?
The reason why I'm asking is because I experienced the same issue.... but after 5+ hours of screensaver.
I tried a bunch of stuff... Regenerating and removing most of my gamelists, building Retropie from master, updating just about everything on my Pi, trying various settings in ES....
And then I noticed a video glitch in the ES interface when a video preview started, then all the interface froze except for the preview.
I changed my GPU overclock from 550 to 525 Mhz.... and no more freeze.
I hope it helps