So most likely the USB hub is disconnecting at some point and EmulationStation crashes during that moment. Can you try and get a separately powered USB HUB ? Or perhaps a newer PC, since the one you have is more than 10 years old ?
Hard to say without any logs to help diagnose what makes EmulationStation crash. Did you have to install any driver to make the controllers work ? I don't think they've needed any drivers on Linux at least.
@mitu Thank you so much for your help. I really appreciate it. I'm sorry ended up with some custom BS image, but at least I'll learn more by starting from scratch.