Also noticed that I don't have to add lots of games. Simply adding more and more systems to retropie will also chew up ES's RAM until it can't launch anything.
I might try playing around with the memory split to see if this helps at all, and allocating more VRAM to emulation station... but it may not work. Currently I have the memory split at 128.