You probably were able to figure this one out since it has been quite some time, though just wanted to reply for anyone stumbling to this; the mixed up colors seem to be an artifact of the SPI bus bandwidth limitations of the ILI9486 controller chip on the display. If the display is driven too fast that the controller simply cannot handle it, the colors on the whole display can get distorted. There is no way around it.
My ILI9486 has maximum speed of 31.88MHz (31880000), and beyond that the colors get messed up. I posted a comment on the thread you linked to with more information about the observed limits of ILI9486 with a video. With fbcp-ili9341 project I was able to squeeze out some more performance from the ILI9486 compared to the dtoverlay method, but beyond simple retro games, it looks like ILI9486 is definitely not a display controller for fast paced games.
@lyle_jp then you need to make this argument on the github tracker for prosystem and convince them - arguing it here will do nothing as we don't develop the emulator. I'm not convinced but perhaps it could be a core option to switch for those that prefer the saturated palette.
Just an update, I fixed the issue and now it's running beautifully. It turns out I was following the directions correctly, but I had no idea I had to remove the #. If I hadn't asked, I wouldn't have known so thank you guys for all your help!