I poked around a bit more on this and noticed this behavior persists also in Mednafen running in Windows. I opened a ticket with the Mednafen as a bug since I believe this core is base off that. I tested out the old Handy emulator and did not see the issue there.