Looks like I accidentally fixed it.
Yesterday I bound all the hotkeys to the same button, since that seemed to be the only way to "free up" the other buttons for normal use. Today I went back through the retroarch hotkey binding menu and re-bound the hotkeys to the proper buttons. Now they work! Button presses register as normal button presses during gameplay, and register as hotkey presses when the hotkey enable butten is also pressed. I tested on a couple different systems to be sure, and it seems that all is now OK.
I have to assume that something was wonky in one of the cfg files, and even though it was showing hotkey bound in the menu, it wasn't actually bound and that caused the problem like you said. I guess the act of manually re-binding them fixed the issue.
Thanks for the help.