I had a bit of the same issue.
If i understand correctly, then when you press the "escape" key on keyboard the menu pops up.
What i did, i assigned in the controller setup a unused key, like the right thumbs key for example. I used the LR3 button to let it assign. In my case it made the "LR3 \XAC" key.
I saved the setting to global.
edited the file " /opt/retropie/configs/psx/pcsx/pcsc.cfg"
Changed the line "bind \xAC = LR3" to "bind \xAC = Enter Menu"
Saved file restart game. Then pressing the Right thumb opened the menu.