Sounds like you've hit the tab key. Your right shoulder on your ps3 controller will be assigned to the tab key of the keyboard. If you hit tab key on your keyboard you will get the same menu. This is the standard menu in mame and allows you to change settings for mame or game specific settings such as difficulty like you have seen.