Tate mode doesn't auto-rotate. TATE mode adjusts the screen to fill your monitor. What you need to do is is specify the exact rotation you want for each game you need to and make a game-specific config file for it. If you have auto-rotation on and auto-rotate on, you will get some weird results. Auto-rotation should be turned off when using TATE mode (setting allow_video_rotate = false). Some games will need 180 degree rotation. You'll have to decide which end of your monitor you will consider your base (the one that will be the bottom edge when you spin it).
For each game you want to have in cocktail mode, you will likely need a cfg file with the setting
video_rotation = 2
for games that need to be flipped. Set it to 0 (or don't write a cfg) for games that sit the right way from the get go.
This post was greatly helpful for me when I was doing my cocktail setup: https://retropie.org.uk/forum/post/39024
