So i looked into my retroarch.cfg files and they are all messy as hell now, so i figured i better follow @Concat 's advice and edit my .cfg files manually, which is really no big deal as soon as you have all your settings figured out through RGUI, so here is the updated guide:
Overlay Settings
Open any Rom within the Emulator you want to set up -> Open RGUI (Hotkey SELECT+X) -> Settings -> Onscreen Overlay
Hide Overlay In Menu -> OFF
Overlay Preset -> Browse to and select your overlay.cfg (I put mine in /home/pi/overlays with WinSCP)
(if you can't get there directly set: Settings -> Directory -> Overlay Dir -> Choose folder with overlays)
Settings -> Video
Aspect Ratio Index -> Custom
Integer Scale -> ON (with this setting the gameimage will be centered, so you can't move it anymore with "Custom Viewport X/Y, but it helps shaders and to keep the original aspect ratio)
Set "Custom Viewport Width/Height"
Settings -> Onscreen Overlay
Set "Overlay Scale" if necessary
Overlay Opacity -> 1.00
Write down all the values you just set
Quit Retroarch without saving (default as long as you didn't change it)
Retropie Menu -> Configuration Editor -> Advanced Configuration -> Configure Libretro options -> Choose emulator you want to set up
Now you can set all the values that you just wrote down.
The only thing i didn't figure out how to set in this menu was the overlay.cfg file.
So for this you will actually have to go into the retroarch.cfg (eg. /opt/retropie/configs/gb/retroarch.cfg) for each emulator and copy the path into it.
Of course you can also go directly into the .cfg and write down the settings manually.
At the end the retroarch.cfg will look like this (here are also shader settings included):
# Settings made here will only override settings in the global retroarch.cfg if placed above the #include line
input_remapping_directory = "/opt/retropie/configs/gb/"
aspect_ratio_index = "22"
input_overlay_enable = "true"
input_overlay = "/home/pi/overlays/gb.cfg"
video_scale_integer = "true"
custom_viewport_width = "640"
custom_viewport_height = "576"
input_overlay_opacity = "1"
input_overlay_scale = "1.04"
video_shader_enable = "true"
video_shader = "/opt/retropie/emulators/retroarch/shader/gameboy-screen-grid.glslp"
#include "/opt/retropie/configs/all/retroarch.cfg"
In case somebody wants to make their own overlays you have to make shure that the part of the picture where the game is going to be is exactly in the middle of the picture and is transparent.
To use it with RGUI you have to make a .cfg file (overlayname.cfg) which looks like this:
overlays = 1
overlay0_overlay = overlayname.png
overlay0_full_screen = true
overlay0_descs = 0
just put it together with the overlayname.png file
Here are my custom overlays for 1280X1024:
https://mega.nz/#!jRMmgThD!5upxCfwq4EKL18FcDa3Irhk5W4AJ3Du-YNkorFln940
This only contains GB/GBC/GBA overlays as with this screen ration these are the only ones that make sense for me, except maybe (vertical) arcade games which i might add later, if i can be bothered to make them :D