For now my conclusion is this:
Progressive scan with RP4b still works. You just have to either set it in config.txt as the appropriate sdtv_mode (16 NTSC/18 PAL), but then Emulationstation will also be in 240, which looks terrible, or use tvservice to enforce it on a per rom/platform basis after retroarch starts to switch back and forth.
You can use my watcher python script and run it on startup in the background and then make an appropriate runcommand-onstart script or make your own solution.
Script: https://retropie.org.uk/forum/post/220137
pi@retropie:~ $ tail -2 /etc/rc.local
su pi -c 'python3 -u /opt/retropie/configs/all/vmodes_watcher.py &> /var/log/vmodes_watcher.log' &
exit 0
# taken from https://github.com/Sakitoshi/retropie-crt-tvout/tree/master/to_configs
pi@retropie:~ $ cat /opt/retropie/configs/all/runcommand-onstart.sh
if [ -f "/opt/retropie/configs/$1/480i.txt" ]; then interlaced=$(tr -d "\r" < "/opt/retropie/configs/$1/480i.txt" | sed -e 's/[/\\[/'); fi > /dev/null
if [ -f "/opt/retropie/configs/ports/$1/480i.txt" ]; then interlaced=$(tr -d "\r" < "/opt/retropie/configs/ports/$1/480i.txt" | sed -e 's/[/\\[/'); fi > /dev/null
if [ ! -s "/opt/retropie/configs/$1/480i.txt" ] && [ ! -s "/opt/retropie/configs/ports/$1/480i.txt" ] || [ -z "$interlaced" ]; then interlaced="empty"; fi > /dev/null
if [ -f "/opt/retropie/configs/$1/240p.txt" ]; then progresive=$(tr -d "\r" < "/opt/retropie/configs/$1/240p.txt" | sed -e 's/[/\\[/'); fi > /dev/null
if [ -f "/opt/retropie/configs/ports/$1/240p.txt" ]; then progresive=$(tr -d "\r" < "/opt/retropie/configs/ports/$1/240p.txt" | sed -e 's/[/\\[/'); fi > /dev/null
if [ ! -s "/opt/retropie/configs/$1/240p.txt" ] && [ ! -s "/opt/retropie/configs/ports/$1/240p.txt" ] || [ -z "$progresive" ]; then progresive="empty"; fi > /dev/null
if tvservice -s | grep NTSC && { ! echo "$3" | grep -wi "$interlaced" || echo "$interlaced" | grep empty; } && ! echo "$interlaced" | grep -xi "all" && { echo "$3" | grep -wi "$progresive" || echo "$progresive" | grep empty; }; then echo 'NTSC 4:3 P' > /opt/retropie/configs/all/desired_mode/value; else echo 'NTSC 4:3 ' > /opt/retropie/configs/all/desired_mode/value; fi
echo "runcommand-onstart $1 $2 $3"
pi@retropie:~ $ cat /opt/retropie/configs/all/runcommand-onend.sh
if tvservice -s | grep NTSC; then tvservice -c "NTSC 4:3"; fbset -depth 8; fbset -depth 32; fi > /dev/null
240p is default, override per platform/ROM in 480i.txt files
pi@retropie:~ $ cat /opt/retropie/configs/mame-libretro/480i.txt
all
As stated in this thread: https://github.com/raspberrypi/firmware/issues/683#issuecomment-283179792 what was done in the firmware to make these new modes work is
A. Setting the progressive scan bit in the VEC (Composite TV out) register set:
https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/vc4/vc4_vec.c#L104
B. Remove the interlace flag for the mode:
https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/vc4/vc4_vec.c#L256
tvservice does essentially the same thing and sets the new modes:
https://github.com/raspberrypi/userland/blob/2448644657e5fbfd82299416d218396ee1115ece/interface/vmcs_host/vc_sdtv.h#L60
https://github.com/raspberrypi/userland/blob/master/host_applications/linux/apps/tvservice/tvservice.c#L703
https://github.com/raspberrypi/userland/blob/master/interface/vmcs_host/vc_vchi_tvservice.c#L1213
https://github.com/raspberrypi/userland/blob/master/interface/vmcs_host/vc_vchi_tvservice.c#L698
Since there is a hardware flag that needs to be set, it won't be possible to resolve this with xrandr modesetting alone as was suggested.
Ideally RetroPie in some form could enforce this natively in a similar fashion by taking the SDTV_MODE as a parameter and enforcing it through VCHI like tvservice does, alongside the modsetting being done in the modified SDL library.
