Normally you'd compile it manually and add it as a separate emulator, but the current version doesn't compile on current RetroPie (Pi3/Pi4). You may wait until the new version of RetroPie (based on the Raspi OS bullseye) is released and the standalone flycast will be installable.
I've started working on a RetroPie module to integrate it, but due to some rendering bugs and the aforementioned incompatibility, I didn't add it to RetroPie proper yet.

Libretro mame master branch has been updated to 0255.
Compiling for arcade and mess sub-targets has been implemented again.

Hopefully everything will work like it did before.

@Treco

A while ago I added the stickfreaks binary install option in my script.
Still in the the secret menu/test menu though.

If you want to try, let me know, I will send you some more info.

I just encountered this same error today on a 2GB Pi4 with this card: https://www.amazon.com/gp/product/B0887GP791

Mine was when hitting the "Update RetroPie-Setup Script".

SAMSUNG: EVO Select 128GB MicroSDXC UHS-I U3 100MB/s Full HD & 4K UHD Memory Card with Adapter (MB-ME128HA)

Fresh installation, and only time this card has been used.

Nothing specific yet, but there are some prospects.

I got the non-libretro version working for pi zero (Retroflag Gpi case) and wrote up a guide in case it helps anyone interested in Cannonball on Pi zero: https://sinisterspatula.github.io/SuperRetropieGuides/cannonball and also, if it helps retropie dev's in adding a pi zero specific version, my modified repository that worked for zero is located here: https://github.com/SinisterSpatula/cannonball

Sorry, I thought I did. But that was in another post.

That said, compiling on a threadripper 2990WX with -J80 is almost hilarious. :-)

@iMacosiris Please don't necro-bump old topics - if you have a similar issue, open a new topic and mention this one, while adding the infromation asked in https://retropie.org.uk/forum/topic/3/read-this-first.

Thanks. I will check tonight. I think , I will just start from the beginning with Ubuntu anyway if that is more stable

Modify the atari800 package script and replace the URL to download with your own URL to the new version. Make sure that it's not a 'redirect' or 'wait x seconds to download` and it's a direct link. After you replace the URL, use RetroPie-Setup to install from source the emulator.

@meleu I use emulated arm chroot with qemu on x86_64 pc with ubuntu, but also cross compiler on the host via distcc. (I don't have time to do any write up but there are some on the net).

@NowArrivingHere bad sdcard perhaps. Make sure you are not over clocked etc too.

Advmame includes advmess also and has binaries

@Rydra I would, so long as nothing else gets broken :)