Okay, there is some problem with the onboard bluetooth. The documentation already mentions that the case interferes with 2.4 GHz wifi. The same is true for bluetooth, and I added a USB bluetooth adapter to mitigate that.
Just for context, here is a comparison of uncached read performance:
SSD
$ sudo hdparm -tT --direct /dev/sda
/dev/sda:
Timing O_DIRECT cached reads: 520 MB in 2.00 seconds = 259.69 MB/sec
Timing O_DIRECT disk reads: 902 MB in 3.00 seconds = 300.36 MB/sec
SD
$ sudo hdparm -tT --direct /dev/mmcblk0
/dev/mmcblk0:
Timing O_DIRECT cached reads: 84 MB in 2.04 seconds = 41.13 MB/sec
HDIO_DRIVE_CMD(identify) failed: Invalid argument
Timing O_DIRECT disk reads: 124 MB in 3.02 seconds = 41.11 MB/sec
Definitely worth the effort to me. Instructions for making the RPi4 boot from USB can be found here, btw.