I am blogging about this exact topic, and have experimented a lot with the Pi 4B and different coolers as well as clock speeds.
With the latest firmware and stock clocks, you can expect to run about 20 minutes of high load before thermal throttling kicks in, with a decent passive heatsink for less than a buck.
However, active cooling is absolutely necessary if you want to even slightly overclock your Pi 4B, or prevent thermal throttling when running for longer periods of time, or both.
I personally recommend this one: https://www.ebay.co.uk/itm/133351554298?ViewItem=&item=133351554298
For me, it was a little more expensive ( around 15€) since I live outside of the United Kingdom, but it was absolutely worth it.
It is hands down the best price/performance case I have seen so far, and I have personally tested 6 cases and more than 20 cooling solutions so far.
With this, I managed an overclock from 1500/500 (CPU/GPU) to 2100/750 at 56°C max temp after 1 hour of stressberry 4 core testing with the cooler at 5V, it goes up to 65°C when the fan is at 3.3V, so even with a 40%/50% overclock and the fan slowed down for silence, it is still far from throttling.
You can further decrease temperatures by using a bigger (up to 7mm high, up to 40x40 broad and long) heatsink, or using copper instead of aluminium, but as you can see from these temps, it is not necessary.
