Major framerate difference in 3d viewport caused by Auto Depth setting

The 3d viewport is very sluggish and if I enable Cavity is becomes a pain to interact with.

When I enable Preferences > Navigation > Orbin & Pan > Auto Depth there is a massive improvement, even with Cavity enabled, but I dislike the way that form of interaction works.

How do I get the same speedup without having to turn on Auto Depth?