Why not hiding the gizmo (or keep it visible but deactivate it) while using a modifier key, so when we press shift or alt or both to make a continuous selection, it deactivates the gizmo and release the modifier key re-activate it ? what you think ?
“It should be possible to click over a Gizmo to select items underneath it, as long as you don’t drag.”
https://developer.blender.org/T57918