Here is my setup:
And the result:
It works if I set shift + double click, but then I need to deselect if want to use it without adding to existing selection.
Wait, sorry! You setup working well!
It should be great to have it as default shortcuts.