This is propably suggestion out of your space. Im mostly Modo user and I`m impressed by hard work of Blender dev team, HQ and whole community.
In Modo by default you have possibility to select all polygons/edges/vertices on current mesh by double click on it (in Modo its LMB but in your case it would be RMB on Edit Mode). Again one click is deselect all polygons/vertices/edges and select one polygon if you point again on mesh or stay deselected if you pointed on empty space. I think it`s quite intuive (IMHO).
My wired mind tells me one more solution which may be less intuive. Triple clicking. It would select all polygons/edges/vertices except current selection 
I think that I would see creepy smiles on some faces after that second suggestion but it would be quite fast selecting solution for mouse users.