While I tried to make Exact Boolean operate the same as Fast Boolean except when the latter fails due to its limitations, in the area of inside/outside testing there is a bit of a divergence, and I want to solicit opinions on what to do about it.
The difference is that Exact Boolean pays attention to normal directions, while Fast Boolean does not.
If you think of a ray shooting way of testing whether or not a point is inside the volume represented by an object, there are two choices: (1) count the number of intersections of the ray with planes (being careful to only count once if you hit a seam between two planes) and call the origin point “inside” if that count is odd; (2) similar to one, but when accumulating the count, add +1 if crossing in the same direction as the face normal, and -1 if crossing in the opposite direction; then declare the ray origin point as inside if the final count is positive.
The difference between these two shows up if you have, say, a sphere totally inside another sphere, both with their normals all facing outward. And then do a Union. With case (1), the inside of the inner sphere will be called “outside”, and thus that inner sphere’s faces will be part of the union. With (2), the inner sphere’s faces will go away because the inside of the inner sphere will be “inside”, so the union of that volume with the volume between the spheres is just the volume contained in the outer sphere. If you want the effect of a hole with (2), you can reverse the inner sphere’s normals.
In my mind, (2) is more correct. But it is clearly different from Fast.
One reason I use (2) for Exact Boolean is that the paper I followed doesn’t use ray shooting. Instead, it classifies cells in space as to their containment in other cells, using a method that depends on paying attention to the normal direction when you cross a face from one cell to another. I’m not sure I can adapt it to act like case (1), though I admit to not thinking very hard about this yet.
What do people think of this inconsistency? Is it OK to leave it? Should I try to change Exact to be like Fast? Or the reverse? Or have an option? (As I said, if the answer is to make Exact behave like current Fast, even optionally, I’m not sure I can do it without switching to ray-shooting, which can be a lot slower.)
This is coming up now because I’m looking at bugs like T89391 and wondering what to do if the object that the Boolean modifier is on has a negative transform matrix. If I keep the current behavior (actually, fixing it to reverse normals of the operand object(s) if their transform matrix has a different sign than the main object’s transform matrix), then the difference with Fast will be quite evident, since looking at normals, the main object will have normals pointing inward and thus, the volume “inside” it is actually all the space the visually looks like its outside.