Fundamental flaws in Blender's constraint philosophy?

[vasiln]

I posted this on Blender Artists a couple of days ago and was encouraged by a few to repost it here. I have criticisms of the underlying philosophy of how Blender’s constraints are handled, but I’m not sure the best way to deliver those criticisms. If I’ve overstepped any bounds, I apologize. The following is a straight repost (with two new footnotes):

It seems to me that the basic philosophy and architecture of Blender’s constraints is flawed. This leads to counterintuitive behavior that confounds “beginners” (really, anybody without a relatively advanced understanding of 3D math) and creates unnecessary limits even on people who do have an understanding of this math. The problems that are created are often pernicious, easily missed until you inspect your interpolations carefully.

Let’s start by looking at how Blender currently handles constraints. I haven’t inspected the code-- this understanding is gained only by seeing what Blender does. I could easily be making some mistakes.

We start with f-curve values, modify/replace by drivers, then turn those into a 4x4 transformation matrix. When a constraint is evaluated, that transformation matrix is converted first into a different space as necessary (world vs bone local vs etc) and is then converted into Euler rotation, translation, and scale triplets. Blender then does some simple addition on these components (or recently, in the case of some scaling operations, multiplication) then converts the updated rotation/translation/scale triplets into a new matrix. Then we look at the next constraint and repeat.

There are two main problems with handling things in this fashion. The first is aliasing – the fact that for any particular orientation, there exist multiple Euler rotation triplets. This isn’t just considering values outside the -180,180 range, and it’s not limited by the angle evaluation order. What that means is that when we convert Euler rotations to matrices and then back to Euler rotations, we don’t get the same values that we put in. In many cases, the individual values of the Euler rotation components are not at all what we expect, even though the complete orientations, composed of the entire triplet, are exactly the same. (And, particularly, when we scale these individual components by some influence multiplier, and take into account funky Euler angle interpolation, we don’t get anything remotely like what we wanted or expected.)

The second problem is that Blender is just replacing the matrix. In some cases, many cases, that’s fine. Where it breaks down is when we have skew: an object or bone with local rotation that has inherited non-uniform scale. Skewed transformation matrices do not have distinct scale vs. rotation components. When we create new matrices out of decomposed scale + rotation, parts of the original transformation are inappropriately discarded.

How should Blender be doing this? It should never be decomposing matrices into components. Constraints should not create numbers that are added to components. Instead constraints should be creating transformations that are matrix multiplied into existing transformation matrices (1).

Rather than addressing this fundamental problem with how constraints are handled, Blender versions have tried to apply band-aid fixes to individual issues on a case-by-case basis. The naive initial approach is the reason scale needed to have a multiplicative version added, it’s the reason that copy rotation offset mode needed to be redone-- none of that would have been an issue if Blender were applying constraints as matrix multiplications (2). It’s responsible for continuing constraint issues, some of which would qualify as bugs, some of which would qualify as feature requests, and most of which would fall into the wide, vague area between the two.

It’s entirely possible I’ve made some mistakes in thinking about this. I’m far from an expert myself. Any kind of response is welcome. Honestly, when it comes to constraints, I’ve just been holding my breath until we get node-based rigging, figuring that given sufficient tools, I can fix these problems myself. At the same time, I keep seeing more work done on these constraints, but the fixes never seem to be made with an understanding of the underlying problem, and so I’ve been increasingly concerned that developers don’t recognize the real issue.

I could certainly go into detail on how these problems impact nearly any (bone) constraint, and offer suggested changes to turn these constraints into matrix transformation operations. I think that in many cases, it’s pretty obvious to anybody conversant enough with 3D math, and if I tried to describe all of the impacts and fixes, it might end up being a short book.

(1) You just don’t try to create orientations by adding up Euler angles. Euler angles are not 3 independent values. Component math does not give the kinds of orientations that people expect. Imagine trying to handle parenting relationships like this instead of via matrix multiplications!

(2) A really good example of a band-aid fix that I originally missed is the recent addition of new ways to measure angles in some constraints (like the “order” field in 2.83’s copy rotation constraint.)

[Hadriscus]

Probably want to ping @angavrilov

[brecht]

Each constraint has as input and output a matrix, and in some cases what constraints do boils down to a matrix multiplication.

But a constraint like Copy Scale needs to decompose scale from the matrix, to replace one scale with another. How else would you do it? There is no need to convert to Euler angles in that constraint of course, components other than scale can remain in matrix form and that’s also how it is implemented.

In Copy Rotation there is a conversion to Euler, but this is done to support the functionality where you can copy a subset of the rotation axes. If you don’t decompose, how else can you manipulate those individual values?

Of course Euler has problems with aliasing and interpolation, and there’s a lot of code in Blender to counteract that. We have code to find closest matching Euler rotation to the input, quaternion slerp, etc. But Euler conversions still happen in places because users want to be able to edit at the level of individual XYZ values.

So I think you really do need to look at each individual constraint to see how the implementation might be improved. I’m sure it’s possible, and maybe there are general principle that can be followed. But I don’t see how never decomposing and only multiplying matrices is practical advice.

[vasiln]

@brecht

How else would you do it?

The scale of an object is the length of its basis vectors. Ie, run (1,0,0) through the bone’s transform and find the length of that vector. Compare scale of source and target and apply interpolation for influence here. That allows you to create a scale matrix that you can multiply into the 4x4.

What about scaling world->world, which may involve scaling unaligned with the basis vectors? First, multiply a rotation matrix to bring one of your basis vectors into alignment with your scale. Figure out your scale from the length of the basis vectors, consider interpolation, and multiply that in. Multiply in the inverse of your rotation matrix. You now have a matrix you can multiply into your starting 4x4.

But I want to make it clear, I’m working off the top of my head here, and I know just enough about matrix operations to be dangerous I’m the kind of person that has to check by hand, every time, to figure out if I should be rotating about my cross product or my negative cross product…

@brecht

In Copy Rotation there is a conversion to Euler, but this is done to support the functionality where you can copy a subset of the rotation axes. If you don’t decompose, how else can you manipulate those individual values?

A copy rotation (and limit rotation) is exactly where this is the biggest problem, and it requires rethinking the nature of the constraint. Part of the problem is that Eulers seem intuitive, but they really aren’t. A proper reworking of copy/limit rotation would not involve a decomposition into Eulers.

Now, of course, a simple 3 axis copy rotation, no skew, full influence, is trivial. However, at less than full influence, you see interpolation problems, and the proper way to handle that is to do a quaternion rotation, via slerp, from the starting orientation to the final orientation.

For looking at 2 axis copy rotation, you need to step back and say, “What do users want when they want to copy 2 axis rotation?” If we think about a bone, pointing in its Y axis, what they want is just to move the bone’s tail, but of course that leaves the bone’s roll undefined. (Because in reality, there’s no such thing as simultaneous 2 axis rotation in 3D.) I believe the best way to handle that is to treat the transformation as an axis-angle rotation from the original tail position to the new tail position. (In the cross product of the two.) And of course, you’d quaternion slerp for interpolation. I think anyone sliding through the interpolation on this kind of 2 axis rotation would have to agree that it was not copying any Y axis rotation.

The XZ bone rotation is the most intuitive way to understand 2 axis copy rotation, but of course it can be done with any 2 axes you’d like. The orientation of a bone tail is just a convention. Note that copy rotation in this manner leaves an undefined pole at 180 degrees, but that’s pretty common with constraints, and you’re going to see big problems with the current implementation as you approach 180 degree rotations anyways. In any case, I can’t see a way to avoid the pole.

Okay, so that leaves 1 axis copy rotation. To do this, let’s imagine that we first did 2 axis copy rotation, and wonder, how would our bone’s roll differ from the roll of the bone from which we were copying rotation? So let’s do a 2 axis copy rotation, then do an axis-angle rotation in the bone’s Y axis so that its xz halfvector is pointing in the same direction as the target’s xz halfvector, and then multiply in the inverse of our 2 axis copy rotation. (Again, Y is intuitive, but it doesn’t matter; again, quaternion slerp for interpolation from influence.)

Since we’re just ending up with matrices that we multiply in, none of this is going to cause any problems for copy rotation to skewed bones. But what about copying rotation from skewed bones? Their axes are no longer orthogonal. I believe that it’s best to have the user offer a preferred axis about which to rebuild the target’s basis vectors. (Like if you rebuild around Y, then X and Z are projected into the Y plane, then rotated toward/away from their half vector sufficiently to make them orthogonal with each other.) It’s not necessary to offer this-- you could rebuild around the average of all three basis vectors-- but I think users are going to prefer being able to think about things in this way. (When you start getting into tracking, or camera transforms, things with very important axes, that’s when the option becomes important.)

@brecht

But I don’t see how never decomposing and only multiplying matrices is practical advice.

I guess I don’t want to say “never ever”. If I said that, I probably got carried away Some constraints seem to work well enough-- child-of, copy transforms haven’t given me problems related to this, and I suspect they’re doing things differently under the hood. (Probably using matrix multiplication.)

And, obviously, if you did revise this, you’d probably want to leave legacy support.

There may be exceptions. A transformation constraint is probably an exception, at least, in its input. (It could benefit more from the “order” options for the copy rotation constraint than the copy rotation constraint can benefit, actually.) But I do think, that as a general principle, you don’t add Euler components. They just don’t work that way.

Edit: Another exception would probably be translation. As far as I can tell, you really can’t go wrong with translation, so you can get away with decomposing and recomposing it if you want.

[Hadriscus]

I think copying a single axis rotation from a target is just meaningless anyway, since euler angles are just a representation, and as you say a number of combinations can lead to a single final rotation matrix. I don’t know if the constraint influence is simply a matter of lerping the values though

[brecht]

For scale, what you describe sounds mathematically equivalent to decomposing, modifying scale and recomposing.

To me it sounds perhaps like you are assuming that decomposition can’t properly preserve shear, and that only multiplying matrices is the solution to that? There is a lot of code in Blender that was not originally designed with shear in mind, and only some of it has been fixed over time, but that’s not a fundamental limitation of decompositions.

For rotation, I think it would be useful to support swing-twist decomposition in more places. It’s a more natural fit for organic joints, with the singularity at a location that you can’t realistically rotate the joint to. I chose it for IK rotation limits for that reason, and Alexander added support for it in drivers recently.

However constraints are also used for mechanical joints, where Euler angles can be the more natural choice. Rigging is not just for characters. Axis-angle may be good too for some cases, I’m not very familiar with the use cases for those. Also something to take into account is the preference of animators, the rig and constraints in places needs to match the representation chosen for that.

[vasiln]

@brecht

For scale, what you describe sounds mathematically equivalent to decomposing, modifying scale and recomposing.

Yes. I believe it is. The difference, with regards to this particular constraint, is that if matrix transformations were used to offset copied scale, the old “scale is added” bug/quirk never would have been an issue. The only reason that seemed like a good idea at the time was that other constraints were doing it, and implementing as a matrix transform would necessarily mean that the scale was multiplied.

Maybe I misunderstood what you’re asking. Are you asking, how would my suggestions fix this?

It would be nice if that could be fixable, but my suggestion wouldn’t do the trick. (The only ways I can imagine to fix that problem strike me as bad ideas, and I wouldn’t begin to know how to interpolate them.)

@brecht

To me it sounds perhaps like you are assuming that decomposition can’t properly preserve shear, and that only multiplying matrices is the solution to that?

Multiplying matrices is a solution to that. Here’s a problem fixable by actually transforming your initial orientation, rather than just overwriting it:

It’s also a solution to other problems. Problems can be dealt with individually, with their own individual fixes (bandaids that I’ve mentioned), but treating constraints in this manner as a general principle would solve a lot of problems, all in one swoop. (That’s not to say that it would be an easier fix. Changing philosophies is usually harder than the individual fixes, sometimes, harder than the sum of all fixes. But being aware of how the philosophy impacts the way that constraints act should help with future decisions.)

@brecht

However constraints are also used for mechanical joints, where Euler angles can be the more natural choice.

It doesn’t work for mechanical rigging either.

That XZY bone is at its rotation limits. (It was interesting to see that changing the bone from Q to euler changed how the rotation limits were evaluated, not sure why that is. Must be the under-the-hood aliasing compensation you were talking about or something. Euler order didn’t appear to change anything though.)

Is that the “right” way to make that structure? It’s the way users are going to try. I think an IK with limits ought to handle the situation properly, as the workaround.

Mechanical rigging is going to work, with any system of constraints, when everything only ever rotates in single local axes. Axis-angle is fine, but so is an Euler in the right axis, so is quaternion (except the quaternion won’t ever give flat angular velocity.) That’s going to be true regardless of running constraints with Euler evaluations or with swing angles, if I’m understanding that term correctly. Even if you need exactly 180 degrees, it shouldn’t be impossible to create a “preferred swing”; for a one-axis bone with even two keyframes for rotations <180, this could be determined automatically. When mechanical rigging tries to rotate in multiple axes, eulers->matrix->eulers isn’t going to work either because of aliasing. (And again, this aliasing is also strongest as the bone approaches 180, although it’s a problem long before then.) While imaginable aliasing compensation might help for evaluation in local space, I have a hard time imagining how it would ever make any sense for evaluations in any other spaces, or how it would work in the presence of stacked constraints.

[brecht]

@vasiln

Maybe I misunderstood what you’re asking. Are you asking, how would my suggestions fix this?

I’m not really asking a question, just trying to make the point that decompositions can work fine and are needed in places. If there is a straightforward way to do things without decomposing that’s of course preferable. But it’s not helpful to restate the problem in some roundabout way in cases where decomposition is a straightforward and correct solution.

@vasiln

Mechanical rigging is going to work, with any system of constraints, when everything only ever rotates in single local axes. Axis-angle is fine, but so is an Euler in the right axis, so is quaternion (except the quaternion won’t ever give flat angular velocity.)

Right, you often rotate over one or two axes, and then you might as well use Euler angles. That was my point, something like swing-twist or quaternion for that case can get in the way more than it helps.

[vasiln]

@brecht

decompositions can work fine and are needed in places

If solutions are equivalent for copy scale, then decomposition works fine, but is not needed.

I suspect that even a carefully reworked transformation constraint wouldn’t be relying on decomposed Euler rotation channels for either input or output.

I do understand that when things are mathematically equivalent, you do whichever gives the best performance. Well, after getting the kinks worked out, at least. I consider the following constraints to all have kinks that seem related (to me) to being calculated with component->matrix->component->math->matrix:

Copy rotation, limit rotation, transformation: lose inherited skew. Do not give intuitive results, likely to result in interpolation problems due to aliasing of Euler components.

Limit scale, maintain volume: does not work properly in presence of skew.

Locked track, damped track, track to: lose inherited skew.

@brecht

you often rotate over one or two axes, and then you might as well use Euler angles.

The euler angles, with 2 angles, are okay for the pre-constraint transformation. Once they get turned into matrices and back into components, the 2 angles no longer represent the rotations expected:

Copying Z rotation (XYZ order) to an XYZ euler, from an XYZ euler bone that has been rotated first in Y, then in X. Rest pose is shown at right.

I believe that it is possible to create copy/limit rotation constraints that behave intuitively and interpolate well, but that ultimately, they’re not going to even use Euler angles. They’d use other more intuitive measures, like angle as projected onto planes, or (for limits) intersections of parametric primitives with the surface of a sphere or circle, or even equirectangular images.

I want to thank you for taking the time to listen and respond. I know it’s probably hard sometimes to explain things to people who know just enough to get themselves into trouble To the extent that you have been considering these kinds of issues, that’s great to hear; to the extent that you haven’t, I’m confident that you’ll be considering them in the future. Blender is of course the most amazing application ever made, the pace of improvements has been ridiculous, and from what I see, that pace doesn’t seem to be slowing any I can afford to be patient waiting for the solutions to some of these problems.

[vasiln]

Here’s a prototype of how a 2 axis copy rotation constraint, designed for mechanical (Euler) use, should work (IMO):

https://pasteall.org/blend/743c2395a62a48179ac1492493c71e62

It’s a side by side comparison of Blender’s existing copy 2 axis constraint with my prototype, which relies on 2-axis sequential rotation (hence, rotation that is about multiplying matrices). There are a pair of keyframes that demonstrate the problem with Blender’s existing implementation, which my prototype fixes. It has some drivers in order to demonstrate interpolation of influence via the position of a single bone.

Additional notes, which nobody has to read if they don’t want:

Since I’m not a mechanical rigger myself, I began by finding the most knowledgeable Blender mechanical rigger I knew and asking them how they felt that a 2-axis copy rotation constraint should work. That didn’t end up being helpful, because they told me they don’t use constraints other than IK. (They use IK and drivers, which doesn’t surprise me, because they avoid the issues I’ve been talking about.)

So instead I thought, if I was the sort of person that thought that 2 axis copies made sense (I don’t think they do, they’re inherently vague), what are the most important principles I would expect them to follow?

1. In the case of 0 influence or no local rotation on the target or constrained bone, the constrained bone should have no rotation. Obviously.

2. Regardless of angle, rotating the target in the uncopied axis should have no effect on the constrained bone. (Blender’s constraint fails this test.)

3. Rotating the constrained bone in its uncopied axis should rotate the constrained bone in a linear fashion-- ie, if it isn’t copying Y, and you rotate it 45 degrees in Y, it should rotate 45 degrees. This should be true regardless of the influence of the constraint.

4. Rotating the target bone in copied axes should rotate the constrained bone in a linear fashion when at full influence. IE, if it’s copying X and Y and the target’s tail rotates 45 degrees off rest, the constrained bone should also rotate 45 degrees. (Blender’s constraint fails this as well.) In the Euler version, however, this doesn’t scale linearly with influence, not necessarily.

After that, I wanted to make a prototype, to make sure that I wasn’t making any mistakes in how I handle things, to see it in action, because I’m not good enough to just see it all in my minds’ eye. Give me a decade

I gave it Euler interpolation, because what you were concerned about was mechanical rigging, even though it doesn’t look like the existing constraint uses Euler interpolation.

This demonstrates copy rotation without offset. Before original or after original are best represented by imagining a virtual parent or child for the structure (a multiplicative relationship, not an additive relationship.)

Order matters, as it does for Euler rotations. A 2-axis copy rotation has two potential orders.

Both the Blender original and my prototype have poles. This seems unavoidable to me, part of the nature of 2 axis rotation being something that doesn’t really make sense. I consider my prototype’s poles to be superior because they exist at predictable locations and because they occupy no extent: unless you hit the exact pole, you will never see my prototype creating twitch. That’s not true of the Blender original.

I believe that this could be done with Euler additions, if working in local space (in world space, it would stop making any sense to do that). The thing is, you’d need to use a special method to create the Euler angle in the specific format you needed for this operation. The standard way to create Euler angles gives the Blender original, with all its problems. It’s all about how you translate the matrix into the Euler, and there are literally an infinite number of different ways to do that. The particular way I measure things is what I was talking about when I said that even a transformation constraint shouldn’t just be reading regular Eulers; if it measured projections onto a plane, as my prototype is doing, you’d have much more intuitive behavior. (Because that twitch that you see in the demonstrated keyframes is also going to show up as twitch on a transformation constraint.)

If you wanted a 3-axis, Euler interpolation copy rotation, you’d want to extend the model I provided with a further locked track, and you’d have 6 potential orders, same as with Euler angle evaluations. A 1- or 3-axis, quaternion slerp interpolation is trivial IMO. A 1-axis Euler interpolation is also trivial, it’s the same thing as a 1-axis quaternion. I also believe that the world-space version of the prototype is obvious.

If there’s some other constraint you have in mind, where it would lead to inferior operation (not counting performance, although the performance issues here should amount to nanoseconds, not the best use of optimization resources) or where it would simply not be possible to do what people expect through the use of actual matrix multiplication instead of decomposition/recomposition, please let me know, I’ll treat it as a challenge in order to develop a prototype like this.