Blendit (Blender + Git) - Version Control for Blender

[Imaginelenses]

What is Blendit (Blender + Git) ?

Blendit is an Application Template which brings Version Control to Blender.

Home

What is Version Control ?

Version Control helps you track different versions of your project. Working on a project overtime you may want to keep track of which changes were made, by whom, and when those changes were made.

Version control has been standard practice in software development to keep track of changes made to source code for years now. However, when it comes to working with files other than textual files you usually out of luck.

While Git and other Version Control Systems (VCS) can track .blend (binary) files it does not make much sense as they are designed for textual files.

That said, according to @sybren on Blender Stack Exchange, Blender Institute uses Subversion.

At the Blender Institute / Blender Animation Studio we use Subversion for our projects. It works fine for blend files, but you have to make sure they are not compressed. Compression can cause the entire file to be different when only a single byte changed, whereas in the uncompressed blend file only that one byte will differ. As a result, binary diffs will be much smaller, and your repository will be faster to work with.

How is Blendit Different?

Instead of tracking the .blend (binary) file itself, Blendit tracks the changes you make to the Blender file in real time. It does so by keeping track of the Python commands, from the Blender API, used to make changes.

This way we only track a textual (.py) file as Git was intended to be used.

Each time you open a Blendit project, it regenerates the Blender file. This means you can delete the Blender file and still retain the project.

In theory the size of the entire project should be lower than using any other VSC.

With all that said this is just a proof of concept and far from production ready. As mentioned earlier Blender does not log all actions out of the box. There are ways to make Blender log all actions, but that would add huge overhead in the system. The way I get around this is by subscribing to required events using Blender’s Message Bus. However, the Message Bus is seemingly in its early stages and cannot subscribe to all types of events. See release notes to know more.

In Conclusion

If you find this interesting, check the website below and download Blendit - It is Free!

Star Blendit on GitHub! This is the easiest way to show support for the project. Do report issues or provide suggestions. Contributions are welcome.

GitHub - imaginelenses/blendit: (Blender + Git) A Git integration for Blender.

Let me know what you think. If you have questions leave them below.

[bsavery]

Interesting idea!
I have a few questions (guess I should download and play around with it to see).

1. When does a “commit” happen? When the file is saved? Or does every action in blender generate some history in git? - If you think like with a python file in VS Code, you don’t really need to save all the text edits, just the diff at commit.
2. If you’re just committing / saving when user saves (or something like that) why is the message bus system needed? Couldn’t you for instance make some custom operator that does a “save / commit” and at that point walk the blender file for data blocks that change and save their “diff”. For example, at that point look for all meshes, and diff from the previous.

Just curious how this works under the hood.

[brecht]

It’s an interesting idea, though I expect that getting Blender to both log and reproduce all actions in a way that’s reliable for version control will never happen. It’s not something that could be handled with some additional work, but rather a ground up redesign. Nevermind getting it to work across Blender versions, operating systems and different CPUs/GPUs.

In a version control system I think you should start from guaranteed data integrity, and then work from there to optimize things while keeping that guarantee.

It’s possible to get much smaller .blend files and deltas between .blend files. Since .blend files are a memory dump, it would help for example to diff against default data values, zero all runtime data, and derandomize pointers. Better compression may be possible by taking into account the data layout.

[Imaginelenses]

When does a commit happen?

When the user explicitly chooses to commit.

What are you actually commiting?

A Python file full of commands like this.

...
bpy.ops.transform.translate(value=(-0, -0, -0.306203), orient_axis_ortho='X', orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(False, False, True), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)
bpy.ops.transform.rotate(value=-0.135777, orient_axis='X', orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(True, False, False), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)
bpy.ops.transform.resize(value=(1.15803, 1, 1), orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(True, False, False), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)
[obj.select_set(False) for obj in bpy.context.view_layer.objects.selected.values()]
[bpy.context.view_layer.objects.get(obj).select_set(True) for obj in ['Suzanne']]
bpy.context.view_layer.objects.active = bpy.data.objects['Suzanne']
bpy.ops.material.new()
bpy.context.object.active_material = bpy.data.materials[-1]
bpy.data.materials["Material"].node_tree.nodes["Principled BSDF"].inputs[0].default_value = (0.8, 0.192546, 0.0567258, 1)
...

Each action you perform on Blender has a corresponding “function call” under the hood which actually performs the action. This is what you see in the Info window

Why do you need the Message Bus?

Because Blender does not log a function call in the Info window for all actions you perform. (There are ways to make Blender log all actions, but that would add huge overhead in the system.)

For instance, when changing the active object by clicking. This does not add to the Info window but this function is required when regenerating the file.
So I use the Message Bus to subscribe to the event of change in Active Object.

Message Bus Subscriber

def subscribe():
    """Subscribes to different event publishers"""

    # Active Object
    bpy.msgbus.subscribe_rna(
        key=(bpy.types.LayerObjects, "active"),
        owner=blenditSubscriber,
        args=(),
        notify=activeObjectCallback,
        options={'PERSISTENT'}
    )

Therefore I’m not actually reading the .blend file and accessing the binary data in any way.

[Imaginelenses]

So I’m not dealing with the .blend binary file at all. I’ve explained in the previous comment about what exactly gets tracked by Git. Could you look at that and let me know what you think?

[brecht]

My comment is just that the information this is relying on is unfortunately not enough to ensure reproducible results. Regardless of the specific add-on implementation, which may well do the best it can with the information it gets, I have not looked into it.

The only practical way I see to make it reliable is to deal with .blend files.

[bsavery]

Agreed, and like I said in the earlier comment, if you’re working in a code editor for example, saving all the typing commands is way overkill and git does not do that. You’re essentially trying to do the same for blender commands.

Serializing and saving diffs of the Blend data seems totally possible…

[sybren]

I agree with @brecht.

To illustrate: there is no way that you can save all the changes to vertex positions of a one hour long sculpting session in the form of Python code that moves all those vertices around. Converting all data modification to text is not a guarantee that it will produce an efficient diff.

As another illustration, give 32-bit floating point numbers, a new 4x4 transformation matrix can be written as 4×16 = 64 bytes. However, this Python code would be around 1 kilobyte, roughly 16 times as much:

bpy.ops.transform.translate(value=(-0, -0, -0.306203), orient_axis_ortho='X', orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(False, False, True), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)
bpy.ops.transform.rotate(value=-0.135777, orient_axis='X', orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(True, False, False), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)
bpy.ops.transform.resize(value=(1.15803, 1, 1), orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(True, False, False), mirror=False, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False)

And, this would be written for each translation, rotation, or scale. Personally I wouldn’t want to store my entire edit history in a versioning system, but just the results of those edits. The same was as I don’t want others to be bothered by all my typos and failed coding attempts, but just see the final shiny success

I think it’s a great idea to critically look at how blend files can be written more compression-friendly. I’m 100% certain the people at the Blender Studio will thank you when you help them making both commits & updates faster. Also I recently had to upgrade the studio SVN server storage from 2TB to 4TB because it was getting full, so smaller commits definitely would help there too.

[Imaginelenses]

Oh, alright I see what y’all mean.

I did think of this early on but chose to work on regenerating the .blend files first and worry about reducing the size of the Python file later. So the way I intend to solve this is by parsing the Python file, once you commit or close Blender, and performing certain optimizations.

However, I don’t think the following paints the entire picture.

@sybren

As another illustration, give 32-bit floating point numbers, a new 4x4 transformation matrix can be written as 4×16 = 64 bytes. However, this Python code would be around 1 kilobyte, roughly 16 times as much

These 64 bytes do not live in isolation. There are other blocks of data in the .blend file required to first identify that these 64 bytes represent numbers, then that they represent a transformation matrix and then that they are associated to a certain object.

Consider this: An empty .blend file (without any objects) is roughly 765kB. A .blend file with the objects below is 892kB. That means everything done in this .blend file only add 127kB.

Now the .py file required to regenerate this .blend file is only 6.5kB. And the entire project (i.e including the .py file and the .git/ folder but excluding the .blend file itself) containing 3 commits is 119kB, roughly 1/6th as big as the original .blend file.

You can download the files here and try it for yourself. Admittedly, I do not know how this would scale.

A .blend file contains a lot of other information you wouldn’t want in a versioning system like the viewport’s camera position and settings (RegionView3D data).

What kind of optimizations can be done?

For instance, in this case we would delete “repeating” commands (Repeating not in the sense that they are all the same commands but that they are modifying the same parameter.)

bpy.data.materials["Material"].node_tree.nodes["Principled BSDF"].inputs[0].default_value = (0.8, 0.192546, 0.0567258, 1)
bpy.data.materials["Material"].node_tree.nodes["Principled BSDF"].inputs[0].default_value = (0.8, 0.192546, 0.0567258, 1)
bpy.data.materials["Material"].node_tree.nodes["Principled BSDF"].inputs[0].default_value = (0.8, 0.123637, 0.0451247, 1)
bpy.data.materials["Material"].node_tree.nodes["Principled BSDF"].inputs[0].default_value = (0.8, 0.123637, 0.0451247, 1)

Or when we translate, rotate and scale multiple times we could just store the object’s final world_matrix in the form of a single command that sets the objects transformation from the world_matrix.

I do realize it isn’t going to be this straightforward. But I see myself drawing parallels to Compiler Design and Optimization. I believe Computer Science is always a balancing act between throwing more memory or computation power at the problem and I’d like to help tip the scales.

[sybren]

@Imaginelenses

I did think of this early on but chose to work on regenerating the .blend files first and worry about reducing the size of the Python file later.

As we’ve already said, the information you’re working with is not enough to actually regenerate the blend file.

@Imaginelenses

These 64 bytes do not live in isolation.

True, but that Python code only concerns that transformation matrix.

@Imaginelenses

A .blend file contains a lot of other information you wouldn’t want in a versioning system like the viewport’s camera position and settings (RegionView3D data).

This I disagree with. The viewport camera shouldn’t be excluded. This can be super important, for example when reporting a bug it helps that the viewport is showing the buggy area of the model. The same can be said for handovers of blend files from one artist/department to another.

I really think that the suggestions of @brecht to make the blend file itself more compression-friendly would be much easier to get right, and be more effective. A whole lot of data changes will not be done via operators, and others are even impossible to do directly from Python.

[kurk]

There are so many instances where this idea of regenerating a .blend file from scratch will fail in real production work. For instance imagine one imports a .obj file and deletes the original .obj. How is Blendit going to rebuild the .blend file properly in this particular case?

This is a great idea but I feel like it wont be much of a use in real production work where the files can be over many GBs and contain thousands of objects and reference .blend files or collections.

[LudvikKoutny]

I second the notion that this approach has nearly 0 chance of working. Even if you managed to deterministically recreate all the steps from the python command log files, which would be miracle, there would be many occassions, especially when working with heavy data, where recreation could take so much time it would not be worth it. You would not want an artist who checks out your project’s repo waiting several days before the files are reconstructed and they can start working.

What would be an interesting idea is if there would be a way to version control not the entire monolithic .blend files, but rather individual data blocks inside them, since .blend files are technically just archives for individual datablock files. But I am not sure blender even has any API which would allow for that.