Guidelines for AI-enabled features

Feature & Design Feedback
1

Blender’s development roadmap is strictly focused on creative tools for artists. There are no plans to develop or implement “generative” AI features (e.g. prompt-based artwork generation, mesh generation, etc.) in Blender. However, other types of AI-enabled features could be researched and explored when looking at future development.

The goal of this document is to define which AI-enabled features are considered helpful and empowering for artists, while respecting Blender’s core privacy, safety and ethical principles.

In order for an AI-enabled feature to be considered, it must meet the following requirements.

  • Local and sovereign: The model can run locally, or be deployed anywhere by the user without restrictions. The model should be available under a copyleft-friendly license.
  • IP conscious: The model should be built on lawfully licensed content, respecting intellectual property.

We only focus on assistive tools and workflows: i.e., features that aim at reducing repetitive and non-creative operations, allowing artists to focus on fun and creative tasks instead.

As an example, some assistive AI has already been available in Blender for several years. The OIDN and OptiX denoisers help lighting and shading artists, drastically reducing render times for previews and final renders.

Here are examples of AI-enabled features that could be considered for research:

  • Render denoising (available since Blender 2.81 (OIDN) and 2.82 (OptiX))
  • Image processing (picture/video denoising, upscaling)
  • Mask generation and keying (object detection in static and moving images)
  • Motion capture and features detection (cleanup and blending, facial animation capture, body/hand detection, gesture recognition)
  • Voice to text, transcriptions (e.g. Sequencer subtitles)
  • Keying (green screen)
  • Animation and rigging (pose to pose interpolation, automatic joint placement and initial skinning)
  • Image to shader

Research and exploration of these AI-enabled features should happen in the context of Blender Lab. This will make it possible to prototype and iterate on ideas, to find out what really works, without the pressure to make the features immediately available as part of an official Blender release.

Several unknowns must be considered and figured out, the main one is how to make models available within Blender. Some models are very large, or depend on specific technologies and libraries. Some models could work directly in Blender: e.g. OIDN, DLSS-RR – well-established technologies from the rendering world. Finally, the tools should clearly reflect their assistive nature, and seamlessly fit within the existing design patterns of Blender.

Blender is a tool for artists and the purpose of AI-enabled features is the same as any other feature: to improve it. And, given today’s sensitive and controversial nature of the AI topic, we wish to make it explicit that researching this should happen in line with the project’s core values.

Check out the Blender roadmap for ongoing and planned Blender features.

38 Likes
2

What about Python scripting, a local model, trained on licensed code and the API documentation (which will likely need some ‘editing’), to allow a non-coding artist to be able to create at least small, production type scripts that automate repetitive tasks, etc?

3

LLMs are being explored as part of the Blender MCP experiment. The main challenge remains with model training, performance, and compliance with the requirements above.

4

Except that is very broad and general, trying to cover just about every function and facet of Blender (even used to create models/scenes, so could be half called ‘generative’) and due to the overall complexity, basically only works using commercial AI models. And even then, based on some reports, not doing it all that successfully.

While I’m talking about a very specific, focused use, much like your listed examples of Python scripting. But yes, the issue of then said model training (focused on the API), etc does remain.

5

I don’t think there is a way to train a model that’s good at Blender Python scripting, that’s somehow significantly simpler than training a general coding model.

These models only become effective by being trained on a very large amount of data. It’s possible to try to distill or post train an existing model to make it smaller while still being good at Blender Python scripting, but based on what I’ve read the gains would likely be modest, and it has all the same licensing questions.

2 Likes
6

This particular task sounds quite generative; replacing the work and value offered by animators and riggers.

7

I may be wrong about this, but two things come to mind. Be it a huge commercial model or a smaller local one, even if they are ‘coding’ specific, they have what I assume would be a lot of useless coding data from a Blender point of view.

So in addition to Python, they have C, C++, Javascript, Rust, etc, etc, none of which is in theory much use in getting a working Python script in Blender.

Then there’s the second problem, more so for smaller local models then the commercial ones and that is knowledge of the latest Blender API or even Blender in general.

I’ve messed around a little with a bunch of local models that I can run on my modest GPU and my first test is always to ask it two questions:

What was the last release of Blenders Python API that you were trained on?

What is the most recent release of the Blender API that you are aware of?

Without fail, I usually get two different answers (as in different Blender versions) from the exact same model to those questions. In every case, the answer is at most somewhere around a 3.X release and whats even more funny, is that in many cases the AI will give the official release date of the Blender version in question. The date it provides is never right.

So in many ways, I wonder if it’s less about very large amounts of data and more about actual specific and correct and complete Blender data.

A bit like say the Voice to text transcription, if you only feed the model a little bit of general French and a whole lot of English, then the resulting English text is likely to be pretty good, but any French text is going to have gaps.

Right now I think many AI models, even just for Python coding, have a lot of Blender gaps, while if Green screen keying is your thing, then I think Corridor Key mostly have that covered.

8

In my (I admit limited) experience and experiments, I found it enough to have a local copy (e.g. html download) of the entire latest python API and a checkout of the blender codebase. Combined with instructions on how to use these two resources (in an md file), even smaller models quickly find the things they need.

9

IP conscious

This one is going to be critical. I think we’re all very well aware that the desire to build shiny new things is outweighing the ethics in this field, and Blender lives by people very much at the other end of that debate. (edit: reading this sentence again, it feels a little harsh. not meant that way!)

What does and does not fall under lawfully licensed should be up to the community too. I’ve published much work under creative commons in the past, and although Creative Commons itself disagrees under much backlash, it being “lawfully licensed” feels like a violation to me. This goes for code, too, of course.

Blender itself will have to set clear boundaries here for the community to accept any such tools.

Pose detection

I can chime in on this one. At work I’m always looking for new pose estimation methods, from full body to hand. These tend to be o.k. license wise as they use custom-made training sets. However, this feels very much in the addon-space, as does much on the list of the original post to be honest.

Synthetic datasets

While we’re on the topic of ethical AI, we can’t look over synthetic datasets. Blender already plays a huge part in the creation of these as-is. Perhaps some add-on to formalize such workflows could be interesting.

2 Likes
10

What about an AI that helps with Blender usage itself ?
(I have this in another App)

For beginners or for tools that you seldom use, …

à la ….

“how can I move the center of an object’s, not the object itself ?”

“On top right of your View Window you find a dropdown “Options”. Open it and under Transforms, affect only, and activate “Origins”. Now you can graphically move the center. Don’t forget to deactivate the setting once you moved all desired centers. Alternatively you can go to Menu / Set Origins to …. “

(Would have saved my from endless post and discussions on Blender Artists at that time)

“how can I get a common simple reliable Texture Box Projection for my Bricks Image to avoid the need of unnecessary complex UV-ing my 50 Walls. Where I can control scale/size in Meters, rotations about XYZ and set XYZ offsets from file origin ?”

”I am sorry, in Blender you will need to create a quite complex node setup for this. Let’s go to Shader tab and I will do that for you“