Guidelines for AI-enabled features

Feature & Design Feedback
1

Blender’s development roadmap is strictly focused on creative tools for artists. There are no plans to develop or implement “generative” AI features (e.g. prompt-based artwork generation, mesh generation, etc.) in Blender. However, other types of AI-enabled features could be researched and explored when looking at future development.

The goal of this document is to define which AI-enabled features are considered helpful and empowering for artists, while respecting Blender’s core privacy, safety and ethical principles.

In order for an AI-enabled feature to be considered, it must meet the following requirements.

  • Local and sovereign: The model can run locally, or be deployed anywhere by the user without restrictions. The model should be available under a copyleft-friendly license.
  • IP conscious: The model should be built on lawfully licensed content, respecting intellectual property.

We only focus on assistive tools and workflows: i.e., features that aim at reducing repetitive and non-creative operations, allowing artists to focus on fun and creative tasks instead.

As an example, some assistive AI has already been available in Blender for several years. The OIDN and OptiX denoisers help lighting and shading artists, drastically reducing render times for previews and final renders.

Here are examples of AI-enabled features that could be considered for research:

  • Render denoising (available since Blender 2.81 (OIDN) and 2.82 (OptiX))
  • Image processing (picture/video denoising, upscaling)
  • Mask generation and keying (object detection in static and moving images)
  • Motion capture and features detection (cleanup and blending, facial animation capture, body/hand detection, gesture recognition)
  • Voice to text, transcriptions (e.g. Sequencer subtitles)
  • Keying (green screen)
  • Animation and rigging (pose to pose interpolation, automatic joint placement and initial skinning)
  • Image to shader

Research and exploration of these AI-enabled features should happen in the context of Blender Lab. This will make it possible to prototype and iterate on ideas, to find out what really works, without the pressure to make the features immediately available as part of an official Blender release.

Several unknowns must be considered and figured out, the main one is how to make models available within Blender. Some models are very large, or depend on specific technologies and libraries. Some models could work directly in Blender: e.g. OIDN, DLSS-RR – well-established technologies from the rendering world. Finally, the tools should clearly reflect their assistive nature, and seamlessly fit within the existing design patterns of Blender.

Blender is a tool for artists and the purpose of AI-enabled features is the same as any other feature: to improve it. And, given today’s sensitive and controversial nature of the AI topic, we wish to make it explicit that researching this should happen in line with the project’s core values.

Check out the Blender roadmap for ongoing and planned Blender features.

48 Likes
2

What about Python scripting, a local model, trained on licensed code and the API documentation (which will likely need some ‘editing’), to allow a non-coding artist to be able to create at least small, production type scripts that automate repetitive tasks, etc?

3

LLMs are being explored as part of the Blender MCP experiment. The main challenge remains with model training, performance, and compliance with the requirements above.

4

Except that is very broad and general, trying to cover just about every function and facet of Blender (even used to create models/scenes, so could be half called ‘generative’) and due to the overall complexity, basically only works using commercial AI models. And even then, based on some reports, not doing it all that successfully.

While I’m talking about a very specific, focused use, much like your listed examples of Python scripting. But yes, the issue of then said model training (focused on the API), etc does remain.

5

I don’t think there is a way to train a model that’s good at Blender Python scripting, that’s somehow significantly simpler than training a general coding model.

These models only become effective by being trained on a very large amount of data. It’s possible to try to distill or post train an existing model to make it smaller while still being good at Blender Python scripting, but based on what I’ve read the gains would likely be modest, and it has all the same licensing questions.

4 Likes
6

This particular task sounds quite generative; replacing the work and value offered by animators and riggers.

3 Likes
7

I may be wrong about this, but two things come to mind. Be it a huge commercial model or a smaller local one, even if they are ‘coding’ specific, they have what I assume would be a lot of useless coding data from a Blender point of view.

So in addition to Python, they have C, C++, Javascript, Rust, etc, etc, none of which is in theory much use in getting a working Python script in Blender.

Then there’s the second problem, more so for smaller local models then the commercial ones and that is knowledge of the latest Blender API or even Blender in general.

I’ve messed around a little with a bunch of local models that I can run on my modest GPU and my first test is always to ask it two questions:

What was the last release of Blenders Python API that you were trained on?

What is the most recent release of the Blender API that you are aware of?

Without fail, I usually get two different answers (as in different Blender versions) from the exact same model to those questions. In every case, the answer is at most somewhere around a 3.X release and whats even more funny, is that in many cases the AI will give the official release date of the Blender version in question. The date it provides is never right.

So in many ways, I wonder if it’s less about very large amounts of data and more about actual specific and correct and complete Blender data.

A bit like say the Voice to text transcription, if you only feed the model a little bit of general French and a whole lot of English, then the resulting English text is likely to be pretty good, but any French text is going to have gaps.

Right now I think many AI models, even just for Python coding, have a lot of Blender gaps, while if Green screen keying is your thing, then I think Corridor Key mostly have that covered.

8

In my (I admit limited) experience and experiments, I found it enough to have a local copy (e.g. html download) of the entire latest python API and a checkout of the blender codebase. Combined with instructions on how to use these two resources (in an md file), even smaller models quickly find the things they need.

1 Like
9

IP conscious

This one is going to be critical. I think we’re all very well aware that the desire to build shiny new things is outweighing the ethics in this field, and Blender lives by people very much at the other end of that debate. (edit: reading this sentence again, it feels a little harsh. not meant that way!)

What does and does not fall under lawfully licensed should be up to the community too. I’ve published much work under creative commons in the past, and although Creative Commons itself disagrees under much backlash, it being “lawfully licensed” feels like a violation to me. This goes for code, too, of course.

Blender itself will have to set clear boundaries here for the community to accept any such tools.

Pose detection

I can chime in on this one. At work I’m always looking for new pose estimation methods, from full body to hand. These tend to be o.k. license wise as they use custom-made training sets. However, this feels very much in the addon-space, as does much on the list of the original post to be honest.

Synthetic datasets

While we’re on the topic of ethical AI, we can’t look over synthetic datasets. Blender already plays a huge part in the creation of these as-is. Perhaps some add-on to formalize such workflows could be interesting.

2 Likes
10

What about an AI that helps with Blender usage itself ?
(I have this in another App)

For beginners or for tools that you seldom use, …

à la ….

“how can I move the center of an object’s, not the object itself ?”

“On top right of your View Window you find a dropdown “Options”. Open it and under Transforms, affect only, and activate “Origins”. Now you can graphically move the center. Don’t forget to deactivate the setting once you moved all desired centers. Alternatively you can go to Menu / Set Origins to …. “

(Would have saved my from endless post and discussions on Blender Artists at that time)

“how can I get a common simple reliable Texture Box Projection for my Bricks Image to avoid the need of unnecessary complex UV-ing my 50 Walls. Where I can control scale/size in Meters, rotations about XYZ and set XYZ offsets from file origin ?”

”I am sorry, in Blender you will need to create a quite complex node setup for this. Let’s go to Shader tab and I will do that for you“

11

Well, I wonder if that could be a Blender lab project, basically a ‘one-click’ setup that does all that (with a tested and recommended model that you’d download from huggingface) with the ability to easily keep the local python API, etc up to date.

My general ‘benchmark’ in this is something like Davinci Resolve and goes direct back to one of listed examples (Mask generation and keying), where Resolve has a ‘Magic Mask’ feature (in addition to various old style masking tools) that to get it to work, only requires clicking the ‘Additional Downloads’ button that then downloads the AI model to do all the work.

Everything about the feature is already in Resolve, only the local model isn’t part of the default download/install. And to be clear, it is local only, any of the few AI features that Resolve has only works using the download of it’s own local model.

12

One way is to pick some smarter SLMs and use vector database which will be basically pyapi documentation. With that there’s no worry about training model. Plus updating
vector database is simpler than training entire model again when there’s slight change in blender api.

13

@fsiddi

I would say those are good guidelines for AI, especially in combination with Blender. Just to add to the list of potential AI tools, I think obvious candidates are tools to help with retopology & unwrapping (like this proprietary AI program that is under development).

In my opinion, AI in Blender should ideally be seen as tools that put modifications (some sort of modifiers) on top of something already existing in your 3D scene. Not just making a 3D object out of thin air from a photo and a text prompt. Applying a modification/modifier on top gives the 3D artist a crucial freedom over the AI modification - to tweak it, dial it down, or turn it off.

To give an example:
I could imagine - down the road with better AI - that you could let an AI tweak the lighting in your scene for you. For instance - you have a 3D interior scene (with properly named lights), and you then show the AI a photo of an interior with a certain aesthetic/atmosphere. Guided by the photo - and follow up text prompts from you - the AI puts modifications on top of your light sources in the scene, and additional some post-production filters. You then, as a user, have full control to tweak these modifications. V-Ray has some pretty advanced tools in its framebuffer to tweak the rendering in post; I could imagine an AI tool applying modifications in a somewhat similar way.

And this above example is (of course) not a feature request, just a imagined possibility sometime in the future.

14

I understand the list is not exhaustive. But what about things like neural shader, neural materials, neural rendering, neural compression? I know these terms are coined by NVIDIA, so I don’t necessarily mean the specific versions made by NVIDIA, but just these technologies in general. Will they be considered?

15

Neural shaders/materials and neural texture compression I think are compression methods that are uncontroversial, and don’t involve training on large amounts of data and IP issues. So they should be fine.

Neural rendering I believe is an umbrella term for various things, so depends on the specifics.

7 Likes
16

Well I am referring to Nvidia’s usage of neural rendering, which seems to be about making path tracing faster, if I understand it correctly.

17

For example DLSS 5 is called a “real-time neural rendering model” by NVIDIA and that’s leaning into generative AI, so could be more controversial. On the other hand DLSS-RR 4.5 with denoising and upscaling for path tracing is in line with existing denoising support.

1 Like
18

Neural rendering is a marketing term that often implies using diffusion models, and also other techniques. This is something that requires very careful evaluation.

1 Like
19

I agree, the interpolation part steps over the artistic sensibility an animator brings to the table. (It’s a bit of a grey area, I can understand how that can seem like “filling in the blanks”- but there’s intent in there)

On the other hand I don’t see a problem with an AI inferring joint placement and doing the weight painting for me : those are just means to an end and I welcome the automation.

Overall I salute the clarity and the unambiguousness of the proposal

20

I understand. Well, there are always some grey areas.