I agree that this is important and that we should do more ambitious changes for following code cleanup days (and outside of them). This is exactly the kind of problem we should be tackling, but I didn’t get around to working it out for the first cleanup day.
I’ll write design docs in the wiki, get them approved by other developers. And then break it down into smaller steps that people can help with (also within a single day). Making datablocks and nodes extensible are the most straightforward ones and would probably be first, but I’m also thinking about depsgraph, DNA and RNA.
For the new function/particle nodes I’m pushing the design to make it so implementing a new node can be done in as few files as possible, ideally just 1-2 C++ files. The cleanups I committed on the cleanup day are work towards making it easier to add more geometry types in Cycles. So it’s definitely something that’s in my mind.
I’m a proponent of having a C++ API. The RNA API that is used by Cycles is work towards that. However it’s a big project to make that good and stable enough for public use. At the time of Blender 2.5 I was hoping someone would pick up that project, but it didn’t happen and we didn’t have time ourselves.
I welcome development in this area, but the developer working on it needs to be prepared to do some big refactoring and deep changes to Blender. Probably for it to be really useful, first we need to do the kind of refactoring that Stefan refers to. The best entry point here is if we have a node system where you can implement a node in a single c++ file, which we are thinking of for the new function nodes design.
The speed people want is depending more on good coding even in python there’s numpy if one really needs c++ outside all that numpy offers one could still call external dlls outside the blender code base.
Not knowing c++ and thinking of other solutions to get a bit faster python code might not be the big gains learning c++ after python might also be a large personal gain, for the thinkers who the also be able to program Arduino rasberipi etc learning a language isn’t that hard, plenty of free quick intro coarses.
And how many of these bullet points would be solved by making a C api that wraps the same functions as the Python one does? I think the answer is: none of them. People use C to make patches and write modifiers because there is no way with Blender’s (Python) API to have the effect they want to have.
The key point is the phrase you used, “a more powerful API”. What you really want is an API that lets developers change the behavior of existing things (what the vast majority of patches do). And that is really hard to do.
Modifiers are a special case and maybe an exception. A modest extension of the API might let one write Python modifiers. Developers have resisted this somewhat because there is a fear that for large meshes, the cost of going in and out of python may make them seem too slow. Which I think is where the desire to have a C API may have originated. But in the long list of things that people want to change about Blender, I think “add a new modifier” is fairly uncommon.
That is not true. People want new modifiers every day, only that we accepted that this list rarely change. Also that is not near to the reality in any software…
90% of max plugins are modifiers
Houdini plugins are nodes that at the end are exact the same that modifiers.
If blender want things like this type of addons
Everthing nodes
Modifiers
Texture Noises
Hair
it needs a C API to make easy the development. especially if everything nodes is a reality in few months/years and the base of blender.
I think it’s important to clearly note the difference between having an internal C++ API and having binary plugins, end users tend to conflate the two (as @Alberto seems to be doing in his posts earlier in this thread) .
These are two rather different beasts and while C++ is a great match for one of them (having an internal API) the ABI issues make it an absolute horrid choice for the other (Binary plugins)
So just to clarify you are talking about having an internal C++ API only here, correct?
No, I’m talking about a public C++ API. The RNA API side steps the issue since the ABI is purely C. I’m not sure that is exactly the approach we want to take but it’s an option. C++ ABI compatibility isn’t as bad as it used to be either, and a lot of applications do have it.
Anyway, this is a complicated project and I don’t think it’s something that the Blender Foundation will tackle in the near term. We also wouldn’t accept a implementation by another contributor unless it meets fairly high standards. But if the resources are available, I think it’s a good feature to have.
Tat’s the key of all this conversation, Python is slow as soon as you work with some production scene/meshes, and it’s not just IF we could have modifiers, it’s a general thing, it’s slow dealing with big chunks of data, that’s one of the major problems.
Also I agree with Alberto that modifiers are needed, a lot.
The so called “Edit Poly” modifier is not useful for modeling at all, I never missed it in Blender, but it’s useful when you want to do a procedural modification and be able to disable/enable that modification, a “Freeze Edit” modifier that allows you to enter in a “different” edit mode, which in reality is not different but just some edits that are stored inside that modifier instead of being inside of the object for example.
Another interesting modifier is the UV Unwrap, which is basically store a UV Channel inside a modifier, for example let’s say you have a mesh with a subdivision surface, you have one UV without that Subdiv, then you want a different UV when you have the Subdiv, or you need to modify those UV’s for whatever reason, but you don’t want to loose the original UV’s, the low resolution version of the model and you don’t want to apply the subdivision surface… right now that’s impossible in Blender, a modifier that is capable of storing different UV channels could be a solution.
There are tons and tons of situations where modifiers would be very welcome.
With that said I think “Function Nodes” generated modifiers solve part of the problem, none of the two I mentioned, but some others could be solved.
This is doable with the almighty Data Transfer modifier- keep a duplicate of the mesh that is subdivided and UV that, then use Data Transfer to transfer the UVs by Topology. Not the fastest or most efficient.
Anyways, more to the point- I agree that Python is too slow for fast mesh access/editing. I think the solution is to use ctypes to hook in c modules— get the data with Python (as a pointer), send it to the C module, and it’s essentially a native C or C++ API in terms of performance, with more flexibility. I would like to see a set of tools/libraries for making these modules more easily. For instance, a minimal set of classes for converting data back and forth or “translating” it from Python objectsto C structs or C++ objects and back. This seems to me the easiest way to solve the problem in the short term (with space to work on a better long term solution). Despite my prodding, I haven’t been able to get any of the more experienced programmers to share their experience with this method of working (I haven’t gotten a chance to do this myself, yet, either). I’d really like to discuss the advantages and disadvantages of this idea, especially since brecht is saying that a C++API isn’t ruled out for the eventual future (news to me, I wonder how the rest of the dev team thinks of the idea?)
A drawback I haven’t seen brought up yet is this: threading support. One of the difficulties with Python performance is that it can’t access Blender data asynchronously. If multiple threads access Blender’s context or blend data, there can be some pretty nasty undefined behavior. Obviously, this limits many operations to single thread performance. I can’t imagine this would be an easier problem to solve for a lower-level API. I think the only way to solve this problem would be to modify the memory directly, maybe as RNA or maybe as data in RAM (via a pointer or something like that). But is it possible to do that in a safe way?
Foreign language interfaces from python to any thing will work, you just have to make sure the underlying code, C or whatever is compiled for the operating system that the code needs to run on. The is possible, but more painful than a fully python solution which just works.
However correct me if I am wrong but there are certain things that have to be built into blender, nodes in the compositor, that you can’t just get via python
I don’t think that edit modifier is necessary in the actual system. also that blender modifiers are only “nodes”. In reality you can’t enter in a modifier with special tools and edit mode, like max.
If the plugin API would be based on C (not C++) what ABI issues would you foresee? And for C++ are you thinking of ABI issues on particular platforms? I can only recall one instance of ABI incompatibilities with G++ from a number of years ago.
Plus, as long as plugins are recompiled for each release of Blender and are mostly distributed as source does the ABI really matter?
I don’t forsee any issues with a C style API, many applications use it, it should have no issues…
C++ however…
It’s nearly impossible to guarantee binary compatibility between compiler versions and platforms, odds that the memory layout of a std::vector<sometype> will match between different compilers or even different versions of the same compiler, or as in msvc’s case, debug vs release builds of the same compiler version. are near the zero side of things.
So any complex types are out.
One could stick to void*'s, char*'s in well defined structs to sidestep all these issues but at that point, you just made a C-API (I left out any inheritance/vtable based issues but those are not easy to solve either and get you to the same point of having a C API)
All these problems go away if the consumers of the API all live in a single codebase (internal C++ API is not a problem) but all hell pretty much breaks lose as soon as you try to expose a C++ API in it’s binary form.
I wonder if there could be a special modifier (say a Script modifier) that compiles Python or even C code with limitied access to certain data, similar to OSL node. Initially it would only supports deform modifier clones.
I mean I don’t think other software API has any relation with Blender possible API, I say that respectfully but I honestly don’t understand why is it interesting
And Blender devs have more than enough experience with API’s, it’s not like when we talk something artist oriented
@JuanGea it’s an objective example of what’s literally being discussed in this thread, a C/C++ API available through a public SDK.
I don’t know about you, but I personally always look at previous works when I’m trying to create something, so I can analyze what decisions they made, the solutions they came up with, what I like and dislike about it… you know, making an informed decision.
Yes, I understand that, what I meant is that regarding an APIs the devs are very versed in several, and Max API is just one more amongst any other, unless you think there is something special about it, some feature or characteristic that makes it different to other APIs
