I think that in these cases they always try to look more at the problems generated by doing something than at the benefits.
I don’t think anyone expects that a new API is not extra work, even that it can complicate development in some ways. But we don’t seem to want to see the positive.
Ease of development of tools by third parties
Removing the entire bottleneck from the patches
Remove the entire bottleneck of repairing patch problems
Remove all the bottleneck from users by asking for new features that must fit into the blender puzzle
Eliminate the bottleneck that prevents the implementation of new modifiers.
Eliminate the need to compile a blender branch for anything
Simplify the development model, allowing people with less knowledge to maintain third party features
What is the point of having an open source program where because a person, let’s say very purist, refuses for half a decade (literally) to implement a modifier that is already programmed as the weld for reasons of purism and internal bureaucracy? Let’s compare both development models
With a more powerful API in the hands of the users/third developers:
Unknown user creates a simple weld modifier, it’s not perfect, but it works for people. It creates it in a week.
People download it and use it, at their own risk.
END (Time needed for one week of work by one person)
With the current model:
Someone has an idea for a modifier and creates their code for the modifier. He creates it in a week.
You must convince, for weeks (or months or years), some developer to review it.
A programmer sees you as having structural problems
Original developer reviews it
Another sees performance problems
Original developer reviews it
One says the comments and the way it’s written is rubbish, let him fix it
Original developer reviews it
After the UI team decides that your implementation is not suitable for the program template, please change it because the usability is not as expected.
After all this the developers decide to stop paying attention to the “boring guy” of the modifier.
Users decide to start compiling their versions, mixing their diff with as many others.
Every year someone critic the situation of the patch on the forums, which because the patch is ignored for months/years
Developers begin to argue that the model works and the decisions were right
After 200 post in the forum someone admits they’ll check it out
In case the original developer is out there, maybe he will update and revise it. Otherwise the developer is said to have abandoned the project and nobody wants to mess with it.
END (Time needed for from months, at best, to years of waiting. It took the time of 5-6 people, hundreds of users complaining, compilations, developers constantly interrupting their work).
This is an absolutely wrong working model from almost any point of view.
And on the topic of the last comments, I wish, simply, that most of the developers would stop adding features to blender for 4-5 months (to say a range of time) and just fix everything that users complain about blender. No bugs, no new features, no magic ideas,… No, just pick up and say, come on, let’s do this
solve all problems generated with blender2.8 or without complete
solve performance issues when modeling
solve the problems of the subdivision
solving multiresolution problems
solving the problems of the undo
to solve the problems of heavy scenes
And a few other things.
And no more “long term project” and have the developers to 20 different tasks. No, just fix that. And until none of that is fixed, nothing new is added or skipped from idea to idea unless there are simply resources left over that don’t make sense to dedicate to these points. It can’t be that we don’t know when the hell the subdivision is going to work properly. That we don’t know what year we’re going to be able to edit heavy meshes. That we don’t know when the undo will start working.
It’s just that nothing that you want to add matters as long as those problems aren’t solved.
What can’t be is that you want to make blender grow in things like “everything nodes” when it doesn’t even seem that anyone is clear that the code on which it will be settled is reliable or that it allows the performance that is logically expected from something like that.
To get bit more back onto the topic, a lot of the points raised above about code complexity, development processes, C vs Python API, etc glance over one major point: what exactly should the Python (or C/C++) API support and to what extent? I.e. what types of extensions and tasks? An external render engine needs a lot more control through the API than a simple import script (which can just generate a set of scene elements). A new compositing operation would only need to take two images and produce a new output one. But an addon that wants to provide GUI elements hooks into a lot of internals.
The current Python API exposes many of the underlying code details, to pretty much give full control and flexibility at the cost of complexity and relative instability as the API will reflect any changes of the underlying code. But this does not have to be that way. One could provide a stable® API based on higher-level concepts that don’t directly map 1:1 to the current Blender implementation. For example, you could have import plugins pass a mesh in a fixed data structure (eg set of arrays) without having to know what the current implementation uses in terms of mesh structure. Conversion could be done in the API layer. For a restricted set of tasks you could provide such an API. At the expense of extra maintenance work and slightly less performance as you’re not working 1:1 on the underlying structures.
I guess it’s partly due to the fact that a lot of the Blender UI is written in Python that the API exposes so much of the internals. Not because this was the best way to provide extension of Blender by external developers, maybe that was a bit of a by-product?
Fun fact: there used to be a time when you could write a sequencer plugin in C (for ancient versions of Blender)
I agree that this is important and that we should do more ambitious changes for following code cleanup days (and outside of them). This is exactly the kind of problem we should be tackling, but I didn’t get around to working it out for the first cleanup day.
I’ll write design docs in the wiki, get them approved by other developers. And then break it down into smaller steps that people can help with (also within a single day). Making datablocks and nodes extensible are the most straightforward ones and would probably be first, but I’m also thinking about depsgraph, DNA and RNA.
For the new function/particle nodes I’m pushing the design to make it so implementing a new node can be done in as few files as possible, ideally just 1-2 C++ files. The cleanups I committed on the cleanup day are work towards making it easier to add more geometry types in Cycles. So it’s definitely something that’s in my mind.
I’m a proponent of having a C++ API. The RNA API that is used by Cycles is work towards that. However it’s a big project to make that good and stable enough for public use. At the time of Blender 2.5 I was hoping someone would pick up that project, but it didn’t happen and we didn’t have time ourselves.
I welcome development in this area, but the developer working on it needs to be prepared to do some big refactoring and deep changes to Blender. Probably for it to be really useful, first we need to do the kind of refactoring that Stefan refers to. The best entry point here is if we have a node system where you can implement a node in a single c++ file, which we are thinking of for the new function nodes design.
The speed people want is depending more on good coding even in python there’s numpy if one really needs c++ outside all that numpy offers one could still call external dlls outside the blender code base.
Not knowing c++ and thinking of other solutions to get a bit faster python code might not be the big gains learning c++ after python might also be a large personal gain, for the thinkers who the also be able to program Arduino rasberipi etc learning a language isn’t that hard, plenty of free quick intro coarses.
And how many of these bullet points would be solved by making a C api that wraps the same functions as the Python one does? I think the answer is: none of them. People use C to make patches and write modifiers because there is no way with Blender’s (Python) API to have the effect they want to have.
The key point is the phrase you used, “a more powerful API”. What you really want is an API that lets developers change the behavior of existing things (what the vast majority of patches do). And that is really hard to do.
Modifiers are a special case and maybe an exception. A modest extension of the API might let one write Python modifiers. Developers have resisted this somewhat because there is a fear that for large meshes, the cost of going in and out of python may make them seem too slow. Which I think is where the desire to have a C API may have originated. But in the long list of things that people want to change about Blender, I think “add a new modifier” is fairly uncommon.
That is not true. People want new modifiers every day, only that we accepted that this list rarely change. Also that is not near to the reality in any software…
90% of max plugins are modifiers
Houdini plugins are nodes that at the end are exact the same that modifiers.
If blender want things like this type of addons
Everthing nodes
Modifiers
Texture Noises
Hair
it needs a C API to make easy the development. especially if everything nodes is a reality in few months/years and the base of blender.
I think it’s important to clearly note the difference between having an internal C++ API and having binary plugins, end users tend to conflate the two (as @Alberto seems to be doing in his posts earlier in this thread) .
These are two rather different beasts and while C++ is a great match for one of them (having an internal API) the ABI issues make it an absolute horrid choice for the other (Binary plugins)
So just to clarify you are talking about having an internal C++ API only here, correct?
No, I’m talking about a public C++ API. The RNA API side steps the issue since the ABI is purely C. I’m not sure that is exactly the approach we want to take but it’s an option. C++ ABI compatibility isn’t as bad as it used to be either, and a lot of applications do have it.
Anyway, this is a complicated project and I don’t think it’s something that the Blender Foundation will tackle in the near term. We also wouldn’t accept a implementation by another contributor unless it meets fairly high standards. But if the resources are available, I think it’s a good feature to have.
Tat’s the key of all this conversation, Python is slow as soon as you work with some production scene/meshes, and it’s not just IF we could have modifiers, it’s a general thing, it’s slow dealing with big chunks of data, that’s one of the major problems.
Also I agree with Alberto that modifiers are needed, a lot.
The so called “Edit Poly” modifier is not useful for modeling at all, I never missed it in Blender, but it’s useful when you want to do a procedural modification and be able to disable/enable that modification, a “Freeze Edit” modifier that allows you to enter in a “different” edit mode, which in reality is not different but just some edits that are stored inside that modifier instead of being inside of the object for example.
Another interesting modifier is the UV Unwrap, which is basically store a UV Channel inside a modifier, for example let’s say you have a mesh with a subdivision surface, you have one UV without that Subdiv, then you want a different UV when you have the Subdiv, or you need to modify those UV’s for whatever reason, but you don’t want to loose the original UV’s, the low resolution version of the model and you don’t want to apply the subdivision surface… right now that’s impossible in Blender, a modifier that is capable of storing different UV channels could be a solution.
There are tons and tons of situations where modifiers would be very welcome.
With that said I think “Function Nodes” generated modifiers solve part of the problem, none of the two I mentioned, but some others could be solved.
This is doable with the almighty Data Transfer modifier- keep a duplicate of the mesh that is subdivided and UV that, then use Data Transfer to transfer the UVs by Topology. Not the fastest or most efficient.
Anyways, more to the point- I agree that Python is too slow for fast mesh access/editing. I think the solution is to use ctypes to hook in c modules— get the data with Python (as a pointer), send it to the C module, and it’s essentially a native C or C++ API in terms of performance, with more flexibility. I would like to see a set of tools/libraries for making these modules more easily. For instance, a minimal set of classes for converting data back and forth or “translating” it from Python objectsto C structs or C++ objects and back. This seems to me the easiest way to solve the problem in the short term (with space to work on a better long term solution). Despite my prodding, I haven’t been able to get any of the more experienced programmers to share their experience with this method of working (I haven’t gotten a chance to do this myself, yet, either). I’d really like to discuss the advantages and disadvantages of this idea, especially since brecht is saying that a C++API isn’t ruled out for the eventual future (news to me, I wonder how the rest of the dev team thinks of the idea?)
A drawback I haven’t seen brought up yet is this: threading support. One of the difficulties with Python performance is that it can’t access Blender data asynchronously. If multiple threads access Blender’s context or blend data, there can be some pretty nasty undefined behavior. Obviously, this limits many operations to single thread performance. I can’t imagine this would be an easier problem to solve for a lower-level API. I think the only way to solve this problem would be to modify the memory directly, maybe as RNA or maybe as data in RAM (via a pointer or something like that). But is it possible to do that in a safe way?
Foreign language interfaces from python to any thing will work, you just have to make sure the underlying code, C or whatever is compiled for the operating system that the code needs to run on. The is possible, but more painful than a fully python solution which just works.
However correct me if I am wrong but there are certain things that have to be built into blender, nodes in the compositor, that you can’t just get via python
I don’t think that edit modifier is necessary in the actual system. also that blender modifiers are only “nodes”. In reality you can’t enter in a modifier with special tools and edit mode, like max.
If the plugin API would be based on C (not C++) what ABI issues would you foresee? And for C++ are you thinking of ABI issues on particular platforms? I can only recall one instance of ABI incompatibilities with G++ from a number of years ago.
Plus, as long as plugins are recompiled for each release of Blender and are mostly distributed as source does the ABI really matter?
I don’t forsee any issues with a C style API, many applications use it, it should have no issues…
C++ however…
It’s nearly impossible to guarantee binary compatibility between compiler versions and platforms, odds that the memory layout of a std::vector<sometype> will match between different compilers or even different versions of the same compiler, or as in msvc’s case, debug vs release builds of the same compiler version. are near the zero side of things.
So any complex types are out.
One could stick to void*'s, char*'s in well defined structs to sidestep all these issues but at that point, you just made a C-API (I left out any inheritance/vtable based issues but those are not easy to solve either and get you to the same point of having a C API)
All these problems go away if the consumers of the API all live in a single codebase (internal C++ API is not a problem) but all hell pretty much breaks lose as soon as you try to expose a C++ API in it’s binary form.
