Improving Python/C/C++ and general development process

[Josephbburg]

It’s worth mentioning that Maya actually has a Python wrapper for the C++ API (well, two actually), and no less than two Python wrappers for its MEL API. Every other major DCC has been in the process of replacing their built-in languages with Python, or at least wrapping them in Python. The trend in the industry is towards Python. Hell, the trend in C++ is towards Python, too! Now Blender is the first and maybe the only DCC app that supports Python natively in a Pythonic way. Just try writing a Maya script with Python- its completely backwards syntax makes it a wrestling match with your own instincts.
I think the answer to the C/C++ question is to improve the Python API. The bpy_struct.as_pointer() method should be sufficient to pass the data to a compiled module that operates directly on the memory address within Blender. Perhaps more work needs to be done to make this process less painful, but Python isn’t too slow to query memory locations and pass int’s back and forth.

I’d like to see a minimum version of Blender or a more officially supported bpy build for compiling C/C++ modules against to be used in a Python ctypes plugin/interface. I’d love to see it implemented in the Apache license like Cycles, to attract developers that wish to make money from the software. Although I’d also love to see it enforce software freedom via the GPL2, but it’s a hypothetical anyways… Perhaps this is easy to do already, in which case documentation is what is needed.

On the opposite, an API is generally stable for some years

But at what cost? Take a look at some of the “industry standard” softwares- what you’ll find is bloated, buggy, slow softwares that have to remain compatible with decades-old APIs. Take Maya as an example- its node-based architecture and extensive MEL API were groundbreaking in the late 90’s when Blender didn’t even have undo support yet. But MEL hasn’t aged well, and Maya is too deeply wedded to this extremely poorly designed language to refactor. Despite being an object-oriented C++ program, Maya’s entire interactive structure is based around a strictly imperative language with only five data types-- and barely any ability to handle exceptions or write structured programs. To be fair, this isn’t a C++ API, but I’d be willing to bet Maya has the same problem in its C++ API. Having to keep compatibility to an API is limiting for the future of a project. That’s why we see such slow development in the proprietary world- they didn’t plan for the future and now they can’t take big steps without losing their customers by breaking compatibility. It also underscores the folly of writing software in a proprietary language/API, but that’s off topic.

One of my favorite things about Blender is the bravery of its development cycle- we’re not afraid to change the architecture of the software in a major way if it leads to improvements. We’re not afraid to spend months or years designing things to be sure we won’t have to refactor again anytime soon. Unlike proprietary tools, we don’t have to worry about losing customers, meeting yearly feature release deadlines, or pleasing shareholders. Free software means that we’re free to focus on the future of Blender, rather than the short-term. There’s no risk of going out of business. Autodesk can’t buy us and shut us down, like they did with SoftImage.
Well… I’ve gone off topic myself now.

[ThatAsherGuy]

I’m not in a position to comment on how the API itself could be sped up, but I do think that a broader set of utility functions could reduce the amount of information that needs to be passed back and forth through it. There are few I’d love to see for the gizmo system, at least.

[c2ba]

Having to keep compatibility to an API is limiting for the future of a project.

I agree, but it’s the same for a python API. The constraints of having to maintain compatibility cannot be an argument for not having a C API, if an API is already provided through another language. Arguments for python would be more easy to code (can be discussed) , no need to build, tons of modules on PyPI, easy reload, I guess. But all these arguments don’t go against the interest of having also a C API (especially if its a mirror).
But I may be missing something about the complexity of the thing, I never had to implement an API for a software and I guess there are other things to consider.

[sybren]

Thanks @JuanGea

For me the major reason to not have a C/C++ plugin API is that it’ll get in the way of Blender development. Such an API should remain as stable as possible, but being in the same language as the core of Blender it’ll be tightly coupled with it. As a result, this would mean that it’ll be very hard to change Blender’s internals. Given that this is already hard with complex code with lots of tweaks, hacks, optimisations, fixes, and more, I really wouldn’t welcome another factor to juggle

@Josephbburg

It’s worth mentioning that Maya actually has a Python wrapper for the C++ API (well, two actually), and no less than two Python wrappers for its MEL API.

Yeah, that’s probably doable when they only release like one version per year and have a big team to work on it. It’s late Friday night here, but if I have the numbers straight there are approximately 18 Blender developers on payroll by Blender Foundation + Institute. It’s never been that many in the past, and it’s awesome this is possible now, but it’s really nothing compared to Autodesk or the other big’uns.

PS: yay it was Code Quality day today, and I managed to add the first unit tests for the animation system and improve readability & understandability of FCurve evaluation. Blender’s code just got that little bit easier to understan.d

[John1]

Not only that C++ is less forgiving and you have to deal with memory management …etc also develop for different OS on different versions.
If you have a bug like index error or divide by zero Blender will just crash while in Python it’s just the addon and you get a stack to debug it, so IMO It’s best to speed up Python than adding another API which is the only benefit of a C++ API.

[Josephbburg]

The point I was making by mentioning the Python wrapper is this: even though they have a C++ API, they made not one but two Python wrappers for it. Python is the trend, and Python is what Blender is doing better than the others. They’re all still using Python 2.7, even Houdini! Blender is ahead of the curve using Python 3.x

that’s probably doable when they only release like one version per year and have a big team to work on it.

No! That’s backwards. They have one version per year and a big team working on it because they have so many different APIs to maintain compatibility with - MEL, Python, PyMEL, C++ API, .NET API, and both Python APIs! There’s no way to build on such a shaky foundation!

@sybren Do you have anything to add about the ctypes and bpy_struct.as_pointer() workflow? I’d like to hear from you about the advantages and disadvantages of working this way for high-performance code. Also, thanks for the Code Quality! Can’t wait to see how you supercharge animation performance in 2020 !

[Howard_Trickey]

I agree with Dr. Sybren’s main point about why there is resistance against a C/C++ plugin API – that it could get in the way of Blender Development. But cb2a may still be unconvinced that it is that different from having a Python API – as he says, “Since there is a Python API that basically wrap C functions calls, why not just doing the exact same in C ?”. Let me add some to what Dr. Sybren said to expand on the reasons why it is different for C than for Python.

First, C (which the vast majority of Blender is written in), has inflexible type signatures for function names. That means, if you expose a C function which takes arguments of a particular type, that function is now forever (well, at least as long as one wants a stable API) tied to take only and exactly those arguments and only those arguments. If Blender is using the same function name internally then developers cannot change it either. If they want to add or change an argument, as is not uncommon, then it would have to be done with a new function name (e.g., add_vert_ex instead of add_vert, just to make up an example) and the whole of Blender code becomes less elegant and understandable because the good name’s meaning has been frozen. A variant of that is if the behavior of the function changes a bit (maybe for efficiency, the caller needs to guarantee some preconditions).

By wrapping the C function in Python, the Python name can stay the same while the implementation just adapts the arguments and preconditions to the changed function.

Even harder is that the data structures manipulated by the API functions have to stay the same. What if one wants to add things to or remove things from a struct? Again, a Python API can adapt the internal data into the API’s stable data structure.

Now, all of these things could be “fixed” in exactly the same way that we do the Python wrapping. Rather than succumb to the natural temptation to make the API be just a subset of the same functions and concrete data structures that Blender itself uses internally, one could define names and data structures that will never clash (maybe they all have API in their names, or something), and they always indirectly call internal Blender functions, and always translate, by copying if necessary, the internal data structures to the API ones. But by the time one does this, how much has really been gained over a Python interface, really? I think the reason people keep asking for a C/C++ API is that they want direct access to the internal Blender data structures (for speed) and they want access to the many more internal C functions than are exposed in the Python API. Both of those advantages mostly go away with the approach I just outlined.

Some of the adaptation would be nicer and easier if C++ were used as the API language, but the basic point of needing to adapt both functions and data remains.

There are other practical reasons why having a C/C++ API is something that most Blender Developers would prefer not to have.

• Linking the plugins with Blender can lead to all kinds of headaches – versions of expected libraries, name clashes, … These can be dealt with (other software does) but there is no doubt that this is an added burden for developers to build and maintain solutions for.
• Now the plugins can crash, something that largely can’t happen with Python addons, and this will annoy users and give more things that Blender Developers have to debug and take account of when trying to deal with user bug reports.
• Python can’t go away, so now there are two APIs that developers have to think about the stability of and deal with whenever they change internal behavior.

[StefanW]

@JuanGea

The only problem right now with C++ in Blender is that there is a big mess in the amount of things you have to modify to affect just one little thing, and that code is not properly documented, so you have variable names that you have to “guess” what does that mean, or have a meeting with the original dev so they can explain what those are doing, or dig during 3 days in the code until you find where that came from, and that’s not related to the language but to the code organisation itself.

This is off topic from the Python discussion, but a valid point. When you look at Brecht’s recent commits for creating a volume object type, it’s a crazy number of places that require little modifications. Now most of this is due to historical reasons and, with lots of patient work, could be refactored. In an ideal case, adding a new object type, modifier or node would be nothing more than subclassing an interface and registering a new class.

In my opinion, that kind of refactoring would be a much more productive use of cleanup fridays than spell-checking variable names. It would be a long term project, but I think in the long run it would contribute to readability, stability and flexibility. I know it’s tempting to pick all the low-hanging fruit, but I think it’s time to be ambitious and tackle some of the hard tasks.

[LazyDodo]

@StefanW

In my opinion, that kind of refactoring would be a much more productive use of cleanup fridays than spell-checking variable names. It would be a long term project, but I think in the long run it would contribute to readability, stability and flexibility. I know it’s tempting to pick all the low-hanging fruit, but I think it’s time to be ambitious and tackle some of the hard tasks.

Think the point of it to start was to clean the low hanging fruit that can be done in less than a day, i’m pretty sure that once all the low hanging fruit is gone one of two things will happen

1. people will take on as you suggest more ambitious projects that will take longer than a day.
2. nothing! back to business as usual, code cleanup day is canceled.

I’m hoping for 1, but honestly can’t rule out 2

[StefanW]

There will always be low hanging fruit like that, and everyone will avoid the big tasks.

Don’t get me wrong, I value code quality. But it would cost a developer of work to make the Blender code base conform to the Oxford dictionary and to compile with zero warnings at -Wall -Wpedantic -Wextra, and in the end the benefit for both users and other developer would still be next to nothing. Time would be much more wisely invested in doing things like what has been mentioned in this thread, such as stable, consistent and performant internal and external APIs.

[LazyDodo]

While agree that comments confirming to the queens English will have very little impact on the average end user, I do think that some of the things done yesterday will have indirect value to them, just sanitizing some of the function names like what was done in T73589 will make it easier for new developers to find their way in the codebase and start contributing patches which will make a difference for end users.

[StefanW]

In my opinion, function naming is not at all what’s keeping new developers from contributing to the code base. There are much bigger hurdles that new developers face, and I don’t see any of them on the cleanup list.

IMHO, a much better exercise would be to hire an intern with C/C++/Python experience but no idea of the Blender codebase and peek over their shoulder while they try do do things like:

• adding a modifier
• adding a new compositor node
• creating a new shader node that works in both Cycles and Eevee
• support a new file format
• create a new object type
• add a new gizmo
• …

I believe it should be apparent right away, what the big areas of improvement for the code base are. Anything to make fundamental tasks like that easier will help all developers, novice or experienced.

[LazyDodo]

@StefanW

• creating a new shader node that works in both Cycles and Eevee

One of the GSOC students actually did that last year, and documented it, it’s somewhat horrific to see how many things you need to edit.

[StefanW]

Exactly. Personally, I think reaching for low hanging fruit while avoiding the elephant in the room* is a waste of valuable developer resources. Making smart people do search/replace every Friday is not helping.

** Seriously. All this to add a single shading node, and the guide is even incomplete (no OSL, SVM or GPU code, just the boilerplate code) That’s what’s wrong with the Blender codebase, not that function naming is inconsistent.

[Josephbburg]

And… we’re off topic again!

[JuanGea]

I don’t think this is off topic at all, the PyPy / Numba implementation has been discarded, but also the conversation evolved into a very interesting one about how to improve developer experience with Blender, in both Python and base source, it’s very interesting and since this has been brought to the table since the beginning I don’t think it’s off topic at all.

[Alberto]

I think that in these cases they always try to look more at the problems generated by doing something than at the benefits.

I don’t think anyone expects that a new API is not extra work, even that it can complicate development in some ways. But we don’t seem to want to see the positive.

• Ease of development of tools by third parties
• Removing the entire bottleneck from the patches
• Remove the entire bottleneck of repairing patch problems
• Remove all the bottleneck from users by asking for new features that must fit into the blender puzzle
• Eliminate the bottleneck that prevents the implementation of new modifiers.
• Eliminate the need to compile a blender branch for anything
• Simplify the development model, allowing people with less knowledge to maintain third party features

What is the point of having an open source program where because a person, let’s say very purist, refuses for half a decade (literally) to implement a modifier that is already programmed as the weld for reasons of purism and internal bureaucracy? Let’s compare both development models

With a more powerful API in the hands of the users/third developers:

• Unknown user creates a simple weld modifier, it’s not perfect, but it works for people. It creates it in a week.
• People download it and use it, at their own risk.
  END (Time needed for one week of work by one person)

With the current model:

• Someone has an idea for a modifier and creates their code for the modifier. He creates it in a week.
• You must convince, for weeks (or months or years), some developer to review it.
• A programmer sees you as having structural problems
• Original developer reviews it
• Another sees performance problems
• Original developer reviews it
• One says the comments and the way it’s written is rubbish, let him fix it
• Original developer reviews it
• After the UI team decides that your implementation is not suitable for the program template, please change it because the usability is not as expected.
• After all this the developers decide to stop paying attention to the “boring guy” of the modifier.
• Users decide to start compiling their versions, mixing their diff with as many others.
• Every year someone critic the situation of the patch on the forums, which because the patch is ignored for months/years
• Developers begin to argue that the model works and the decisions were right
• After 200 post in the forum someone admits they’ll check it out
• In case the original developer is out there, maybe he will update and revise it. Otherwise the developer is said to have abandoned the project and nobody wants to mess with it.
  END (Time needed for from months, at best, to years of waiting. It took the time of 5-6 people, hundreds of users complaining, compilations, developers constantly interrupting their work).

This is an absolutely wrong working model from almost any point of view.

[Alberto]

And on the topic of the last comments, I wish, simply, that most of the developers would stop adding features to blender for 4-5 months (to say a range of time) and just fix everything that users complain about blender. No bugs, no new features, no magic ideas,… No, just pick up and say, come on, let’s do this

• solve all problems generated with blender2.8 or without complete
• solve performance issues when modeling
• solve the problems of the subdivision
• solving multiresolution problems
• solving the problems of the undo
• to solve the problems of heavy scenes
• And a few other things.

And no more “long term project” and have the developers to 20 different tasks. No, just fix that. And until none of that is fixed, nothing new is added or skipped from idea to idea unless there are simply resources left over that don’t make sense to dedicate to these points. It can’t be that we don’t know when the hell the subdivision is going to work properly. That we don’t know what year we’re going to be able to edit heavy meshes. That we don’t know when the undo will start working.

It’s just that nothing that you want to add matters as long as those problems aren’t solved.

What can’t be is that you want to make blender grow in things like “everything nodes” when it doesn’t even seem that anyone is clear that the code on which it will be settled is reliable or that it allows the performance that is logically expected from something like that.

[testure]

yeah… now we’re pretty far off topic. I actually had to double check to make sure I was in the same thread

[PaulMelis]

To get bit more back onto the topic, a lot of the points raised above about code complexity, development processes, C vs Python API, etc glance over one major point: what exactly should the Python (or C/C++) API support and to what extent? I.e. what types of extensions and tasks? An external render engine needs a lot more control through the API than a simple import script (which can just generate a set of scene elements). A new compositing operation would only need to take two images and produce a new output one. But an addon that wants to provide GUI elements hooks into a lot of internals.

The current Python API exposes many of the underlying code details, to pretty much give full control and flexibility at the cost of complexity and relative instability as the API will reflect any changes of the underlying code. But this does not have to be that way. One could provide a stable® API based on higher-level concepts that don’t directly map 1:1 to the current Blender implementation. For example, you could have import plugins pass a mesh in a fixed data structure (eg set of arrays) without having to know what the current implementation uses in terms of mesh structure. Conversion could be done in the API layer. For a restricted set of tasks you could provide such an API. At the expense of extra maintenance work and slightly less performance as you’re not working 1:1 on the underlying structures.

I guess it’s partly due to the fact that a lot of the Blender UI is written in Python that the API exposes so much of the internals. Not because this was the best way to provide extension of Blender by external developers, maybe that was a bit of a by-product?

Fun fact: there used to be a time when you could write a sequencer plugin in C (for ancient versions of Blender)