Improving Python/C/C++ and general development process

Please don’t take me the wrong way. You know I’m friendlier than that :wink:

What I meant to say is that there is, at least in the way that I look at it, a big difference between teaching people the basics of Blender scripting with Python and teaching someone all of C++ to a high enough level to help with Blender development. With just very limited knowledge of Python and copy-pasting some operators from Blender’s UI you can already create a script that makes your life easier. Taking this up a notch to understanding things a bit better and getting a bit more complex, that’s what I’ve done with Scripting for Artists. My point is that teaching people C++ from the same starting level as Scripting for Artists, or even experienced developers in non-C/C++ languages, is beyond what I can do next to my usual development work. Instead of basing themselves on whatever I can hack together in what little time I have available, I’d rather have people go to get proper education in the field from an appropriate educational institute (like a university). That was my point.

Of course there is a huge difference between that and an introduction into the structure of Blender for people who are already familiar with C and C++. We may do a training video series for that, maybe not, I don’t know. I think that better written developer documentation (and making sure it stays up to date) will take less time and be more useful for a wider audience.

4 Likes

But I’m not selling to anyone that I do an Open Source project, that I systematically refuse to make an API in C++ so that people can implement their ideas freely because I want it to be in the form of patches, and then I ignore the patches.

It’s quite insulting that comment about " smart people", I don’t know if you think that because we are artists we are not smart or we are not able to understand how a development works. I don’t know, do I have to show you my results on an intelligence test? Since you seem to need it, I studied an engineering degree.

1 Like

@Alberto I think you missed the part where I agreed with you about how patch review should be. I also said I don’t like the current situation.

Any insult you may read in my text is in your head, not mine. Saying that developers are smart does NOT imply anything about anybody else.

About a C++ plugin framework, this has been discussed in plenty of other places already, so I won’t reiterate that here.

On a final note, everything I write and have written are my own opinion. I’m not a spokesperson for the Blender Foundation, nor so I speak for any other developer. I only tried to explain why I think things are as they are. There is plenty of stuff I’m also not happy with, and I’m doing what I can to improve this.

1 Like

In case the case against the two libraries has been agreed upon, the title makes it an open ended question: good coding practices. Although T70016: Document performance profiling Blender and an equivalent in wiki about profiling python catch attention now.

Ok, I can understand part of your reasoning.

But then I don’t understand this:

If you want to flip around things then we need better possibilities to be able to modify Blender to extend it’s capabilities, so or we have better accessibility to code, or we have better tools and performance out of python, but we need a production ready tool set that allows to generate tools that will full-fill the needed areas, like a proper scatter tool that works properly with a LARGE amount of objects, something that right now it’s half possible because of several limitations, and a viewport visbility mode that will ease the work with such scenes.

And TBH I totally disagree with you about going to university to learn C++, and you have there a co-worker that is also one of the most pro-efficient developers right now that says that university was kind of useless for him, C++ is not so complex at all, I know Objective-C and it was easy to learn thanks to my previous knowledge of different script languages, after that I learned Java and C#, later Python, and it’s just a matter of studying it, C++ is more of the same thing, and even @Alberto also says this, and he went to the university for such a degree, and it’s not the only one.

Also there are people that cannot go to university for many reasons, not enough money, not enough time, family matters, personal matters, but even with all that they want to learn, they want to contribute, they have awesome ideas, they have the will, that “go to university” thing is just discouraging them from improving themselves and being more proactive, to be honest even is disheartening for me because as I said, I was not expecting that, the problem are not the words, but the thoughts behind them, only “university” guys can teach/learn C++.

The only problem right now with C++ in Blender is that there is a big mess in the amount of things you have to modify to affect just one little thing, and that code is not properly documented, so you have variable names that you have to “guess” what does that mean, or have a meeting with the original dev so they can explain what those are doing, or dig during 3 days in the code until you find where that came from, and that’s not related to the language but to the code organisation itself. I can understand that it’s as it is, no argument here, but a better explanation, walk through and/or documentation could be very welcome, because sometimes only the original creator of a module is able to modify the module, just because there is no proper documentation on the code, and I received that answer when I spoke with a module owner to analyse the options we had to hire an experienced developer to make an implementation, and the answer was honest, it’s better to not do it, right now only the owner of the module can really touch it without trouble, and I won’t mention who he was or what was this about, just that this was the situation, and I’ll always be grateful to the dev for being clear on the topic.

I don’t care about having a C++ API or not, if there is not going to be a C++ API, good, I don’t have a problem, but every time we say Python performance is BAD (because it’s BAD when you try to do anything half-big, like the importers) I don’t want to receive the answer about “modify Blender code”, because hey, I’m all about it, but then give the resources to understand Blender code, because even an experienced C++ programmer is not able to properly follow and understand Blender code to do small fixes, and I’ve read Blender code TONS of times.

Also we need to do tools for production in the same way anyone can do those tools with proper performance for Max/Maya/C4D, why? because we are competing with them, we live in the same market, we fight against them for projects, we support Blender and the Blenedr Foundation, Blender Institute and Blender Cloud so Blender can support us, exactly the same as the Blender Animation Studio team I suppose, some tools will be available as addons, with or without Blender, but we need to solve problems, and for that Python, if it’s the only option to generate such tools, has to have the same performance as C++ or at the very least similar performance, otherwise the thing is that Studios are purposely being limited on what they can do without modifying Blender code, which BTW is hard to modify because all what I said.

I did not liked that you went out with the “go to university” thing, and it felt personally bad, may make sense or not to you, but to me it was disheartening as I said, we are supporting open source, free software, freedom of knwoledge, and for that we have to support free knowledge, as I do for 3D in my channel, even if we can do also paid courses, because everyone has a rent to pay.

Personally you don’t have to do the course if you don’t want to of course, I understand that and support you if you say “I don’t want to create that course” for the reasons you said or for no reason at all, but if you want to make more people contribute, more developers being excited about helping with Blender, with bug fixing, then IMHO the “university” answer is plainly wrong.

Sorry to be so hard at you, but it really reached my emotions.

5 Likes

I’ve been following this discussion a bit, but the above kind of sums up the Python vs C++ debate for me. Experienced programmers will (of course) be able to learn C++. But while someone who has never coded has a good change of being able to pick up Python, C++ is simply a step too far as a first language. Your prior programming experience matters at lot. The current Python API is fairly easy to get into, once you get used to all the quirks, which makes it more accessible to a lot of people. And yes, don’t go to university to learn coding :wink:

Python performance is only bad when using pure Python to do all kinds of computations or data processing. The reason the importers are slow is that they do all parsing of text input in Python itself, instead of calling into a C/C++ library (like assimp) to do the heavy lifting. That’s why things like numpy exist, Python merely becomes the glue for specifying the high-level operations, the actual number crunching work is done outside of Python.

Having a C++ API would only make it somewhat easier to couple C/C++ code to Blender. The current Python API already allows that, cf. the comments on Cython earlier. Plus, if you know what you’re doing (i.e. experienced C++ programmers) you can use pybind11 or Boost.Python to create a Python module for a piece of C++ that is fast but can also be called from a Blender Python script.

But I think there’s currently a few things suboptimal in the Python API when it comes the above:

  • Integration with the Python buffer protocol isn’t fully there, so passing large chunks of data (which should be a cheap pointer copy) is sometimes very slow as the data itself is copied. This hurts the overall performance of the Python API. See Pass a render result as a numpy array to bpy.types.RenderEngine posted today, or Bpy.data.images perf issues. These’s even a patch in the latter thread to vastly improve image copying performance.

  • Building a Python extension module that works with Blender is not as easy as it can be (e.g. the missing headers noted in Including Python headers in distribution? by myself, challenges in building the module, how to properly install new Python modules in Blender’s environment, etc.). Improving this might help in attracting C++ developers that can provide optimized modules for certain tasks without having to know the Blender C++ code itself. Merely having a good API (the current Python API) is enough. Case in point: my PLY importer for Blender, written in C using a C library, but available as a Python library. But turning this into a true add-on is then not that straightforward again, as it requires the user to compile the source (although I haven’t looked if there’s general instructions for distributing Blender add-ons containing system-specific binaries, like a Python module).

what about an external merging software, were the users can merge their own .zip file (of C code) to the main build, then automatically compiling to create their own custom builds of various implementation? it would look like a C++ Plugins from the user point of view.

all this within a sexy GUI for users?

we can chose a clear name for this kind of addition to make a clear distinction between Python addons and “External Merge Parts”

a bit like what they are doing with the nexus mod community.
it can be done, no ? the software just need to merge lines of codes and keep track of them.

I guess you see extra benefit for this approach because it doubles as a distribution point for addons? Because I would always want to locally build the addon during development anyway, I wouldn’t use the online environment for that (it slows iteration during development).

But it’s an interesting idea.

1 Like

it’s true that this software can become a more general “package manager” that can manage “merge parts”, addons, and why not libraries too.

I think one of the targets is to avoid the general idea of “package manager” as max turned out, and I agree with that view, because it’s dangerous, and the idea is to try to merge as much as possible in Blender and make addons a side-tool, not part of the main workflow with Blender, that’s the major problem with max for example, max without plugins is nothing.

But at the same time this presents some other problems, so the thing is that a middle ground could be great, that’s why I think Python is ok, but it needs improvements to make performance visible.

@PaulMelis if that’s the case (which I don’t doubt at all) why the official OBJ importer or the official FBX importers are so slow and the Gsoc project was suspended becasue it was using Boost too much? (not sure if that’s the reason exactly, is what I remember from the reason the student said he was given, also don’t know who was the mentor).

And the other thing is that it’s very hard to keep pace of state of the art, and for example Cycles is kind of stagnated, having custom AOV’s just now, Ligh Groups still waiting, and no other performance improvements for years it’s a problem, because the only improvements awaiting review right now are:

  • Adaptive Sampling: that helps in specific cases, not always, and it’s not super dramatic difference if it’s not one of those cases

  • Scramble distance, that is there since 2016

  • Dithered Sobol, that is also there since 2016

So if the last two get implemented Cycles will see kind of a speed up that we’ve been using, but it’s a speed up with old technology, something that could have been there since 2016, and there are no plans to improve this, not earlier nor faster, and that’s now with the current funds, that I could understand that the funds are limited, and are still limited, so it’s a must to be wise in choosing fights, but then… we need to be able to have alternatives that fill the gaps left out, and that’s the middle ground, that’s why Python has to be improved, because if we need a proper alternative to Cycles, we need it working at full speed for viewport rendering or for final rendering, and there are some limits created by Python/Blender right now, no matter how experienced is the actual dev of such engine.

And I’m thinking of LuxCore, they are doing an amazing job with the addon, right now it’s as fast as Cycles to export millions of instances, but there are some operations that could not be done faster because of Python limitations.

We’ve been forced to switch to LuxCore for our ArchViz work because we could not get good interior renders in a reasonable amount of time with state of the art hardware, we could not afford to not have caustics nowadays, the clients require that, and we have nothing in cycles about that, and there is nothing planned, and so on, and what pisses me off more is when I receive the answer “we don’t need that in the studio, so we may not do it”… when there is donation to the. development fund I hope the development direction is not handled towards the interest of the Animation Studio, but towards the general interest and to keep Blender as an amazing state of the art tool.

Don’t take me wrong, I love cycles, I just know it’s limitations right now for some type of work, I would love to not have those limitations, but it’s normal, but if it’s normal then we need to be a bit more open minded towards other options that could be used together with Blender, and I’m not talking about closed source software, just to be clear.

So I think the key is to be in a middle ground, Python is IMHO the best way to go so anyone can improve Blender, C++ would be amazing (without API, don’t care, I mean contributing to master source), but it needs more knowledge of course, but if we want anyone contributing we have to encourage that in both ways, Python and C++, not just one.

I like the analysis of python you did @PaulMelis

Now the @BD3D merge thing… I dont’ understand that too much, but I think that we should avoid the “maxification” of the Blender environment :slight_smile:

Also @BD3D that there is GPL, so I think that will help avoiding that “maxification”, and that’s why I like GPL, that’s why there is no reason to have fear to addons, and that’s why I defend that middle ground, addons yes, but we need performance, in-code improvements h*ll yes, middle ground.

I want to clarify that in this case I refer only to the devs from the Institute paid by the Dev Fund, not to the devs paid by the Blender Cloud, which I assume will look for the best interests of Cloud customers and pipeline work for the studio as it’s stated as main goal (always for the best interest of Blender in general, but different priorities).

6 Likes

If the core idea of blender is that it’s a software build by its users, then we should give more power to its users. (duh)

Right now people who have other visions / visions that aren’t compatible with others, are simply forgotten In the dark, end users may never experience their ideas.

If a dev want to share his (forgotten) vision for blender he should have a platform somewhere to share it,to show it to everyone. GraphicAll and the blender market is a good start. But it need to be something much bigger than that imo.

If such platform doesn’t exist, « visions » will compete with each other’s and users will need to choose sides, which is not a good thing (See E-Cycles builds vs other vs main ect…). That’s why transforming builds into « pieces of puzzles » that can stack-up is a good solution for everyone imo. Even For the main devs. As they can see which vision is appreciated by the public and merge them.

In the end we always end up talking about the same problems that have their origin in the same place. C++ API

  • Patches that are not implemented because of the bottleneck in the reviews.
  • Patches that are not implemented because of some negative review due to internal policy
  • Patches that are not allowed because they do not have the necessary code quality
  • Complexity of the blender code
  • Addon performance without complicating the developers
  • To allow the development of modifiers freely and with good performance
  • Slow blender core development

All these problems and many others that always come up every little time have only one origin, the limitations that arise due to the blender plugin model.

But since this is something you never want to discuss seriously…

3 Likes

That’s why I’m proposing a more libertarian solution that will entirely remove the bottle neck and will reach the end user for sure no matter what

Sorry if some of your response was not attributed to me but there’s a few things I don’t understand (was it ? I’m confused)

What’s the link between an external code merging software and python plugin ? As I understand, the limitations can be removed with a « C » plugin

Me or Juan ? It’s the first time I’m having this discussion I don’t understand

The developers and foundation

1 Like

Oh @BD3D , you bet I discussed this with @Alberto when we met for Blendiberia 2019 hahaha :slight_smile:

2 Likes

I have to say that I think that I took @sybren a bit by the wrong path, and he didn’t meant to say what I understood, so I want to publicly say sorry to him, I’ve been a bit too hard on him, he is a nice guy, that cares, and maybe the “university” thing was not what I understood andhey, if he don’t have time to do the course, he should not do it at all!!! there are other people hre that can create such course if they want… step ahead people!! hehehe

I just want to say that I’m sure that everyone can learn C++ with a bit of effort and a pair of books/courses, and that everyone, specially the ones with a programming background in other languages can improve themselves and get to a point where they can contribute to Blender, as anything it will take effort and time, but I want to encourage all of you that wants to learn and to contribute, please don’t hesitate and contribute, I’m sure review process and implementations will improve (right @dfelinto :wink: ) , and you can count on me and with our build to test out things for the general public :slight_smile:

5 Likes

Can we get back on topic? Python hasn’t been mentioned in quite a while. We all know there’s not going to be a C/C++ API, it’s listed as an anti-feature. This thread isn’t about that. I’m interested in discussions on how to improve the Python API that we already have.

1 Like

I’m pretty sure C/C++ is not the issue, like any language it can be learned with a bit of effort and good online resources. What is hard to learn is the specific architecture of a big project like Blender, and it would be true even if the soft was made in another language. Lot of modules, interdependant parts, legacy code, etc. Its true for Blender but also any software of that size. And it’s also not stable in time, big parts of the architecture can be completely changed in the future, so any course on that would become obsolete at some point (not to say it is useless to create one, but it has to be maintained at least).
On the opposite, an API is generally stable for some years. And you can extend the software without understanding of the whole architecture. When I started digging in Blender few month ago, I agreed that since the soft is open source, a C/C++ API is not required. But now I think having to change the core code base to implement efficiently some functionalities that could be in an addon is a big overhead, in terms of time spent understanding the existing code, time implementing the feature, and time spent by reviewers checking and integrating the changes. It does not scale very well.
People mentionned that is is possible to implement things efficiently in python, but it generally requires implementing python modules in C/C++, and if you want to manipulate Blender’s data you have to do some unclean things anyway. I see that as a workaround.
So I must admit I don’t understand the choice of not having a C API. Since there is a Python API that basically wrap C functions calls, why not just doing the exact same in C ?
And I wonder if someone outside of the BF implements it, would it be accepted in the main repository (assuming the author accepts to maintain it) ? Because if not, for political/philosophical reason, I see that as a danger for Blender. If at some point many animation studios end up using Blender, and have a real need for a C API, they might choose to support a fork that provide one instead of the original. That’s not the case right now but IMHO it is something to think about.

Now when the goal is to do some quick hacks/fixes in Blender’s code base, indeed knowing C/C++ is required (but not at a high level, the code remains quite easy to read in my opinion), and I find a lot of values in all the differentials https://developer.blender.org/differential/. Just reading some of them gives a lot of clues about how to do things. I’m pretty sure it would be interesting to have tutorials that just explain how to implement some specific features with the associated differential as support material.

Sorry @Josephbburg for continuing the discussion on that ^^’ I wanted to give my opinion and there is definitely some interesting things to discuss in this thread.

2 Likes

It’s worth mentioning that Maya actually has a Python wrapper for the C++ API (well, two actually), and no less than two Python wrappers for its MEL API. Every other major DCC has been in the process of replacing their built-in languages with Python, or at least wrapping them in Python. The trend in the industry is towards Python. Hell, the trend in C++ is towards Python, too! Now Blender is the first and maybe the only DCC app that supports Python natively in a Pythonic way. Just try writing a Maya script with Python- its completely backwards syntax makes it a wrestling match with your own instincts.
I think the answer to the C/C++ question is to improve the Python API. The bpy_struct.as_pointer() method should be sufficient to pass the data to a compiled module that operates directly on the memory address within Blender. Perhaps more work needs to be done to make this process less painful, but Python isn’t too slow to query memory locations and pass int’s back and forth.

I’d like to see a minimum version of Blender or a more officially supported bpy build for compiling C/C++ modules against to be used in a Python ctypes plugin/interface. I’d love to see it implemented in the Apache license like Cycles, to attract developers that wish to make money from the software. Although I’d also love to see it enforce software freedom via the GPL2, but it’s a hypothetical anyways… Perhaps this is easy to do already, in which case documentation is what is needed.

On the opposite, an API is generally stable for some years

But at what cost? Take a look at some of the “industry standard” softwares- what you’ll find is bloated, buggy, slow softwares that have to remain compatible with decades-old APIs. Take Maya as an example- its node-based architecture and extensive MEL API were groundbreaking in the late 90’s when Blender didn’t even have undo support yet. But MEL hasn’t aged well, and Maya is too deeply wedded to this extremely poorly designed language to refactor. Despite being an object-oriented C++ program, Maya’s entire interactive structure is based around a strictly imperative language with only five data types-- and barely any ability to handle exceptions or write structured programs. To be fair, this isn’t a C++ API, but I’d be willing to bet Maya has the same problem in its C++ API. Having to keep compatibility to an API is limiting for the future of a project. That’s why we see such slow development in the proprietary world- they didn’t plan for the future and now they can’t take big steps without losing their customers by breaking compatibility. It also underscores the folly of writing software in a proprietary language/API, but that’s off topic.

One of my favorite things about Blender is the bravery of its development cycle- we’re not afraid to change the architecture of the software in a major way if it leads to improvements. We’re not afraid to spend months or years designing things to be sure we won’t have to refactor again anytime soon. Unlike proprietary tools, we don’t have to worry about losing customers, meeting yearly feature release deadlines, or pleasing shareholders. Free software means that we’re free to focus on the future of Blender, rather than the short-term. There’s no risk of going out of business. Autodesk can’t buy us and shut us down, like they did with SoftImage.
Well… I’ve gone off topic myself now.

4 Likes

I’m not in a position to comment on how the API itself could be sped up, but I do think that a broader set of utility functions could reduce the amount of information that needs to be passed back and forth through it. There are few I’d love to see for the gizmo system, at least.

1 Like