Improving Python/C/C++ and general development process

Many speed disadvantages in Python can be gotten around with clever, Pythonic programming and the use of efficient libraries like numpy. The only bottlenecks I’ve ever run into (that weren’t my fault) were interaction with Blender, e.g. bmesh and accessing a very large armature object, or situations where a call to bpy.ops was necesary. What I’d like to see is a bit of profiling to see which interactions are slow, and improve those. I’d like it if there were fewer situations where bpy.ops was necessary in an otherwise pure bpy.data script.
If you want to add new functionality, I agree, doing it within Blender is probably the best way. But it’s also possible to pass Python objects and functions to/from C with ctypes. I’d like to see some guides or documentation on the right way to do this with Blender. I think it could be especially helpful to people trying to create GPL-compliant bridges to other softwares that are themselves not free (as in free speech). And while I’m personally unhappy with the mere existence of any non-free software, it would be beneficial for Blender and Blender artists to be more integrated in the proprietary software world without compromising the integrity of the GPL. Perhaps Blender can be more friendly to this kind of a workflow? What are the challenges to working this way already? I haven’t tried it myself.

7 Likes

@sybren

Hey hey hey! Don’t skip the bullet!

That “C++ for TD’s” course is a thing… we want you on that… even more than how much we want you improving Alembic and USD… and we want that a lot… :sunglasses:

I totally understand the PyPy thing, and I also understand that it may not be worth the effort, specially with Function Nodes coming, many things will be possible with nodes and the performance will be better than Python for sure.

But also I don’t think C++ to modify Blender is so hard to teach/learn, If someone knows C# or Java or Python, it’s not so complex to learn C++ itself, it’s just a bit of syntax change and having to understand a pair of concepts that are slightly different (do you know how much time I’ve been wondering what a Vector was until some simple book told me that it was basically the same as an Array?! ) and that made with an actual practical course with some targets, like hunting fixing some simple bugs, or improving some already existing tool.

So I like your idea of focusing more in the C++ side of things, because the other side is going to improve (we still lack an idea to give function nodes some interface, but that will come with time) but we need some documentation or even better, a small course that can bring Addon/Script developers near C++ side of things, and yes, things can go into custom builds, or maybe not the whole feature would be accepted, but some interesting and useful patches can be extracted from that full feature, there is a long etc of possible situations, but I’m all into making people understand the C++ side of things much more.

Right now it’s too difficult, even for experienced C++ programmers, the structure of information in code it’s super specific (which is normal) and the documentation available scratches the surface, so a TD cannot go directly into the C++ code and modify improve things easily, a TD or a TA are part artists part programmers, and some course in the same fashion to the one you did could be very beneficial in general :slight_smile:

@Josephbburg
Regarding the proprietary software side of things, I don’t really think being more into the proprietary world is a good thing, with that I mean, being inside any of the 3d indsutries that use proprietary software is good, complying with them is a whole different story, and the trend I think is to do things more and more open and free, specially for bridges, so I relaly don’t think it’s a need.

But with that said it’s true that in Blender some things evolve at a slow pace and allowing other developers to chime in and give solutions that don’t require custom builds it’s a good thing, because for example Cycles is a bit stagnated, and I know some new object types are being developed, and that’s super exciting, but performance wise and render engine feature wise is stagnated, I won’t mention all the things because there are many threads about that :slight_smile: and it’s clear that more funding can help, but since there is no clear way of stablishing priorities we cannot wait 2 years until we get more rendering speed with som GI cache because now we are getting a point cloud object, don’t take me wrong, it’s super hard to stablish priorities and it’s normal that internal devs have their own priorities, but if we need a third party render engine we need it at full speed, LuxCore is doing a super good job on that using as less python as they can and as much C++ as they can, but even with that there are situations that are simply limited by Python being slow, and that’s the part we need a solution for, the parts of Python that are slow, and when you work with big scenes (actual production scenes) even renaming objects in batches or creating new objects is slow.

So IMHO it’s a matter of giving solutions so addon developers can do their thing, because I doubt 5 different render engines can get accepted in master, no matter if they are open source like Lux, and also maybe developers don’t want it INSIDE master, they want to maintain an addon development, and I think that’s something that should be respected too, there is no need to make Blender a mamooth with ALL the tools in the industry, that’s why Addons are great, and we need them, but we need them with high performance even with complex scenes :slight_smile:

And while we can hire a developer, the true thing is that not all of us can hire a developer that charges proper hourly rates, the idea of “fund it yourself” don’t always work (sometimes work :slight_smile: ), and some developers simply have their own render engine or their own addon and want to develop it in a different way :slight_smile:

My 2 euros here (because all this is a bit more than 2 cents hehehe)

1 Like

This is the best approach IMHO. There are many things that Python scripts can do, and there may not be a one-size-fits-all approach to fix performance problems. Identifying the bottlenecks in a handful of use cases could allow us to get improvements with limited effort.

6 Likes

I’ll be completely honest here, one of the things that prevents me from contributing are things like this just sitting there tagged with needs review and TONS of community support:
https://developer.blender.org/D5336
https://developer.blender.org/D4786

These are just two examples off the top of my head- things people have asked for repeatedly, somebody put in the effort to do it, and they just sit there needing review for months. All the while the author has to keep up with merging back to master to make sure their code can be reviewed. At some point they just give up and it withers on the vine.

I understand that there are reasons, limited reviewer time, etc- but for someone like me who could help it’s my prime motivation for not helping. It’s for reasons like this that people like myself depend on a robust python api, because we don’t have the time to maintain a fork of Blender or babysit a diff waiting for reviewers for 6+ months with no sign of that ever changing.

Seriously, when I open up differential and sort by creation date (oldest) with a status of “needs review” and see hundreds of patches going all the way back to 2014 that people wasted their time working on because nobody ever even looked at it, my motivation drops to zero. If some of these diffs with lots of support can’t even get eyes on it, what chance does any change I make have?

18 Likes

I’m not saying it’s right, but I found that sometimes you have to ask for reviews from the devs rather than wait for them to happen. Just hanging out on chat going "hey xxx if you have 5 minutes can you take a look at Dxxxx` once in a while does wonders.

Personally I pick my battles, something that doesn’t matter if it lands today or next month: meh it can wait, reviewers will get to it when they get to it. If it’s something I desperately want in 2.82 before the window closes i chase the devs to get a review.

Don’t get me wrong, things could definitely be better, but things are not nearly as bad as you imagine they are.

8 Likes

@testure damn i’m following this from a year, i was thinking the project was just dead, but apperently it’s just a review issue ?


4 Likes

Don’t you think the patch review should be objective, aseptic and not dependent on the knowledge of blender’s internal bureaucracy?

1 Like

Yeah, that’s the part where i said

1 Like

I’m just saying that to someone like me it looks like an uphill fight to try to make any headway on contributing to Blender, and hopefully give you guys some ‘third party insight’ into why someone with development experience like myself would actively avoid making contributions to Blender. As you say, it may not be as bad as I imagine it, but I’m not the only one imagining it and certainly not the only one who does not submit patches as a result.

I’m not trying to derail the thread- i’m just providing a counter-point to @sybren’s comment about more developers always being welcome and that we should just introduce our ‘wildly new behavior’ directly into Blender.

Personally I think the Python API is already pretty fantastic, with some minor exceptions (as @Josephbburg pointed out) where api access doesn’t exist and we have to fallback to bpy.ops, but I find those situations few and far between.

3 Likes

I don’t think it’s good enough, an example would be the OBJ importer or any other importer/exporter written in Python, performance is awful.

I’m doing a python fracture, and performance bottlenecks are bpy.ops themselves. Depending on the number of chunks it takes longer for the operator to duplicate or the bisect operator.

By the way, to duplicate, with the “bpy.ops.object.duplicate()” operator is much slower than duplicating in this other way:

new_obj = obj.copy()
new_obj.data = obj.data.copy()
bpy.context.collection.objects.link(new_obj)

I’m trying to improve performance with cython but I don’t know much about it yet, yet the bpy.ops no matter how much cython I use I don’t think it’s going to improve performance just by using cython.

1 Like

I think the Code Quality Days are going to help a bit in that respect. Cleaning up & clarifying Blender’s code will benefit all developers, and that of course includes TDs. Personally I really welcome this, I’m sure I’ll have fun cleaning things up.

Learning the language C++ can be pretty hard, but I don’t think it’s up to the Blender Foundation or Institute to provide training for this. Go to university, get a degree in Computer Science.

a TD cannot go directly into the C++ code and modify improve things easily

I think this is the case for any program of the size, age, and complexity of Blender.

Yes, that’s frustrating. That’s also why there now is the Tracker Curfew to get such contributions either into Blender or rejected for clear reasons. At least there should be an answer, and not hang in limbo like that.

I agree with @LazyDodo though, in that it really helps to poke developers on Blender Chat and make sure that things stay active in their minds. With as many open patches as there are now in the tracker, it’s easy to overlook some.

Yes I do. But have you ever seen a collaboration of dozens of people working together, and then as an outsider tried to influence them? It always helps to know how those people are organised, and which paths are the smoothest to walk.

Just know that things hanging in limbo isn’t because the devs want it to be that way. It’s a side-effect of smart people having to do very difficult things that require all of their brains, just to do their development jobs. On top of that comes planning for future features & redesigns, decoding & fixing bug reports, and doing patch reviews. One thing we’re also working on is improving the code quality and way of working, so that less bugs are introduced, freeing up time to do more of the rest of the work.

3 Likes

You know that is as pedantic as it can be right?

It’s not up to the blender foundation to teach people Python but you made a course on it on the blender cloud, if the difficulty is the problem, then maybe the animation course should not be possible either, better go to the university and get someone teach you body mechanics, anatomy and many many other things.

I would have never expected that answer, but hey, don’t worry, no need for that course, I’ll tell many programmers I know that they should stop working and go to university, like the RedShift addon programmer or many others, because C++ is hard to learn and we should go to university, because is hard to learn C++

Duly noted.

4 Likes

Please don’t take me the wrong way. You know I’m friendlier than that :wink:

What I meant to say is that there is, at least in the way that I look at it, a big difference between teaching people the basics of Blender scripting with Python and teaching someone all of C++ to a high enough level to help with Blender development. With just very limited knowledge of Python and copy-pasting some operators from Blender’s UI you can already create a script that makes your life easier. Taking this up a notch to understanding things a bit better and getting a bit more complex, that’s what I’ve done with Scripting for Artists. My point is that teaching people C++ from the same starting level as Scripting for Artists, or even experienced developers in non-C/C++ languages, is beyond what I can do next to my usual development work. Instead of basing themselves on whatever I can hack together in what little time I have available, I’d rather have people go to get proper education in the field from an appropriate educational institute (like a university). That was my point.

Of course there is a huge difference between that and an introduction into the structure of Blender for people who are already familiar with C and C++. We may do a training video series for that, maybe not, I don’t know. I think that better written developer documentation (and making sure it stays up to date) will take less time and be more useful for a wider audience.

4 Likes

But I’m not selling to anyone that I do an Open Source project, that I systematically refuse to make an API in C++ so that people can implement their ideas freely because I want it to be in the form of patches, and then I ignore the patches.

It’s quite insulting that comment about " smart people", I don’t know if you think that because we are artists we are not smart or we are not able to understand how a development works. I don’t know, do I have to show you my results on an intelligence test? Since you seem to need it, I studied an engineering degree.

1 Like

@Alberto I think you missed the part where I agreed with you about how patch review should be. I also said I don’t like the current situation.

Any insult you may read in my text is in your head, not mine. Saying that developers are smart does NOT imply anything about anybody else.

About a C++ plugin framework, this has been discussed in plenty of other places already, so I won’t reiterate that here.

On a final note, everything I write and have written are my own opinion. I’m not a spokesperson for the Blender Foundation, nor so I speak for any other developer. I only tried to explain why I think things are as they are. There is plenty of stuff I’m also not happy with, and I’m doing what I can to improve this.

1 Like

In case the case against the two libraries has been agreed upon, the title makes it an open ended question: good coding practices. Although T70016: Document performance profiling Blender and an equivalent in wiki about profiling python catch attention now.

Ok, I can understand part of your reasoning.

But then I don’t understand this:

If you want to flip around things then we need better possibilities to be able to modify Blender to extend it’s capabilities, so or we have better accessibility to code, or we have better tools and performance out of python, but we need a production ready tool set that allows to generate tools that will full-fill the needed areas, like a proper scatter tool that works properly with a LARGE amount of objects, something that right now it’s half possible because of several limitations, and a viewport visbility mode that will ease the work with such scenes.

And TBH I totally disagree with you about going to university to learn C++, and you have there a co-worker that is also one of the most pro-efficient developers right now that says that university was kind of useless for him, C++ is not so complex at all, I know Objective-C and it was easy to learn thanks to my previous knowledge of different script languages, after that I learned Java and C#, later Python, and it’s just a matter of studying it, C++ is more of the same thing, and even @Alberto also says this, and he went to the university for such a degree, and it’s not the only one.

Also there are people that cannot go to university for many reasons, not enough money, not enough time, family matters, personal matters, but even with all that they want to learn, they want to contribute, they have awesome ideas, they have the will, that “go to university” thing is just discouraging them from improving themselves and being more proactive, to be honest even is disheartening for me because as I said, I was not expecting that, the problem are not the words, but the thoughts behind them, only “university” guys can teach/learn C++.

The only problem right now with C++ in Blender is that there is a big mess in the amount of things you have to modify to affect just one little thing, and that code is not properly documented, so you have variable names that you have to “guess” what does that mean, or have a meeting with the original dev so they can explain what those are doing, or dig during 3 days in the code until you find where that came from, and that’s not related to the language but to the code organisation itself. I can understand that it’s as it is, no argument here, but a better explanation, walk through and/or documentation could be very welcome, because sometimes only the original creator of a module is able to modify the module, just because there is no proper documentation on the code, and I received that answer when I spoke with a module owner to analyse the options we had to hire an experienced developer to make an implementation, and the answer was honest, it’s better to not do it, right now only the owner of the module can really touch it without trouble, and I won’t mention who he was or what was this about, just that this was the situation, and I’ll always be grateful to the dev for being clear on the topic.

I don’t care about having a C++ API or not, if there is not going to be a C++ API, good, I don’t have a problem, but every time we say Python performance is BAD (because it’s BAD when you try to do anything half-big, like the importers) I don’t want to receive the answer about “modify Blender code”, because hey, I’m all about it, but then give the resources to understand Blender code, because even an experienced C++ programmer is not able to properly follow and understand Blender code to do small fixes, and I’ve read Blender code TONS of times.

Also we need to do tools for production in the same way anyone can do those tools with proper performance for Max/Maya/C4D, why? because we are competing with them, we live in the same market, we fight against them for projects, we support Blender and the Blenedr Foundation, Blender Institute and Blender Cloud so Blender can support us, exactly the same as the Blender Animation Studio team I suppose, some tools will be available as addons, with or without Blender, but we need to solve problems, and for that Python, if it’s the only option to generate such tools, has to have the same performance as C++ or at the very least similar performance, otherwise the thing is that Studios are purposely being limited on what they can do without modifying Blender code, which BTW is hard to modify because all what I said.

I did not liked that you went out with the “go to university” thing, and it felt personally bad, may make sense or not to you, but to me it was disheartening as I said, we are supporting open source, free software, freedom of knwoledge, and for that we have to support free knowledge, as I do for 3D in my channel, even if we can do also paid courses, because everyone has a rent to pay.

Personally you don’t have to do the course if you don’t want to of course, I understand that and support you if you say “I don’t want to create that course” for the reasons you said or for no reason at all, but if you want to make more people contribute, more developers being excited about helping with Blender, with bug fixing, then IMHO the “university” answer is plainly wrong.

Sorry to be so hard at you, but it really reached my emotions.

5 Likes

I’ve been following this discussion a bit, but the above kind of sums up the Python vs C++ debate for me. Experienced programmers will (of course) be able to learn C++. But while someone who has never coded has a good change of being able to pick up Python, C++ is simply a step too far as a first language. Your prior programming experience matters at lot. The current Python API is fairly easy to get into, once you get used to all the quirks, which makes it more accessible to a lot of people. And yes, don’t go to university to learn coding :wink:

Python performance is only bad when using pure Python to do all kinds of computations or data processing. The reason the importers are slow is that they do all parsing of text input in Python itself, instead of calling into a C/C++ library (like assimp) to do the heavy lifting. That’s why things like numpy exist, Python merely becomes the glue for specifying the high-level operations, the actual number crunching work is done outside of Python.

Having a C++ API would only make it somewhat easier to couple C/C++ code to Blender. The current Python API already allows that, cf. the comments on Cython earlier. Plus, if you know what you’re doing (i.e. experienced C++ programmers) you can use pybind11 or Boost.Python to create a Python module for a piece of C++ that is fast but can also be called from a Blender Python script.

But I think there’s currently a few things suboptimal in the Python API when it comes the above:

  • Integration with the Python buffer protocol isn’t fully there, so passing large chunks of data (which should be a cheap pointer copy) is sometimes very slow as the data itself is copied. This hurts the overall performance of the Python API. See Pass a render result as a numpy array to bpy.types.RenderEngine posted today, or Bpy.data.images perf issues. These’s even a patch in the latter thread to vastly improve image copying performance.

  • Building a Python extension module that works with Blender is not as easy as it can be (e.g. the missing headers noted in Including Python headers in distribution? by myself, challenges in building the module, how to properly install new Python modules in Blender’s environment, etc.). Improving this might help in attracting C++ developers that can provide optimized modules for certain tasks without having to know the Blender C++ code itself. Merely having a good API (the current Python API) is enough. Case in point: my PLY importer for Blender, written in C using a C library, but available as a Python library. But turning this into a true add-on is then not that straightforward again, as it requires the user to compile the source (although I haven’t looked if there’s general instructions for distributing Blender add-ons containing system-specific binaries, like a Python module).

what about an external merging software, were the users can merge their own .zip file (of C code) to the main build, then automatically compiling to create their own custom builds of various implementation? it would look like a C++ Plugins from the user point of view.

all this within a sexy GUI for users?

we can chose a clear name for this kind of addition to make a clear distinction between Python addons and “External Merge Parts”

a bit like what they are doing with the nexus mod community.
it can be done, no ? the software just need to merge lines of codes and keep track of them.