Improving Python/C/C++ and general development process

Archive Blender Development
1

I do not know if it is possible or feasible or to what extent it could improve the overall performance of Python. But I’m mentioning it here in case someone has more knowledge about it.

There are two libraries that catch my attention:

Pypy is a python interpreter 100% compatible with standard python code that compiles to c at runtime. With this interpreter I think that no changes in the current python codes would be necessary.

Numba with this is necesary to add small codes to the python codes but it allows among other things to execute python code from the GPU and I think it could be interesting.

What do you think about that?
It would be worthwhile to include some of these technologies in blender?.
I think it could improve the performance of most addons and python in general without much change, don’t you think?.

1 Like
2

wow, this seems pretty interesting.

@sybren @brecht @dfelinto what do you think about this?

1 Like
3

PyPy is not a practical option if we want to be compatible with the VFX reference platform, and it’s not compatible with some commonly used Python modules. In general they are not really targetting the embedding use case, it’s not something I would count on to be supported well now and in the future.

GPU acceleration is not going to help many Python scripts. The complexity of GPU programming and the overhead of transferring data between CPU and GPU is too big. There are much easier and more practical optimizations possible using just the CPU.

2 Likes
4

But what about the situation in PyPy? I mean, being able to pre-compile the addons to C inside blender at runtime would help a lot, isn’t it?

I don’t say that we should use PyPy, but if there is something that can be done to facilitate this to Python developers/users it could be great, because right now as soon as you give python a big chunk of work is super slow, a good example is the addon Sorcar, it’s a node based modeling addon, and it’s super slow as soon as you want to use it with something mid sized.

1 Like
5

There are other options for that already, like compiling with Cython. And I think Brecht makes some good points already against using PyPy.

Why not flip it around? Instead of trying to get add-ons fast enough so that they can introduce wildly new behaviour in Blender, add that behaviour to Blender itself. It’s Open Source, and more Blender developers are always welcome.

7 Likes
6

That could be the best thing :slight_smile:

And I agree with you, but for example I have some scripts not even addons that I quickly made to solve production problems, some may be useful, but I still don’t have a proper idea on how to convert those addons into proper C++ code to be implemented in Blender, I would love too, I can assure you, but I lack the knowledge.

A similar thing happens to @BD3D with the scatter addon, at some point he could implement part or the whole addon into Blender in C++ but he lacks the knowledge to do it, Python is accessible, C++ is not as accesible.

We need the course I told you som time ago @sybren… C++ for Artists hahaha ir at least for TD’s , it could be more complex, but it will help a lot of people to be able to customize things in Blender, to fix things in Blender and to provide patches to make Blender evolve :slight_smile:

In other words… give me a course like your Python course and you will get at the very least 1 more developer collaborating hehehe

6 Likes
7

two things:

  1. What if the proposal is rejected because the institute don’t like the idea ? the only solution to share the creator perspective to users is via a plugin

  2. What about the learning curve?
    Many peoples who are coding addons don’t have any other Programmation experience (like myself)

C++ for Artists

I agree that a course is needed, but at the same time, we may want to let the pros code blender itself. let’s throw shades here.

python is accessible, within a year you can learn and create cool and super useful addons, but C demand much much more experience (from what i heard) so a C course may flood merge proposals with bad codes that will waist everyone times

( edit: hmmm but at the same time, the bests ideas of tools and UI are coming from the users directly, ideas implemented by a non-user are prolly bad )

1 Like
8

Cython with blender for artists, are welcome too :slight_smile:

9

Python has many advantages over C++, and I’m certainly not blind to that. I certainly understand the difficulty of learning C++ well enough to do Blender development. My point is mostly that such a switch to PyPy would take away a lot of development time from the core developers. This will decrement the amount of time available to fix bugs and add other features.

It’s the Blender developers who decide, also from outside the Blender Institute :wink: Also, it’s possible to fork Blender and add features to your own build. It really helps to convince that something is good to have in Blender when there are already lots of people using that feature. Same goes for making something in Python first, showing that people want & use it, then convert to C/C++ for inclusion in Blender if that is what the Blender devs want as well.

2 Likes
10

Many speed disadvantages in Python can be gotten around with clever, Pythonic programming and the use of efficient libraries like numpy. The only bottlenecks I’ve ever run into (that weren’t my fault) were interaction with Blender, e.g. bmesh and accessing a very large armature object, or situations where a call to bpy.ops was necesary. What I’d like to see is a bit of profiling to see which interactions are slow, and improve those. I’d like it if there were fewer situations where bpy.ops was necessary in an otherwise pure bpy.data script.
If you want to add new functionality, I agree, doing it within Blender is probably the best way. But it’s also possible to pass Python objects and functions to/from C with ctypes. I’d like to see some guides or documentation on the right way to do this with Blender. I think it could be especially helpful to people trying to create GPL-compliant bridges to other softwares that are themselves not free (as in free speech). And while I’m personally unhappy with the mere existence of any non-free software, it would be beneficial for Blender and Blender artists to be more integrated in the proprietary software world without compromising the integrity of the GPL. Perhaps Blender can be more friendly to this kind of a workflow? What are the challenges to working this way already? I haven’t tried it myself.

7 Likes
11

@sybren

Hey hey hey! Don’t skip the bullet!

That “C++ for TD’s” course is a thing… we want you on that… even more than how much we want you improving Alembic and USD… and we want that a lot… :sunglasses:

I totally understand the PyPy thing, and I also understand that it may not be worth the effort, specially with Function Nodes coming, many things will be possible with nodes and the performance will be better than Python for sure.

But also I don’t think C++ to modify Blender is so hard to teach/learn, If someone knows C# or Java or Python, it’s not so complex to learn C++ itself, it’s just a bit of syntax change and having to understand a pair of concepts that are slightly different (do you know how much time I’ve been wondering what a Vector was until some simple book told me that it was basically the same as an Array?! ) and that made with an actual practical course with some targets, like hunting fixing some simple bugs, or improving some already existing tool.

So I like your idea of focusing more in the C++ side of things, because the other side is going to improve (we still lack an idea to give function nodes some interface, but that will come with time) but we need some documentation or even better, a small course that can bring Addon/Script developers near C++ side of things, and yes, things can go into custom builds, or maybe not the whole feature would be accepted, but some interesting and useful patches can be extracted from that full feature, there is a long etc of possible situations, but I’m all into making people understand the C++ side of things much more.

Right now it’s too difficult, even for experienced C++ programmers, the structure of information in code it’s super specific (which is normal) and the documentation available scratches the surface, so a TD cannot go directly into the C++ code and modify improve things easily, a TD or a TA are part artists part programmers, and some course in the same fashion to the one you did could be very beneficial in general :slight_smile:

@Josephbburg
Regarding the proprietary software side of things, I don’t really think being more into the proprietary world is a good thing, with that I mean, being inside any of the 3d indsutries that use proprietary software is good, complying with them is a whole different story, and the trend I think is to do things more and more open and free, specially for bridges, so I relaly don’t think it’s a need.

But with that said it’s true that in Blender some things evolve at a slow pace and allowing other developers to chime in and give solutions that don’t require custom builds it’s a good thing, because for example Cycles is a bit stagnated, and I know some new object types are being developed, and that’s super exciting, but performance wise and render engine feature wise is stagnated, I won’t mention all the things because there are many threads about that :slight_smile: and it’s clear that more funding can help, but since there is no clear way of stablishing priorities we cannot wait 2 years until we get more rendering speed with som GI cache because now we are getting a point cloud object, don’t take me wrong, it’s super hard to stablish priorities and it’s normal that internal devs have their own priorities, but if we need a third party render engine we need it at full speed, LuxCore is doing a super good job on that using as less python as they can and as much C++ as they can, but even with that there are situations that are simply limited by Python being slow, and that’s the part we need a solution for, the parts of Python that are slow, and when you work with big scenes (actual production scenes) even renaming objects in batches or creating new objects is slow.

So IMHO it’s a matter of giving solutions so addon developers can do their thing, because I doubt 5 different render engines can get accepted in master, no matter if they are open source like Lux, and also maybe developers don’t want it INSIDE master, they want to maintain an addon development, and I think that’s something that should be respected too, there is no need to make Blender a mamooth with ALL the tools in the industry, that’s why Addons are great, and we need them, but we need them with high performance even with complex scenes :slight_smile:

And while we can hire a developer, the true thing is that not all of us can hire a developer that charges proper hourly rates, the idea of “fund it yourself” don’t always work (sometimes work :slight_smile: ), and some developers simply have their own render engine or their own addon and want to develop it in a different way :slight_smile:

My 2 euros here (because all this is a bit more than 2 cents hehehe)

1 Like
12

This is the best approach IMHO. There are many things that Python scripts can do, and there may not be a one-size-fits-all approach to fix performance problems. Identifying the bottlenecks in a handful of use cases could allow us to get improvements with limited effort.

6 Likes
13

I’ll be completely honest here, one of the things that prevents me from contributing are things like this just sitting there tagged with needs review and TONS of community support:
https://developer.blender.org/D5336
https://developer.blender.org/D4786

These are just two examples off the top of my head- things people have asked for repeatedly, somebody put in the effort to do it, and they just sit there needing review for months. All the while the author has to keep up with merging back to master to make sure their code can be reviewed. At some point they just give up and it withers on the vine.

I understand that there are reasons, limited reviewer time, etc- but for someone like me who could help it’s my prime motivation for not helping. It’s for reasons like this that people like myself depend on a robust python api, because we don’t have the time to maintain a fork of Blender or babysit a diff waiting for reviewers for 6+ months with no sign of that ever changing.

Seriously, when I open up differential and sort by creation date (oldest) with a status of “needs review” and see hundreds of patches going all the way back to 2014 that people wasted their time working on because nobody ever even looked at it, my motivation drops to zero. If some of these diffs with lots of support can’t even get eyes on it, what chance does any change I make have?

18 Likes
14

I’m not saying it’s right, but I found that sometimes you have to ask for reviews from the devs rather than wait for them to happen. Just hanging out on chat going "hey xxx if you have 5 minutes can you take a look at Dxxxx` once in a while does wonders.

Personally I pick my battles, something that doesn’t matter if it lands today or next month: meh it can wait, reviewers will get to it when they get to it. If it’s something I desperately want in 2.82 before the window closes i chase the devs to get a review.

Don’t get me wrong, things could definitely be better, but things are not nearly as bad as you imagine they are.

8 Likes
15

@testure damn i’m following this from a year, i was thinking the project was just dead, but apperently it’s just a review issue ?

4 Likes
16

Don’t you think the patch review should be objective, aseptic and not dependent on the knowledge of blender’s internal bureaucracy?

1 Like
17

Yeah, that’s the part where i said

1 Like
18

I’m just saying that to someone like me it looks like an uphill fight to try to make any headway on contributing to Blender, and hopefully give you guys some ‘third party insight’ into why someone with development experience like myself would actively avoid making contributions to Blender. As you say, it may not be as bad as I imagine it, but I’m not the only one imagining it and certainly not the only one who does not submit patches as a result.

I’m not trying to derail the thread- i’m just providing a counter-point to @sybren’s comment about more developers always being welcome and that we should just introduce our ‘wildly new behavior’ directly into Blender.

Personally I think the Python API is already pretty fantastic, with some minor exceptions (as @Josephbburg pointed out) where api access doesn’t exist and we have to fallback to bpy.ops, but I find those situations few and far between.

3 Likes
19

I don’t think it’s good enough, an example would be the OBJ importer or any other importer/exporter written in Python, performance is awful.

20

I’m doing a python fracture, and performance bottlenecks are bpy.ops themselves. Depending on the number of chunks it takes longer for the operator to duplicate or the bisect operator.

By the way, to duplicate, with the “bpy.ops.object.duplicate()” operator is much slower than duplicating in this other way:

new_obj = obj.copy()
new_obj.data = obj.data.copy()
bpy.context.collection.objects.link(new_obj)

I’m trying to improve performance with cython but I don’t know much about it yet, yet the bpy.ops no matter how much cython I use I don’t think it’s going to improve performance just by using cython.

1 Like