Introduction
Hi Blender developers!
I want to talk a bit about the Blender make bpy recipe and the unsupported and undistributed Blender as a Python module. I wanted to share some of my opinions about it and discuss some ways that I would like to make it better but do not know how. Apologies because this is a huge post but I don’t know where the best place would be to post this or what format it would be best in. Also the topic is very broad and deep, and I didn’t really want to split the topic up.
Overview
Python packaging is a very broad topic and because of that I would like to discuss the structure of what is being done today in the make bpy recipe in small chunks, first talking about what happens in the build and its impact on the usability of the resultant module.
Then I want to talk about what I think would be very helpful to be done in the future, first by discussing what changes, then by talking about how it impacts the user experience.
Some of the stuff I will talk about is a bit technical but I want to say that most of it is surface level and written from a user-perspective as I think about myself as a user first and I do not have a formal background in the numerous and powerful tools that Blender Foundation uses.
That said these are all just opinions and more than anything I just wanted to foster a broader discussion about Blender as a Python module because I think that the Python module can really get there with some effort and it would make it much more usable for all. Also I want to say that I mean no ill will, and shoutout to ideasman42 and the rest of the Blender team (internal at BF and just throughout the world) for making Blender better and better all the time.
Background
I want to talk a bit about what Python modules, packages, and extensions are, some schools of thought on how to structure and use Python modules, packages, and extensions. I think it’s useful to contextualize how Blender is currently making the Python extension, and how the user currently uses other Python extensions versus Blender’s bpy.so/.pyd Python extension.
I think many people on this forum will already know the stuff about Python so if reading that is a waste of time for you, then just skip to The Build Today.
What is a Python module?
According to the Python documentation
A module is a file containing Python definitions and statements. The file name is the module name with the suffix
.pyappended.
Additionally, files with the suffix .pyd or .so can also be modules, provided they contain Python definitions and statements. These files typically do so by interacting with Python through the Python C API, and are called “Extension Modules”. Blender’s bpy extension is one of these.
Python projects can often be expansive, with many definitions and statements spanning many lines of code. It is not always practical to put all the code for a Python package into one file because of this. For this reason, modules are typically small in scope and a higher level concept is usually used to manage this, called “packages”.
What is a Python package?
According to the Python documentation
Packages are a way of structuring Python’s module namespace by using “dotted module names”.
That is not all, however, because the Python Package index exists, users can install a Python project from the script pip.
All that is required to install a Python project that exists locally is:
pipis installed- The project has a
setup.pyfile that accurately describes the project layout when callingsetup()
For reference, the setuptools documentation has some examples of how simple some packages can be structured. I don’t think that Blender could get that simplified but some work could be done to make it easier.
Generally speaking, when a project is structured using packages and declares them properly, and its modules are highly modular, the code is very nice to use.
What is a Python extension
From the Python docs
extension modules can do two things that can’t be done directly in Python: they can implement new built-in object types, and they can call C library functions and system calls.
Extension Modules are just like Python modules with the added flexibility of being able to define new built in types and
Blender is a mixed language compiled application that runs natively, and because of that needs to be first built into a Python extension to work with Python without some kind of exchange layer in between.
Blender can build as a Python extension, but an extension on its own, with no packaging tools to speak of is very difficult for users of Python, and especially those who may be using Python for the first time, to use.
Overview of Python package distribution technologies
Python like many other programming languages out there can at times be rife with strange assumptions and pitfalls and nowhere have I seen that more in Python than in it’s package management system, but on the whole it is actually quite simple. You have (basically) two options:
- Source distribution
- Binary
wheeldistribution
Source distribution is the overall simplest for basic Python only projects. The sources get zipped up. Python can compile the modules the first time they are loaded (sort of - it’s the __pycache__ folder) and everything is simple and kind of magically works.
For more complex Python and hybrid (some Python module, some extension module) projects the packaging gets a little more complex but, not by much if all you have is a “ready to compile” set of .c and .h files. If the project is structured like that and does not have complex dependencies the pure Python sources and native sources can be sent to the user in a source distribution and compiled in-situ. You just need to list the extension in the setup.py
Blender can’t do this currently and I’m not convinced that I would ever be able to do this. So a binary distribution it is.
With binaries, it’s tricky because how do you know it’s going to work on the target system? Well, that’s where wheels (sort of) solve part of that problem. With a wheel you can distribute the built version of the required libraries and the scripts and everything, and the distribution will just be limited to whether the cpu architecture was supported or not and if the platform was the same.
If you are curious then you can read more about wheels here at realpython.com.
This is great unless you want to host the wheel on PYPI and your platform is linux. Then, you will get rejected that it’s not a manylinux flavored wheel. Basically linux wheels need to be built on an old linux and audited. This (sort of) ensures that the lib c being used is cross linux compatible. Note that this still doesn’t work on Alpine though. There are cases where this doesn’t result in something perfectly portable. Here is a great talk about wheels, manylinux wheels, what they are and why they work:
For Blender, and other CMake based solutions… guess what, there is no python built system for that. Sorry but you will have to manage it basically by yourself by calling all the right things at exactly the right time and trick parts of the Python packaging system (setuptools and distutils) into thinking that build_ext ran and mark the files yourself for install. That’s okay because after we do it, the installation works to an extent, but it’s worth mentioning that there are options out there for managing Python extension modules that do not require the heartache. It is something, that I feel, is potentially worth looking into.
The Build Today
Users can right now build Blender as a Python Extension Module using the make bpy helper script. This is documented in the Blender wiki, and works pretty reliably as long as the build instructions for your given platform are followed diligently.
When compiling WITH_PYTHON_MODULE=ON and WITH_PYTHON_INSTALL=OFF on Windows, for example the following output is produced:
This intuitively contains the .pyd extension module, and the .dll files that are required for the native module, as well as Blender’s necessary startup scripts.
Benefits with the build today
Gives you just what you need
Builds and configures with a helper script.
Risks with the build today
Getting to this point is kind of tough for many users
Actually, if you currently do just make bpy with the prebuilt libraries, the Blender scripts and modules directory (a.k.a, 2.91/) sometimes ends up in the prebuilt libraries’ .../lib/python/.../site-packages path.
Also site-packages is not always the correct directory for 2.91/, depending on the platform.
User is left to copy everything to the proper directory, and may not know where that is since the Blender wiki tells them the wrong thing (only has instructions for a “user” or “site-wide” install, which isn’t repeatable on venv for some platforms) and plus they are just used to pip install bpy, that’s it.
Enhancements
Currently the build works the way it does, and I have been able to use bpy.so on every machine that I have compiled it on. I am slowly getting more familiar with CMake so that I can change a few issues I see, but I am very inexperienced with this build system. If someone else doesn’t get to it first, I will eventually post diffs for these issues which I consider the biggest hindrance to bpy.so/.pyd usability:
2.91/(or whatever version) ← This folder is required to be in different locations depending on the platform
a. MacOS:venv/lib/python3.7/Resources
b. Windows: must be sibling to the currentpython.exe
c. Linux: in current python’ssite-packages- Blender mangles the
sys.pathwhen imported, which, ironically, breaks its own import see also - Related to importing the Blender startup scripts, because the module depends on the startup scripts it breaks pyinstaller apps
I am planning on working on things that have some impact though, and just submitted some files that should in theory allow the user to install bpy on their system by performing the following:
python3 setup.py install && bpy_post_install
They will be able to do that at least… provided those files end up in the build directory after the build occurs. How can I get the build system to do this? I try Googling for CMake quick starts but I think I want something even far more basic than that.
Also, full disclosure, I have been working on the blenderpy repo which I am hoping will be able to get bpy installable on any system.
In that repo, I also have vscode tasks that automate the build process. By Clicking “View → Command Pallette → Tasks: Run Task” you should be able to build bpy in an automated fashion.
Those are my thoughts, FWIW, and I will try and commit more to Blender soon.