Python module build process: Select Python version

Archive Python API
1

Hello all,

First time poster here: I am the “maintainer” of a module called ‘bpy’ on pypi. It allows prospective addon developers to install the Blender as a Python module into their venv for unit testing, or for the purposes of making a 3d enabled application using Blender as the driver for the Graphics.

Currently, however, I am facing a problem. I use an automated build process that follows very similarly to the @Ideasman42 topic on the Blender official wiki “Building Blender as a Python module” (credit to the original author, although my bookmark to this page has been broken since the wiki’s migration…)

The problem is that, after building Blender as the Python module bpy, which I am doing for multiple Python versions (maybe there is my issue, maybe that just won’t work…?), the bpy module is looking for the incorrect pythonXX.dll; it always seems to be looking for python36.dll.

My distribution process is like this:

  1. I enter my venv: call activate venv
  2. then build: python setup.py bdist_wheel [1]
  3. exit the venv: call deactivate
  4. Repeat steps 1 through 3 for each version of Python I have a venv for [2]
  5. Finally, upload: twine upload dist/*

[1] This retrieves the git sources, svn libs if the platform is Windows, and performs the build action using cmake
[2] My optimistic goal is support for all (32-bit or 64-bit) Python versions 3.4 or greater; I currently only support Windows

My thought is, that by entering into the Python virtual environment prior to configuring and building, cmake would recognize the current version of Python (I think activate sets the PYTHON_PATH variable locally…?) and therefore would build to the correct version of Python.

However, that seems to not be the case, as on import bpy I immediately get an Import Error:

import bpy

ImportError: DLL load failed: The specified module could not be found

Attempting to trace this import error down, I made a temporary copy of the current pythonXX.dll [3], and renamed it to python36.dll on a hunch. Lo and behold, just as I thought, it is trying to reach out to python36.dll! (Or at least, that’s how I’m understanding this?)

[3] I believe I was in my newly created Python 3.7 Virtual environment at the time

venv\3.7-32\Scripts\activate
(3.7-32) copy venv\3.7-32\Scripts\python37.dll venv\3.7-32\Scripts\python36.dll
(3.7-32) python

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Intel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.

>>> import bpy

ImportError: Module use of python36.dll conflicts with this version of Python

>>> quit()
(3.7-32) del venv\3.7-32\Scripts\python36.dll

What the above error is telling me, and the fact that it is a different error than before the copy rename operation took place, is that when I import bpy, regardless of the system settings of Python that were present at cmake’s Configure/Build time, bpy will still attempt to latch onto Python 3.6’s dll.

This is rather unfortunate in my opinion. First, there are some proprietary, prebuilt pyd / so Python packages out there that could potentially work very well in combination with the Blender as a Python module bpy. Autodesk’s fbx Python sdk comes to mind, and only works on Python 3.3. So the choice in Python version can sometimes be a motivating factor due to surrounding factors.

Secondly, as new Python versions come out (3.7 is a recent addition), it can be useful for Blender addon maintainers to unit test their addons with the new Python runtimes and potentially address issues ahead of time, or test out new language features and possible performance upgrades introduced in the latest major release.

With a flexible build process, that could create the bpy Python module for any version of Python, using proprietary Python modules or integrating unit tests for a Blender addon would be a snap. It could even potentially be part of continuous integration.

In my opinion this is a huge missed opportunity. But I am only a guy on the internet, and this is just my opinion.

But I want to do better than that; I would like to remedy this myself, or help remedy this for others so that they can start using the bpy Python module in their projects, regardless of what their Python version of choice is (within reason).

That being said, I have my own hypothesis on what the potential causes could be. For starters, I find it very odd that, when we have to download the svn libs for Windows, it comes with the relevant Python .dll and .lib files, and during the configuration process, I can’t seem to find the cmake entry for PYTHON_LIBRARY.

Shouldn’t we be able to override which .dll and .lib files are included into the C build process?

Regardless, I would like to be able to help remedy this issue at the very least. If this is something on the side of the C/C++ API, while I’m not extremely familiar with C/C++, I’m at least comfortable enough to perform my own builds, do my own tests, and Debug code. The main point is, I would like to not just fix this, but fix it "The Right Way"™, and am ready and willing to provide help and information.

Thank you so much for all of your time reading this. If this simply isn’t feasible, that’s fine, I just can’t think of a reason why it wouldn’t be at the moment.

Here is my repository:

Blenderpy

And my initial Blender Stack Exchange post regarding this (my be removed as off-topic in the near future)

Blender as Python module; link to the correct pythonXX.dll?

Thanks in advance!

2

Can’t say i’m a super huge fan of bpy being available though pip, blender as a python module always was something that worked but was unsupported if you were an expert user that knew how to build it odds were you were also able to solve any of the issues that may come with it, however now that it is available as easy as a pip install people have already shown up in some of the support channels asking for help with it. Having to tell them ‘you shouldn’t be using this, because it is unsupported’ isn’t great… not great at all.

3

I can just take it down; is that preferred?

4

@ideasman42 is the module owner, lets wait for his opinion.

5

I’m of two minds on this. On one hand, the fabled Blender 3.0 is (rumored) to be a Python module. Do “import blender” and you are off to the races. The bpy module gives us a glimpse of that glorious day.

On the other hand, Blender is built with a specific version of Python. Changing the Python version is more than just changing a version string. Blender itself needs to be recompiled and relinked since each Python version has its own structure sizes. New Python methods are added and old ones deleted or deprecated. Any addon’s are affected.

Using pip to build Blender with randomly chosen Python versions sounds extremely challenging. I think it is outside the scope of the Blender project, but that doesn’t mean you shouldn’t be doing it. I do think you are underestimating the size of the problem, though.

side note: Windows Python dlls are in SVN so Windows users don’t have to build or install the correct version of the libs themselves

2 Likes
6

I undoubtedly am underestimating the size of the problem.

However 3.6 builds and imports fine. If I could get the Blender Python module to use the correct pythonXX.dll on any other version of Python, (possibly at that point hard-crashing due to the mismatching calls(?)) I could blacklist those versions of Python (can’t be installed through pip) while I search for a solution. I don’t know the best way to go about that, hence the post.

I’m looking for the best way of doing this. My way might not be the best way and probably isn’t, but I am hopeful and willing to put the time and effort into this, so if anyone can guide me to a better way or criticize I am all ears.

If, however the Blender community deems that it is unwanted, and potentially damaging to the community at large to produce pip-installable bpy wheels and distribute them (either because the backing developer community just doesn’t want to deal with question askers or because there is some vulnerability or critical instability I don’t know about), then I will take it down.

If people think it is bad, and unwanted, I will take it down no problem.

But I am willing to try and make it better, for the long term. I really would prefer to put in some hard work to make this better if someone can help guide me in the right direction.

Blender 3.0 sounds awesome. That day would be glorious indeed.

The hope is to maybe be able to serve wheels to Python users version 3.4 and above. If I can only serve wheels to users 3.5 or 3.6 and above or something like that, then that would be okay to me. I guess though that my main curiosity is if I could checkout a specific git tagged version of Blender, say a release that came bundled with 3.4, and then just build the 3.4 wheel using those sources. I wouldn’t know where to look for that, though.

But if the idea is rejected in large by the community I’ll do everyone a favor and let sleeping dogs lie, I guess.

Presumably a prospective user wants to build the module to use it, right? Can we check known Python installation paths? %PYTHON_PATH% environment variables? For instance, I found Python36.dll here:

C:\Users\TGubs\AppData\Local\Programs\Python\Python36

The reason I ask is that if the git tag method worked, theoretically speaking or course, then obviously there’d have to be a different version of Python in the svn library path, as it would be trying to compile to a different version of Python (that’s my reasoning at least? Maybe that doesn’t make too much sense)

Besides that, BF trusts Linux users to install the correct python-dev package from apt. Just saying from the building perspective a user has to know what they’re doing at some point.

7

if you go back in time to a release that shipped with 3.4 , you would also get a bpy api from that era as well, one version will have things that another version doesn’t, and would lead to more support questions.

Selecting the pyton version is controlled through build_files\cmake\platform\platform_win32.cmake however, one we upgrade to a python version, new api calls available in that new version are generally used, trying to build current master against 3.3/3.4/3.5 will be problematic for this reason.

8

@TylerGubala looking into this, I had a report that win32 needs to link with the Python library (DLL).
Unfortunately I can’t find the report, possibly it was informally over IRC.

Try remove these lines in ./build_files/cmake/macros.cmake

	# since we are using the local libs for python when compiling msvc projects, we need to add _d when compiling debug versions
	if(WITH_PYTHON)  # AND NOT WITH_PYTHON_MODULE  # WIN32 needs
		target_link_libraries(${target} ${PYTHON_LINKFLAGS})

		if(WIN32 AND NOT UNIX)
			file_list_suffix(PYTHON_LIBRARIES_DEBUG "${PYTHON_LIBRARIES}" "_d")
			target_link_libraries_debug(${target} "${PYTHON_LIBRARIES_DEBUG}")
			target_link_libraries_optimized(${target} "${PYTHON_LIBRARIES}")
			unset(PYTHON_LIBRARIES_DEBUG)
		else()
			target_link_libraries(${target} ${PYTHON_LIBRARIES})
		endif()
	endif()

If this works we could make it check for:

if(WITH_PYTHON AND NOT WITH_PYTHON_MODULE)

1 Like