Building plender as a python module fails with dynamic/static library linking problems

Building Blender
1

I am attempting to build blender as a python module. I start by running the commands:

cmake-3.17.0 -DWITH_PYTHON_MODULE:BOOL="ON" \
         -DWITH_PYTHON_INSTALL:BOOL="OFF" \
         -DWITH_PYTHON_INSTALL_NUMPY:BOOL="OFF" \
         -DWITH_PYTHON_INSTALL_REQUESTS:BOOL="OFF" \
         -DWITH_AUDASPACE:BOOL="OFF" \
         -DWITH_OPENAL:BOOL="OFF" \
         -DWITH_JACK:BOOL="OFF" \
         -DWITH_DRACO:BOOL="OFF" \
         -S "<BLENDER_FOLDER>/blender" -B "<BLENDER_FOLDER>/build_linux"
cd ../build_linux
cmake-3.17.0 --build .

Which results in the error:

/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vc1dsp_loopfilter.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_5' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vc1dsp_mc.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_9' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vc1dsp_mmx.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_9' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9intrapred.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_4' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9intrapred_16bpp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9itxfm.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pd_8192' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9itxfm_16bpp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_1023' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9lpf.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_80' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp9mc.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_64' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(hevc_cabac.o): requires dynamic R_X86_64_PC32 reloc against 'ff_h264_cabac_tables' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(cavsdsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_96' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(cavsidct.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_4' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(diracdsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_80' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h263_loopfilter.o): requires dynamic R_X86_64_PC32 reloc against 'ff_h263_loop_filter_strength' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_chromamc.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_8' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_qpel_10bit.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_1023' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_qpel_8bit.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_5' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(hevc_add_res.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_1023' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(hevc_idct.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_64' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(hpeldsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(idctdsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_80' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(me_cmp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(pngdsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_255' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(qpeldsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_3' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(rv40dsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_32' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp3dsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(vp8dsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_256' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_cabac.o): requires dynamic R_X86_64_PC32 reloc against 'ff_h264_cabac_tables' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(ac3dsp.o): requires dynamic R_X86_64_PC32 reloc against 'ff_ac3_bap_bits' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(dirac_dwt.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_2' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_chromamc_10bit.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_8' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_deblock.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_idct.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_32' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_idct_10bit.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pd_32' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_intrapred.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pb_3' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavcodec.a(h264_intrapred_10bit.o): requires dynamic R_X86_64_PC32 reloc against 'ff_pw_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libswscale.a(swscale.o): requires dynamic R_X86_64_PC32 reloc against 'ff_M24A' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libswscale.a(rgb2rgb.o): requires dynamic R_X86_64_PC32 reloc against 'ff_w1111' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(cabac-a-8.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_range_lps' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(quant-a-8.o): requires dynamic R_X86_64_PC32 reloc against 'x264_decimate_table4' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(cabac-a-10.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_range_lps' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(quant-a-10.o): requires dynamic R_X86_64_PC32 reloc against 'x264_decimate_table4' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(trellis-64-8.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_entropy' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libx264.a(trellis-64-10.o): requires dynamic R_X86_64_PC32 reloc against 'x264_cabac_entropy' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: <BLENDER_FOLDER>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libxvidcore.a(qpel_mmx.o): requires dynamic R_X86_64_PC32 reloc against 'xvid_FIR_14_3_2_1' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(png.o): requires dynamic R_X86_64_PC32 reloc against 'png_user_version_check' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngerror.o): requires dynamic R_X86_64_PC32 reloc against 'stderr' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngget.o): requires dynamic R_X86_64_PC32 reloc against 'png_get_pixels_per_meter' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngmem.o): requires dynamic R_X86_64_PC32 reloc against 'png_malloc_base' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngread.o): requires dynamic R_X86_64_PC32 reloc against 'png_sRGB_table' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngrio.o): requires dynamic R_X86_64_PC32 reloc against 'png_default_read_data' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngrutil.o): requires dynamic R_X86_64_PC32 reloc against 'png_crc_error' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngset.o): requires dynamic R_X86_64_PC32 reloc against 'png_set_sCAL_s' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngtrans.o): requires dynamic R_X86_64_PC32 reloc against 'png_set_filler' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngwio.o): requires dynamic R_X86_64_PC32 reloc against 'png_default_write_data' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngwrite.o): requires dynamic R_X86_64_PC32 reloc against 'png_sRGB_delta' which may overflow at runtime; recompile with -fPIC
/usr/bin/ld.gold: error: /usr/lib/x86_64-linux-gnu/libpng.a(pngwutil.o): requires dynamic R_X86_64_PC32 reloc against 'png_write_tEXt' which may overflow at runtime; recompile with -fPIC
collect2: error: ld returned 1 exit status
source/creator/CMakeFiles/blender.dir/build.make:464: recipe for target 'bin/bpy.so' failed
make[2]: *** [bin/bpy.so] Error 1
CMakeFiles/Makefile2:6552: recipe for target 'source/creator/CMakeFiles/blender.dir/all' failed
make[1]: *** [source/creator/CMakeFiles/blender.dir/all] Error 2
Makefile:179: recipe for target 'all' failed
make: *** [all] Error 2

A solution attempt

Which this website suggests happens because there is an attempt to use a static library as a dynamic one. So I then attempt to link against dynamic libraries, by rerunning the command:

cmake-3.17.0 -DWITH_PYTHON_MODULE:BOOL="ON" \
         -DWITH_PYTHON_INSTALL:BOOL="OFF" \
         -DWITH_PYTHON_INSTALL_NUMPY:BOOL="OFF" \
         -DWITH_PYTHON_INSTALL_REQUESTS:BOOL="OFF" \
         -DWITH_AUDASPACE:BOOL="OFF" \
         -DWITH_OPENAL:BOOL="OFF" \
         -DWITH_JACK:BOOL="OFF" \
         -DWITH_DRACO:BOOL="OFF" \
         -DFFMPEG_LIBRARIES:STRING="/usr/lib/x86_64-linux-gnu/libavcodec.so;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavformat.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavutil.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libavdevice.a;/usr/lib/x86_64-linux-gnu/libswscale.so;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libswresample.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libtheora.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libtheoradec.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libtheoraenc.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libvorbis.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libvorbisenc.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libvorbisfile.a;<BLENDER_DIR>/blender/../lib/linux_centos7_x86_64/ffmpeg/lib/libogg.a;/usr/lib/x86_64-linux-gnu/libx264.so" \
         -DPNG_LIBRARY_RELEASE="/usr/lib/x86_64-linux-gnu/libpng.so" \
         -S "<BLENDER_FOLDER>/blender" -B "<BLENDER_FOLDER>/build_linux"

Note that all references to the offending libraries in the earlier error message now reference dynamic system libraries.

The build now succeeds (in the sense there is no critical errors), but when importing bpy.so into python, I get the following error:

Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: <BLENDER_FOLDER>/build_linux/bin/bpy.so: cannot allocate memory in static TLS block

From my reading on the subject of thread-local storage, I figured out that this problem doesn’t exist when you have statically compiled libraries, which suggests my idea of going against the normal cmake compliation process by choosing to reference dynamic libraries was a bad one.

What do I need to do to compile blender as a python module?

More context and other failed attempts to find a solution:

To build blender as a python module, I have been following the instructions on the official website and this stack exchange answer, which when I combine, results in an initialisation process that looks like this:

mkdir ~/blender-git
cd ~/blender-git
git clone https://git.blender.org/blender.git
git checkout blender-v2.82-release
cd blender
git submodule update --init --recursive
git submodule foreach git checkout blender-v2.82-release
git submodule foreach git pull --rebase origin blender-v2.82-release

I have chosen to use the 2.82 release branch, because my objective is to reliably build blender as a python module (as opposed to having the latest possible version of blender). I can replicate this problem repeating the same process for v2.80, v2.81 and master branches of the blender source code.

I am using Ubuntu 18.04, with cmake-3.17.0.

Some other things I have tried include:

  • Building FFMpeg from source, using blender/build_files/build_environment/install_deps.sh
  • make bpy
  • Referencing the static versions (i.e. the *.a version) of the system libraries with -DWITH_STATIC_LIBS:BOOL="ON" option.
2

The standard configuration for the Python module is build_files/cmake/config/bpy_module.cmake. It’s what used when running make bpy. Maybe try using that to see if it resolves the problem.

cannot allocate memory in static TLS block seems related to jemalloc which is disabled in that configuration.

4

Taking @brecht’s advice, and with a bit of debugging myself, my configure command now looks like this:

cmake-3.17.0 -C<BLENDER_FOLDER>/blender/build_files/cmake/config/bpy_module.cmake \
-Wl,--verbose \
-DWITH_OPENMP:BOOL='0' \
-DWITH_OPENMP_STATIC:BOOL='1' \
-DPNG_LIBRARY_RELEASE:FILEPATH=<BLENDER_FOLDER>/lib/linux_centos7_x86_64/png/lib/libpng16.a \
-DWITH_MEM_JEMALLOC:BOOL=OFF \
-S <BLENDER_FOLDER>/blender \
-B <BLENDER_FOLDER>/build_linux_2.82

Which successfully builds bpy.so, though when importing into python 3.7, I now get this error:

ImportError: <BLENDER_FOLDER>/build_linux_2.82/bin/bpy.so: undefined symbol: omp_get_num_threads
1 Like
5

I’m having the exact same problem. Have you resolved this issue with either a different cmake or fixing the omp_get_num_threads some other way?