Raspbian - Blender as a Python Module: Build Errors

Hi Blender development forum!

I am trying to build Blender as a Python module. I am also trying to do this on Raspbian.

I know both the environment and the “as a Python module” are unsupported, but I just wanted to document some of the difficulties that I had along the way, and if anyone has any ideas for how to correct the errors, that would be much appreciated!

Limitations

First off, the whole setup process is manual cmake. I basically went through some of the dependencies and installed what I could, but some functionality will not be compilable due to unavailability of some of the dependencies.

NOTE: I’m unable to build with Cycles since OpenEXR>=2 is not provided in Raspbian

What makes matters more difficult is that compiling with make install -j 4 is unfeasible due to the low memory of the Pi (I am using Pi3). So the trial and error method is somewhat limited.

Build Script

mkdir .blenderpy
mkdir .blenderpy/master
mkdir .blenderpy/master/build_bpy_raspbian

cd .blenderpy/master

git clone http://git.blender.org/blender.git

cd blender

git submodule update --init --recursive
git submodule foreach git checkout master
git submodule foreach git pull --rebase origin master

make update

build_files/build_environment/install_deps.sh

cd ../build_bpy_raspbian

cmake ../blender -DWITH_CYCLES=OFF -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON

make install

Output

[ ...]

[ 29%] Building C object source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o
In file included from /home/tylergubala/.blenderpy/master/build_bpy_debian/source/blender/makesdna/intern/dna_verify.c:2:
/home/tylergubala/.blenderpy/master/blender/source/blender/blenlib/BLI_assert.h:102:37: error: static assertion failed: "DNA struct size verify"
 #  define BLI_STATIC_ASSERT(a, msg) _Static_assert(a, msg);
                                     ^~~~~~~~~~~~~~
/home/tylergubala/.blenderpy/master/build_bpy_debian/source/blender/makesdna/intern/dna_verify.c:6299:1: note: in expansion of macro ‘BLI_STATIC_ASSERT’
 BLI_STATIC_ASSERT(sizeof(struct bGPDstroke) == 372, "DNA struct size verify");
 ^~~~~~~~~~~~~~~~~
/home/tylergubala/.blenderpy/master/blender/source/blender/blenlib/BLI_assert.h:102:37: error: static assertion failed: "DNA struct size verify"
 #  define BLI_STATIC_ASSERT(a, msg) _Static_assert(a, msg);
                                     ^~~~~~~~~~~~~~
/home/tylergubala/.blenderpy/master/build_bpy_debian/source/blender/makesdna/intern/dna_verify.c:8821:1: note: in expansion of macro ‘BLI_STATIC_ASSERT’
 BLI_STATIC_ASSERT(sizeof(struct Scene) == 6092, "DNA struct size verify");
 ^~~~~~~~~~~~~~~~~
make[2]: *** [source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/build.make:125: source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:5918: source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

I am wondering if maybe this is due to the arm 32-bit platform, and the size of various structures is different…?

Also, this seems very similar.

Technically, 32 bit is no longer officially supported, so you might have more luck building 2.80 release instead of master. While I don’t expect any deliberate changes, the lack of 32bit testing of changes could allow a change to break 32bit builds. It may also be an arm (PI?) specific adjustment is needed.

The makesdna/makesrna part of blender is very picky when it comes to aligning struct members, this is to allow both 32 and 64 bit builds to read the same data, which keeps blend files readable on every system.

Compilers align struct members based on data size, which varies per target cpu, what that means is a short followed by a long could get two bytes of padding between them on a 64bit system but not a 32bit. To build blender we need to manually add padding to always match all possible compiler alignments. Also see blenders expected alignment sizes listed at the end of this post.

Your errors show failure in bGPDstroke as well as Scene. The fix is to find the struct members that are out of alignment and add padding.

all fields as far as makesdna is concerned are aligned properly for the layouts it tests for (if not it would have warned saying hey this field needs X bytes padding), what is happening in this instance is makesdna is happy with how things look, any by its math the bGPDstroke structure should be 372 bytes, the compiler however did something unexpected and inserted some extra padding ‘somewhere’ leading to a larger structure.

The trick now is gonna be locating where it did that, making an identical change in the blender headers and hope it is not violating makesdna’s alignment rules.

Yes, the “compiler did something unexpected” is what we need to adjust for, so that the compiler always aligns members to the location we expect. The extra padding added is why the expected size does not match the actual size of the struct.

Build and run this on your PI and compare output to an x86 build.

sizeof char - 1
sizeof short - 2
sizeof int - 4
sizeof long - 4
sizeof long long - 8
sizeof float - 4
sizeof double - 8
sizeof long double - 12
sizeof void* - 4
sizeof size_t - 4

Offsets:
next=0;
prev=4;
points=8;
triangles=12;
totpoints=16;
tot_triangles=20;
thickness=24;
flag=26;
_pad=28;
inittime=32;
colorname=40;
mat_nr=168;
caps=172;
gradient_f=176;
gradient_s=180;
_pad_3=188;
dvert=192;
_pad3=196;
runtime=200;
sizeof(bGPDstroke)=372

@sambler Thanks for the insight, I have done the test, here are the results:

tylergubala@raspberrypi3:~/.blenderpy $ mkdir test-blendbGPDstroke
tylergubala@raspberrypi3:~/.blenderpy $ cd test-blendbGPDstroke
tylergubala@raspberrypi3:~/.blenderpy/test-blendbGPDstroke $ wget https://gist.githubusercontent.com/sambler/c8d9e036723f96c16b9cf3d0877fe5ee/raw/8a236d25ef7688a2c177aea19fd3985095b3cf8e/test-blend-bGPDstroke.c
--2019-11-17 23:26:25--  https://gist.githubusercontent.com/sambler/c8d9e036723f96c16b9cf3d0877fe5ee/raw/8a236d25ef7688a2c177aea19fd3985095b3cf8e/test-blend-bGPDstroke.c
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 151.101.184.133
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|151.101.184.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3768 (3.7K) [text/plain]
Saving to: ‘test-blend-bGPDstroke.c’

test-blend-bGPDstroke.c       100%[=================================================>]   3.68K  --.-KB/s    in 0s       
2019-11-17 23:26:25 (9.36 MB/s) - ‘test-blend-bGPDstroke.c’ saved [3768/3768]

tylergubala@raspberrypi3:~/.blenderpy/test-blendbGPDstroke $ ls
test-blend-bGPDstroke.c
tylergubala@raspberrypi3:~/.blenderpy/test-blendbGPDstroke $ cc test-blend-bGPDstroke.c -o testpad
tylergubala@raspberrypi3:~/.blenderpy/test-blendbGPDstroke $ ./testpad
sizeof char - 1
sizeof short - 2
sizeof int - 4
sizeof long - 4
sizeof long long - 8
sizeof float - 4
sizeof double - 8
sizeof long double - 8
sizeof void* - 4
sizeof size_t - 4

Offsets:
next=0;
prev=4;
points=8;
triangles=12;
totpoints=16;
tot_triangles=20;
thickness=24;
flag=26;
_pad=28;
inittime=32;
colorname=40;
mat_nr=168;
caps=172;
gradient_f=176;
gradient_s=180;
_pad_3=188;
dvert=192;
_pad3=196;
runtime=200;
sizeof(bGPDstroke)=376
tylergubala@raspberrypi3:~/.blenderpy/test-blendbGPDstroke $

This looks like it’s ARM related, I’ve built on i386, and only needed to make a minor change - unrelated to DNA offsets.

If it’s as simple as modifying a source file then I can attempt and document my results.

any progress on this? - anything i can do to help?

i have the same problem building for parabola/arch in a generic armv7h QEMU VM - the result is the same with or without the “cycles” option, and is not related to python modules, nor peculiar to the OP’s armv8 - FWIW, the same codebase and configuration builds fine on x86_64, i686 and aarch64 - in contrast to what the thread title indicates, sambler’s presumption was most likely correct, that this is not peculiar to any particular device, OS, or features, but is probably general to 32-bit ARM

$ cc test-blend-bGPDstroke.c -o ./testpad
$ diff ./testpad-x86_64-sample <(./testpad)
8c8
< sizeof long double - 12
---
> sizeof long double - 8
32c32
< sizeof(bGPDstroke)=372
---
> sizeof(bGPDstroke)=376


$ cmake \
  -GNinja \
  -C../build_files/cmake/config/blender_release.cmake .. \
  -DCMAKE_INSTALL_PREFIX=/usr \
  -DCMAKE_BUILD_TYPE=Release \
  -DWITH_INSTALL_PORTABLE=OFF \
  -DWITH_PYTHON_INSTALL=OFF \
  -DWITH_CYCLES_CUDA_BINARIES='OFF' \
  -DWITH_CYCLES_DEVICE_CUDA='OFF' \
  -DPYTHON_VERSION=3.8 \
  -DPYTHON_LIBPATH=/usr/lib \
  -DPYTHON_LIBRARY=python3.8 \
  -DPYTHON_INCLUDE_DIRS=/usr/include/python3.8
....
$ ninja
....
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
....
[580/4058] Building C object source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o
FAILED: source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o 
/usr/bin/cc -DNDEBUG -DWITH_DNA_GHASH -DWITH_FREESTYLE -DWITH_OPENGL -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D__LITTLE_ENDIAN__ -I../intern/atomic -I../intern/guardedalloc -I../source/blender/blenlib -I../source/blender/makesdna -Isource/blender/makesdna/intern -Wall -Wcast-align -Werror=implicit-function-declaration -Werror=return-type -Werror=vla -Wstrict-prototypes -Wmissing-prototypes -Wno-char-subscripts -Wno-unknown-pragmas -Wpointer-arith -Wunused-parameter -Wwrite-strings -Wlogical-op -Wundef -Winit-self -Wmissing-include-dirs -Wno-div-by-zero -Wtype-limits -Wformat-signedness -Wrestrict -Wnonnull -Wuninitialized -Wredundant-decls -Wshadow -Wno-error=unused-but-set-variable -Wimplicit-fallthrough=5 -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O2 -pipe -fno-plt -g -fvar-tracking-assignments -fdebug-prefix-map=/build/blender/src=/usr/src/debug -fuse-ld=gold -fopenmp -std=gnu11 -pipe -fPIC -funsigned-char -fno-strict-aliasing -O2 -DNDEBUG -MD -MT source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o -MF source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o.d -o source/blender/makesdna/intern/CMakeFiles/bf_dna.dir/dna_verify.c.o   -c source/blender/makesdna/intern/dna_verify.c
In file included from source/blender/makesdna/intern/dna_verify.c:2:
../source/blender/blenlib/BLI_assert.h:102:37: error: static assertion failed: "DNA struct size verify"
 #  define BLI_STATIC_ASSERT(a, msg) _Static_assert(a, msg);
                                     ^~~~~~~~~~~~~~
source/blender/makesdna/intern/dna_verify.c:6291:1: note: in expansion of macro ‘BLI_STATIC_ASSERT’
 BLI_STATIC_ASSERT(sizeof(struct bGPDstroke) == 372, "DNA struct size verify");
 ^~~~~~~~~~~~~~~~~
../source/blender/blenlib/BLI_assert.h:102:37: error: static assertion failed: "DNA struct size verify"
 #  define BLI_STATIC_ASSERT(a, msg) _Static_assert(a, msg);
                                     ^~~~~~~~~~~~~~
source/blender/makesdna/intern/dna_verify.c:8813:1: note: in expansion of macro ‘BLI_STATIC_ASSERT’
 BLI_STATIC_ASSERT(sizeof(struct Scene) == 6092, "DNA struct size verify");
 ^~~~~~~~~~~~~~~~~
....
ninja: build stopped: subcommand failed.

Are there straightforward instructions on how to setup parabola Linux on armv7h QEMU?

Checking https://wiki.parabola.nu/ARM_Installation_Guide looks focused on hardware installation, not QEMU.

I could repro with win32, try this

diff --git a/source/blender/makesdna/DNA_gpencil_types.h b/source/blender/makesdna/DNA_gpencil_types.h
index 33dfe66a151..5cf86164920 100644
--- a/source/blender/makesdna/DNA_gpencil_types.h
+++ b/source/blender/makesdna/DNA_gpencil_types.h
@@ -226,6 +226,7 @@ typedef struct bGPDstroke {
   void *_pad3;
 
   bGPDstroke_Runtime runtime;
+  void *_pad4;
 } bGPDstroke;
 
 /* bGPDstroke->flag */
diff --git a/source/blender/makesdna/DNA_scene_types.h b/source/blender/makesdna/DNA_scene_types.h
index 7cfd19b4bd9..4b989e52f81 100644
--- a/source/blender/makesdna/DNA_scene_types.h
+++ b/source/blender/makesdna/DNA_scene_types.h
@@ -1796,6 +1796,7 @@ typedef struct Scene {
 
   struct SceneDisplay display;
   struct SceneEEVEE eevee;
+  void *_pad9;
 } Scene;
 
 /* **************** RENDERDATA ********************* */

the ‘dna_verify’ code compiled using LazyDodo’s patch - that appears to have worked; but i cant verify total success yet, because i hit a different error - the error seems t be related to the toolchain though - i will need to look into it more

i got past the toolchain problem and blender at git tag ‘v2.81a’ , compiled successfully in the parabola/arch armv7h QEMU VM, with LazyDodo’s patch applied

seeing this is a forum and not a bug tracker, i suppose i should ask if there is any chance of getting the bug verified by the dev team, and this patch upstreamed - is that why you were asking for a VM image @ideasman42 ? - the images need to be bootstrapped from a running arch-like system, and we dont normally publish any; but i could publish one if that would be useful

An easy way to test this on ARM32 + QEMU would be nice, pre-built image or not. From the wiki I didn’t see straightforward instructions to set this up.
Someone who has done this before could list the steps in a single document.

this is that document; but it requires a running arch-like system to bootstrap from, with the ‘pacstrap’ program

https://git.parabola.nu/parabola-vmbootstrap.git/tree/README

there is just not any straight-forward general way to do it, without a pre-made image, or bootstrapping a fresh one with the ‘parabola-vmbootstrap’ program - the standard procedure, described on the wiki, is to explode a pre-made tarball onto a real device, or to bootstrap a fresh install from another system

it would be no trouble for me to publish a “ready-to-boot out-of-the-box” QEMU image - there is none now, only because it is just not a popular use-case

Hey, I tried building 2.81a on ARMv7 and it still fails with the same errors. Was this already fixed in the future 2.82?

2.81a does not have this fix, it only landed recently.

Both the 2.82 and master branches have it so give that a try.

Thanks! The important thing is that the future 2.82 has this fix. Cheers!

It has it, since we don’t build for arm ourselves it wouldn’t hurt if you could give it a try and validate it works now, there is still time for additional fixes for 2.82.

1 Like

I was able to build master (2.83) today on a raspberry pi 3 on Buster Lite, albeit without Cycles since no compatible openexr lib for raspbian exists.