Tips for SVN traversal in python

Blender Development
#1

Hi all!

I am making a set of tooling for building Blender as a Python module in Python. The goal of this tooling is to provide early support for building Blender as a Python module for Windows, MacOS and Linux. I have had success in following the build scripts on all three systems now that I have one of each.

The build tooling that I am making for this compares the release tags of the Git Repo to the release tags of the SVN Repo. Then, inside of the SVN repo we determine for which platforms precompiled libraries exist.

The traversal of the SVN repo is really slow however. I retrieved a complete list of which Python versions were found for which operating system and it took a little over a half an hour. Is anyone familiar with an efficient svn search algorithm that could perhaps reduce the time it takes to map these objects? Or maybe (more likely) there is a place in my code which is inefficient?

The library where this development is occurring is bpy-make in “sources.py”. The slow code is in BlenderSvn.platforms.

I am however happy with the results I am getting:

Click to expand results 
<Version('2.26')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.27')>: [None],
 <Version('2.28a0')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.28rc0')>: [None],
 <Version('2.28')>: [None],
 <Version('2.30')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.31a0')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.31')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.32')>: [{'darwin-6.1-powerpc/': set(), 'windows/': {(2, 2)}}],
 <Version('2.33a0')>: [{'darwin-6.1-powerpc/': set(),
                        'windows/': {(2, 3), (2, 2)}}],
 <Version('2.33')>: [{'darwin-6.1-powerpc/': set(),
                      'windows/': {(2, 3), (2, 2)}}],
 <Version('2.34')>: [{'darwin-6.1-powerpc/': set(),
                      'windows/': {(2, 3), (2, 2)}}],
 <Version('2.35a0')>: [{'darwin-6.1-powerpc/': set(),
                        'windows/': {(2, 3), (2, 2)}}],
 <Version('2.35')>: [{'darwin-6.1-powerpc/': set(),
                      'windows/': {(2, 3), (2, 2)}}],
 <Version('2.36')>: [{'darwin-6.1-powerpc/': set(),
                      'windows/': {(2, 3), (2, 2)}}],
 <Version('2.37a0')>: [{'darwin-6.1-powerpc/': set(),
                        'windows/': {(2, 3), (2, 2)}}],
 <Version('2.37')>: [{'darwin-6.1-powerpc/': set(),
                      'windows/': {(2, 3), (2, 2)}}],
 <Version('2.40')>: [None],
 <Version('2.41')>: [None],
 <Version('2.42a0')>: [None],
 <Version('2.42')>: [None],
 <Version('2.43')>: [None],
 <Version('2.44')>: [None],
 <Version('2.45')>: [None],
 <Version('2.46')>: [None],
 <Version('2.47')>: [None],
 <Version('2.48a0')>: [None],
 <Version('2.48')>: [None],
 <Version('2.49a0')>: [None, None],
 <Version('2.49b0')>: [None],
 <Version('2.49')>: [None],
 <Version('2.50')>: [None],
 <Version('2.51')>: [None],
 <Version('2.52')>: [None],
 <Version('2.53')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 1)},
                      'win64/': {(3, 0), (2, 5), (2, 6), (3, 1)},
                      'windows/': {(3, 1)}}],
 <Version('2.54')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 1)},
                      'win64/': {(3, 1)},
                      'windows/': {(3, 1)}}],
 <Version('2.55')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 1)},
                      'win64/': {(3, 1)},
                      'windows/': {(3, 1)}}],
 <Version('2.56a0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 1)},
                        'win64/': {(3, 1)},
                        'windows/': {(3, 1)}}],
 <Version('2.56')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 1)},
                      'win64/': {(3, 1)},
                      'windows/': {(3, 1)}}],
 <Version('2.57a0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2), (3, 1)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.57b0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2), (3, 1)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.57')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2), (3, 1)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.58a0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2), (3, 1)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.58')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2), (3, 1)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.59')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2), (3, 1)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.60a0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.60')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.61')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.62')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.63a0')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.63')>: [{'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.64a0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 2)},
                        'win64/': {(3, 2)},
                        'windows/': {(3, 2)}}],
 <Version('2.64')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 2)},
                      'win64/': {(3, 2)},
                      'windows/': {(3, 2)}}],
 <Version('2.65a0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 3)},
                        'win64/': {(3, 2), (3, 3)},
                        'windows/': {(3, 2), (3, 3)}}],
 <Version('2.65')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 2), (3, 3)},
                      'windows/': {(3, 2), (3, 3)}}],
 <Version('2.66a0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 3)},
                        'win64/': {(3, 2), (3, 3)},
                        'windows/': {(3, 2), (3, 3)}}],
 <Version('2.66')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 2), (3, 3)},
                      'windows/': {(3, 2), (3, 3)}}],
 <Version('2.67a0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 3)},
                        'win64/': {(3, 2), (3, 3)},
                        'windows/': {(3, 2), (3, 3)}}],
 <Version('2.67b0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 3)},
                        'win64/': {(3, 2), (3, 3)},
                        'windows/': {(3, 2), (3, 3)}}],
 <Version('2.67')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 2), (3, 3)},
                      'windows/': {(3, 2), (3, 3)}}],
 <Version('2.68a0')>: [{'android/': set(),
                        'darwin-6.1-powerpc/': {(3, 1)},
                        'darwin-8.0.0-powerpc/': {(3, 1)},
                        'darwin-8.x.i386/': {(3, 1)},
                        'darwin-9.x.universal/': {(3, 3)},
                        'win64/': {(3, 2), (3, 3)},
                        'win64_vc11/': {(3, 3)},
                        'windows/': {(3, 2), (3, 3)},
                        'windows_vc11/': {(3, 3)}}],
 <Version('2.68')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 2), (3, 3)},
                      'win64_vc11/': {(3, 3)},
                      'windows/': {(3, 2), (3, 3)},
                      'windows_vc11/': {(3, 3)}}],
 <Version('2.69')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': {(3, 1)},
                      'darwin-8.0.0-powerpc/': {(3, 1)},
                      'darwin-8.x.i386/': {(3, 1)},
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 2), (3, 3)},
                      'win64_vc11/': {(3, 3)},
                      'windows/': {(3, 2), (3, 3)},
                      'windows_vc11/': {(3, 3)}}],
 <Version('2.70')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 3)},
                      'win64/': {(3, 3)},
                      'win64_vc11/': {(3, 3)},
                      'win64_vc12/': {(3, 3)},
                      'windows/': {(3, 3)},
                      'windows_vc11/': {(3, 3)},
                      'windows_vc12/': {(3, 3)}}],
 <Version('2.71')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4), (3, 3)},
                      'win64_vc12/': {(3, 4), (3, 3)},
                      'windows/': {(3, 4), (3, 3)},
                      'windows_vc12/': {(3, 4), (3, 3)}}],
 <Version('2.72')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4), (3, 3)},
                      'win64_vc12/': {(3, 4)},
                      'windows/': {(3, 4), (3, 3)},
                      'windows_vc12/': {(3, 4)}}],
 <Version('2.73')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4), (3, 3)},
                      'win64_vc12/': {(3, 4)},
                      'windows/': {(3, 4), (3, 3)},
                      'windows_vc12/': {(3, 4)}}],
 <Version('2.74')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4), (3, 3)},
                      'win64_vc12/': {(3, 4)},
                      'windows/': {(3, 4), (3, 3)},
                      'windows_vc12/': {(3, 4)}}],
 <Version('2.75')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4), (3, 3)},
                      'win64_vc12/': {(3, 4)},
                      'windows/': {(3, 4), (3, 3)},
                      'windows_vc12/': {(3, 4)}}],
 <Version('2.76')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 4)},
                      'win64/': {(3, 4)},
                      'win64_vc12/': {(3, 4)},
                      'windows/': {(3, 4)},
                      'windows_vc12/': {(3, 4)}}],
 <Version('2.77')>: [{'android/': set(),
                      'darwin-6.1-powerpc/': set(),
                      'darwin-8.0.0-powerpc/': set(),
                      'darwin-8.x.i386/': set(),
                      'darwin-9.x.universal/': {(3, 5)},
                      'win64/': {(3, 4)},
                      'win64_vc12/': {(3, 5)},
                      'win64_vc14/': {(3, 5)},
                      'windows/': {(3, 4)},
                      'windows_vc12/': {(3, 4), (3, 5)}}],
 <Version('2.78')>: [{'android/': set(),
                      'darwin-9.x.universal/': {(3, 5)},
                      'win64/': {(3, 4)},
                      'win64_vc12/': {(3, 5)},
                      'win64_vc14/': {(3, 5)},
                      'windows/': {(3, 4)},
                      'windows_vc12/': {(3, 4), (3, 5)},
                      'windows_vc14/': {(3, 5)}}],
 <Version('2.79a0')>: [{'android/': set(),
                        'darwin-9.x.universal/': {(3, 5)},
                        'darwin/': {(3, 5)},
                        'win64/': {(3, 4)},
                        'win64_vc12/': {(3, 5)},
                        'win64_vc14/': {(3, 5)},
                        'windows/': {(3, 4)},
                        'windows_vc12/': {(3, 5)},
                        'windows_vc14/': {(3, 5)}}],
 <Version('2.79')>: [{'android/': set(),
                      'darwin-9.x.universal/': {(3, 5)},
                      'darwin/': {(3, 5)},
                      'win64/': {(3, 4)},
                      'win64_vc12/': {(3, 5)},
                      'win64_vc14/': {(3, 5)},
                      'windows/': {(3, 4)},
                      'windows_vc12/': {(3, 5)},
                      'windows_vc14/': {(3, 5)}}],
 <Version('2.80')>: [{'darwin/': {(3, 7)},
                      'win64_vc14/': {(3, 7), (3, 6), (3, 5)},
                      'windows_vc14/': {(3, 7), (3, 6), (3, 5)}}],
 <Version('2.81')>: [{'darwin/': {(3, 7)},
                      'win64_vc14/': {(3, 7), (3, 6), (3, 5)}}]

Any guidance is appreciated!

#2

Careful there, for quite a bit of time we shipped a python version build with a different compiler and linked against a different c runtime than the main python did on windows. so mixing and matching mainline blender and our blender may look like it works, until it doesn’t and when it doesn’t it takes down the process without as much as an error message.

As long as you aim for 2.80 and up where we realligned with mainline python you should be ok though.

#3

Thanks for the insight. Is this just on Windows that this occurred?

My main focus is Python support for current and future Blender versions. I was going to try maybe going back as far as Blender 2.77 with Python 3.5, just to see what could be different.

My goals with this are to document some of the caveats with Blender and make a few notes. There has been a decent bit of response on Github for people using Blender in interesting ways. My hope is that by documenting some of these caveats if/when Blender becomes officially supported as a Python module some of these caveats would have already been reviewed.

#4

No idea, I’m not sure python has a preferred compiler version on any other platform than windows honestly.