Now, I’m wondering if there are any additional steps I should take after compiling to ensure that my build is functioning correctly.
While exploring the Blender Wiki (Tools - Blender Developer Wiki), I came across some testing tools mentioned there. However, I’m unsure if these tools are primarily designed to check the integrity of the build or if they serve another purpose.
I’ve also read about reproducible builds (Building Blender/Options - Blender Developer Wiki), but given that my build differs from the default release, I’m uncertain if that concept applies in this case.
Am I overthinking this? Can I consider a successful build as sufficient evidence that everything will likely work as intended?
Most of the time you should just be able to use the version of Blender you compiled and if nothing is glaringly wrong, then it should be fine to use.
But if you want to verify that Blender is working exactly as intended, then you can run various tests.
Many of those tests ensure that your build of Blender produces the exact same results as the version of Blender the Blender foundation produces. So if you want to test that Blender is working exactly as intended, then you would want to enable those tests,
I will use the Cycles render test as an example.
When the Cycles render test is enabled, then a wide range of tests will be run covering many features that Cycles has along with testing specific cases that were known to have bugs in the past. If a single pixel from a test is incorrect (even if you personally can’t tell the difference), then that test fails.
If any test fails, then you will need to investigate why it failed. Did it fail because you disabled a feature (E.G. You disabled ffmpeg). Or did it fail due to some other reason (E.G. You have a compiler bug, or you changed a piece of code that impacts that specific test case, or your tests and Blender code are out of sync)
I run make update 2019b on the source folder just to make sure:
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.27
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Updating Precompiled Libraries and Tests
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive cleanup C:/blender-git/blender\..\lib\tests
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive switch https://svn.blender.org/svnroot/bf-blender/tags/blender-3.3-release/lib/tests C:/blender-git/blender\..\lib\tests
At revision 63439.
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive update C:/blender-git/blender\..\lib\tests
Updating 'C:\blender-git\lib\tests':
At revision 63439.
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive cleanup C:/blender-git/blender\..\lib\win64_vc15
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive switch https://svn.blender.org/svnroot/bf-blender/tags/blender-3.3-release/lib/win64_vc15 C:/blender-git/blender\..\lib\win64_vc15
At revision 63439.
C:\Program Files\SlikSvn\bin\svn.exe --non-interactive update C:/blender-git/blender\..\lib\win64_vc15
Updating 'C:\blender-git\lib\win64_vc15':
At revision 63439.
Updating Submodules
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe fetch origin
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe pull --rebase origin blender-v3.3-release
From https://projects.blender.org/blender/blender-addons
* branch blender-v3.3-release -> FETCH_HEAD
Already up to date.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe fetch origin
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe checkout blender-v3.3-release
Switched to a new branch 'blender-v3.3-release'
branch 'blender-v3.3-release' set up to track 'origin/blender-v3.3-release'.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe pull --rebase origin blender-v3.3-release
From https://projects.blender.org/blender/blender-addons-contrib
* branch blender-v3.3-release -> FETCH_HEAD
Already up to date.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe fetch origin
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe checkout blender-v3.3-release
Switched to a new branch 'blender-v3.3-release'
branch 'blender-v3.3-release' set up to track 'origin/blender-v3.3-release'.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe pull --rebase origin blender-v3.3-release
From https://projects.blender.org/blender/blender-translations
* branch blender-v3.3-release -> FETCH_HEAD
Already up to date.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe fetch origin
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe checkout blender-v3.3-release
Switched to a new branch 'blender-v3.3-release'
branch 'blender-v3.3-release' set up to track 'origin/blender-v3.3-release'.
C:\Users\[USER_NAME]\AppData\Local\Programs\Git\cmd\git.exe pull --rebase origin blender-v3.3-release
From https://projects.blender.org/blender/blender-dev-tools
* branch blender-v3.3-release -> FETCH_HEAD
Already up to date.
Blender repository skipped: no remote branch to pull from
The output seems to indicate that the test files path is recognized. But when I run make test 2019b from the source directory to run all tests I get this:
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.27
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Building blender with VS2019 for x64 in C:\blender-git\blender\..\build_windows_x64_vc16_Release
C:\Program Files\CMake\bin\ctest.exe . --output-on-failure -C Release -O C:\blender-git\blender\..\build_windows_x64_vc16_Release\tests\log_Release.txt
Test project C:/blender-git/build_windows_x64_vc16_Release
No tests were found!!!
Simply running make test for me gives me an output like this:
E:\blender-git\blender>make test
No explicit msvc version requested, autodetecting version.
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.5.5
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Compiler Detection successful, detected VS2022
Building blender with VS2022 for x64 in E:\blender-git\blender\..\build_windows_x64_vc17_Release
C:\Program Files\CMake\bin\ctest.exe . --output-on-failure -C Release -O E:\blender-git\blender\..\build_windows_x64_vc17_Release\tests\log_Release.txt
Test project E:/blender-git/build_windows_x64_vc17_Release
Start 1: script_load_keymap
1/249 Test #1: script_load_keymap ................................................... Passed 0.59 sec
Start 2: script_validate_keymap
2/249 Test #2: script_validate_keymap ............................................... Passed 6.14 sec
Start 3: script_load_addons
3/249 Test #3: script_load_addons ................................................... Passed 4.90 sec
Start 4: script_load_modules
4/249 Test #4: script_load_modules .................................................. Passed 2.12 sec
Maybe combining the two build requests doesn’t work (E.G. Combining make test and make 2019b)? Or something is weird with your build environment? I’m not sure.
One other possibility that comes to mind is the WITH_GTEST=ON option. If that option is on, then when you run make, a bunch of tests will be made in and put in the /blender-git/build_windows_x64/tests/ folder. And then when you run make test those tests, along with the test in /blender-git/lib/tests/ are run.
Maybe since you enabled WITH_GTEST then went straight to running make test, it’s complaining that it can’t find any tests because no tests exist in the /blender-git/build_windows_x64/tests/ folder. To fix this, run make (or make 2019b?) then run make test (or make test 2019b?) and see if that fixes it.
If that doesn’t help, then you may need to wait for a response from someone more knowledgeable in this area than myself on what’s going on and how to fix it.
One thing I wanted to note:
Although there are many tests available, they do not cover every possible aspect of Blender. And as such, even if all the tests pass, your build of Blender may still be slightly wrong and produce slightly incorrect results in some areas compared to the official builds from the Blender foundation. It’s also likely that there will still be bugs, because sometimes bugs are just limitations in a feature, or the code working as normal, but not the way the user expects.
But if all the tests do pass, then it’s highly likely that your build of Blender will operate the same as one from the Blender foundation.
When I ran make update 2019b I was just trying to make sure if my build environment was able to see the test files.
I actually ran it after building Blender as I had just downloaded src and libraries, so I knew everything was up to date. It’s just that that was the only command I knew that could indicate to me if the test directory was visible. I assume that if it wasn’t, it would not show up or show a “test files missing” error.
I didn’t combine the Blender build process with the test. I first went through the build process by:
1-make full nobuild 2019b
2-Make alterations on CmakeCache.txt; Save
3-rebuild 2019b
4-Only once I had a working build did, I try to run the tests (as a separate command)
Maybe I mixed up some of the steps? Should I have built the test before building blender?
I checked my build_windows_x64/tests/ folder and it’s not empty. I ran the Windows “tree” command to list its contents:
A small note: I enter 2019b at the end because I get this whenever I just enter make build, make test, etc, I get this:
C:\blender-git\blender>make test
No explicit msvc version requested, autodetecting version.
Compiler Detection failed. Use verbose switch for more information.
Visual Studio not found (try with the 'verbose' switch for more information)
I don’t know if this happens because I just have the Build Tools (with no IDE) installed or if I’m missing something in my PATH settings.
it uses the vs version and build flavor, to try and figure out your build folder, so you’d either have to run make full 2019b test of just go into your build folder and run CTest directly . In your case that likely be something like running test -C Release in your build_windows_x64_vc16_Release folder.
I was expecting some failures due to the lack of FFMPEG. 6 Failed Tests have “movie_clip” in their name, so I assume they have something to do with FFMPEG.
Just one also related to the compositor that I’m not so sure about: “node_keying_screen”
114/233 Test #114: compositor_input_test ................................................***Failed 2.29 sec
[==========] Running 16 tests from 1 test case.
[ RUN ] node_bokeh_image
[ RUN ] node_mask
[ RUN ] node_movie_clip_alpha
[ RUN ] node_movie_clip_angle
[ RUN ] node_movie_clip_image
[ RUN ] node_movie_clip_scale
[ RUN ] node_movie_clip_x
[ RUN ] node_movie_clip_y
[ RUN ] node_r_g_b
[ RUN ] node_texture_color
[ RUN ] node_texture_value
[ RUN ] node_time
[ RUN ] node_track_pos_speed
[ RUN ] node_track_pos_x
[ RUN ] node_track_pos_y
[ RUN ] node_value
[ OK ] node_bokeh_image
[ OK ] node_mask
Render result is different from reference image
[ FAILED ] node_movie_clip_alpha
Render result is different from reference image
[ FAILED ] node_movie_clip_angle
Render result is different from reference image
[ FAILED ] node_movie_clip_image
Render result is different from reference image
[ FAILED ] node_movie_clip_scale
Render result is different from reference image
[ FAILED ] node_movie_clip_x
Render result is different from reference image
[ FAILED ] node_movie_clip_y
[ OK ] node_r_g_b
[ OK ] node_texture_color
[ OK ] node_texture_value
[ OK ] node_time
[ OK ] node_track_pos_speed
[ OK ] node_track_pos_x
[ OK ] node_track_pos_y
[ OK ] node_value
[==========] 16 tests from 1 test case ran. (2218 ms total)
[ PASSED ] 10 tests.
[ FAILED ] 6 tests, listed below:
[ FAILED ] node_movie_clip_alpha
[ FAILED ] node_movie_clip_angle
[ FAILED ] node_movie_clip_image
[ FAILED ] node_movie_clip_scale
[ FAILED ] node_movie_clip_x
[ FAILED ] node_movie_clip_y
119/233 Test #119: compositor_matte_test ................................................***Failed 2.70 sec
[==========] Running 19 tests from 1 test case.
[ RUN ] double_edge_mask_adjacent_bleed_out
[ RUN ] double_edge_mask_adjacent_keep_in
[ RUN ] double_edge_mask_all_bleed_out
[ RUN ] double_edge_mask_all_keep_in
[ RUN ] node_box_mask
[ RUN ] node_channel_matte
[ RUN ] node_chroma_matte
[ RUN ] node_color_matte
[ RUN ] node_color_spill
[ RUN ] node_cryptomatte
[ RUN ] node_cryptomatte_legacy
[ RUN ] node_diff_matte
[ RUN ] node_distance_matte
[ RUN ] node_ellipse_mask
[ RUN ] node_keying
[ RUN ] node_keying_edge
[ RUN ] node_keying_matte
[ RUN ] node_keying_screen
[ RUN ] node_luma_matte
[ OK ] double_edge_mask_adjacent_bleed_out
[ OK ] double_edge_mask_adjacent_keep_in
[ OK ] double_edge_mask_all_bleed_out
[ OK ] double_edge_mask_all_keep_in
[ OK ] node_box_mask
[ OK ] node_channel_matte
[ OK ] node_chroma_matte
[ OK ] node_color_matte
[ OK ] node_color_spill
[ OK ] node_cryptomatte
[ OK ] node_cryptomatte_legacy
[ OK ] node_diff_matte
[ OK ] node_distance_matte
[ OK ] node_ellipse_mask
[ OK ] node_keying
[ OK ] node_keying_edge
[ OK ] node_keying_matte
Render result is different from reference image
[ FAILED ] node_keying_screen
[ OK ] node_luma_matte
[==========] 19 tests from 1 test case ran. (2635 ms total)
[ PASSED ] 18 tests.
[ FAILED ] 1 tests, listed below:
[ FAILED ] node_keying_screen
I attempted to google each (failed) test, but unfortunately, I couldn’t locate any relevant information. The generated report only provides the local result, the expected outcome, and a diff between the two.
Is there any info available on what each test does?
there’s no better description of what each test does, the name often gives a pretty good indication though.
The test do mostly assume you build with out default release configuration, and you’ll have some failing tests if you disable some functionality, out of curiosity why did you disable ffmpeg?
Ideally, I would build/configure Blender’s ffmpeg to exclude non-free components and keep it (like I think some Linux distros do when they include Blender in their repos).
As that seems more complicated than just removing it from Blender, I opted for the removal.