Buildbot asserts, linting, sanitizer

Technical Feedback
1

A few changes have been made on the Buildbot.

  • Non-package builds now enable asserts, to catch more bugs in tests. This works for daily and patch builds. Packaged builds do not have asserts as this would affect performance.
  • Daily and patch builds now have a lint builder, which currently checks that make format produces no differences. This is indicated as a failed build (there is no mechanism in Gitea to have just a warning). Let me know if this gets in the way too much. More checks could be enabled, they would have to be reasonably fast and easy to keep always passing. A single script in the tools/ directory to run all such checks would be helpful, then I can hook it up to the Buildbot.
  • Pull requests and commit status now shows for individual platforms instead of only the coordinator.
  • In the Buildbot admin interface, there is now an option to use build configuration “sanitizer” in addition to the existing “release” and “debug”. This enables RelWithDebInfo + asserts + address sanitizer on Linux and macOS. However it is currently failing in build and tests. Anyone is welcome to try to get the build and tests passing so this can become useful in practice and run more regularly. How often is still unclear, build time overhead would need to be checked.
10 Likes
2

Great news Brecht, thanks a lot! :smiley:

Did you use our ASAN/LSAN suppression rules ? FYI am using this on my machine to run tests with ASAN:

LSAN_OPTIONS=print_suppressions=false:suppressions=/home/guest/blender/main/src/tools/config/analysis/lsan.supp ASAN_OPTIONS=allocator_may_return_null=true ctest

But even with these, we still have a mysterious reported leak in libstdc++ with no clear call-back, and I do not know a way to filter this one out (without also filtering out a lot of potential valid errors that is)…

I’ll have a look at a build log too later today.

3

I didn’t use those rules. If these are needed, they should be used enabled by default in CMake instead, so anyone running this locally sees the same thing as the buildbot. We already have on lsan option in add_blender_test_impl, these could be added there as well.

4

They are needed for the tests (unless you use leak_check_at_exit=0, which means no ‘memory leak’ error from ASAN will make tests fails, which would be less useful imho).

Regarding build failing, at least on linux builder it seems that there is simply no ASAN installed (or it’s not ‘imported’ into the chroot or something like that):

usr/bin/ld.gold: error: cannot open libasan_preinit.o: No such file or directory
usr/bin/ld.gold: error: cannot find -lasan
usr/bin/ld.gold: error: cannot find -lubsan
5

The use of LSAN_OPTIONS=exitcode=0 already effectively disabled tests failing due to leaks. I agree using suppressions is better.

Here is a PR that adds your options:

This was fixed.

1 Like
6

Here is quick update on the new sanitizer and debug builds:

  • exitcode=0 has been removed from the CMake configuration, along with other tweaks and fixes. This means that the buildbot are now able to perform meaningful checks/tests when using the sanitizer type of build.
  • The sanitizer build now fully passes for main branch for Linux and MacOS x86_64 builds.
  • MacOS arm64 compiles, but has one weird failure in modifiers tests reported here, for both sanitizer and debug builds.
  • Windows builds are sometimes failing at compile step (compiler crash), this needs to be investigated further. There is no support for sanitizer tools on Windows currently anyway.
  • When they compile, windows build systematically fail tests (for both debug and sanitizer), seems to be an issue with python binding of USD . No idea why this does not seem to affect release builds?