2.8 Unit tests

Now that blender 2.8 has gotten into a stabilizing phase, I’d like to once more bring attention to our unit tests.

Given the fast pace 2.8 had been moving in the past they have been neglected (and that is ok) but i feel the time has come to bring them back where they need to be.

I ran them earlier today (with latest updates from svn for the test files) on win64 and the results we somewhat horrific

77% tests passed, 23 tests failed out of 101

Cycles nearly 200 failures mostly small noise differences but also some larger looking disparities between the reference and the tests, and a single crash on one of the renders as well.

If there’s any windows specific issues I’d be happy to look at them but at this point I’m unable to tell what is actually broken due to windows/compiler oddities or what has just bit-rotted away.

I’ll make a blender2.7 branch for the Cycles tests, then we can update the reference images for master. Most of the failing tests are expected differences, though a few still need to be fixed.

Whoa, this message prompted me to go take a peek as I have a personal addon that sucks to test. Perhaps I’ll even add a few tests myself for blender proper along the way.

Today, in things that will nicely surprise you: I had no idea that bpy.types.Mesh.unit_test_compare existed! This will be useful :slight_smile:

Today, in things that will completely terrify you: Looks like there’s some tests that fail, but don’t really fail. The very important new_from_object call fails but test is marked as Passed (happens in various other tests too; Needs Exception handling)

11/21 Testing: bmesh_split_faces
11/21 Test: bmesh_split_faces
Command: "C:/Users/redact/Source/Repos/blender-git/build_windows_Full_x64_vc15_Release/bin/Debug/blender.exe" "--background" "-noaudio" "--factory-startup" "--env-system-scripts" "C:/Users/redact/Source/Repos/blender-git/blender/release/scripts" "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend" "--python-text" "run_tests"
Directory: C:/Users/redact/Source/Repos/blender-git/build_windows_Full_x64_vc15_Release/tests/python
"bmesh_split_faces" start time: Mar 14 14:23 Pacific Daylight Time
Blender 2.80 (sub 48) (hash 81a03e17f759 built Thu 03/14/2019 11:38 AM)
found bundled python: C:\Users\redact\Source\Repos\blender-git\build_windows_Full_x64_vc15_Release\bin\Debug\2.80\python
Read blend: C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend
WARN (bke.customdata): c:\users\redact\source\repos\blender-git\blender\source\blender\blenkernel\intern\customdata.c:3655 CustomData_verify_versions: .blend file read: removing a data layer that should not have been written
WARN (bke.customdata): c:\users\redact\source\repos\blender-git\blender\source\blender\blenkernel\intern\customdata.c:3655 CustomData_verify_versions: .blend file read: removing a data layer that should not have been written
Error: Engine 'CYCLES' not available for scene 'Manual' (an add-on may need to be installed or enabled)
[==========] Running 17 tests from 2 test case.
[----------] 15 tests from Manual
[ RUN      ] SmoothSuzanne.30
Traceback (most recent call last):
  File "C:/Users/redact/Source/Repos/blender-git/blender/release/scripts\modules\addon_utils.py", line 351, in enable
    mod = __import__(module_name)
ModuleNotFoundError: No module named 'cycles'
Traceback (most recent call last):
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 192, in <module>
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 187, in main
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 160, in run_all_test_cases
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 125, in run_test_case
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 99, in run_test
  File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 52, in test_object
TypeError: BlendDataMeshes.new_from_object(): takes at most 4 arguments, got 5

Blender quit
<end of output>
Test time =   1.53 sec
Test Passed.
"bmesh_split_faces" end time: Mar 14 14:23 Pacific Daylight Time
"bmesh_split_faces" time elapsed: 00:00:01

I will look at making some of the non-cycles one pass. (I’d already done that for the bevel and boolean regression tests, which are good examples of using bpy.types.Mesh.unit_test_compare).

I fixed the bmesh_split_faces one. Thanks for noticing the pass-that-was-really-a-failure.

Cycles tests were updated to master now. I didn’t get any crash, that might be Windows or compile flag specific.

Cool. There’s another one though: modifier_stack/array_test is similarly broken with the same new_from_object call.

Now’s a good time to ask in case I do want to contribute in the future: What’s the tradeoff between external py files and embedding in the .blend like this? Obviously there’s this issue that makes it difficult to search etc. (as well as a bit of duplication in the script logic) Was embedding in the .blend done for another reason? Hypothetically, if one were to extract out those .py’s and offer that for a commit, would that be accepted?

Lucky find there! The test output for that one is different for me so I didn’t notice it was the same problem :-/ But yeah, seems to be the same issue of new_from_object

85% tests passed, 15 tests failed out of 101

Quite the improvement since yesterday, cycles is down to 18 failures (link in opening post updated) I’ll take a look at the crash over the weekend.

Took a quick peek at the crash, looks like there’s still some code paths inside BlenderSync::sync_curves that count on UV’s having float3’s. @StefanW mind taking a look there? Repro just render T50634_2.blend from the unit tests in a debug build and you’ll hit an assert.

I can reproduce, I’ll take a look at it on monday.

I just ran tests on latest build, got 73 out of 102 Failing

Link to log and cycles report data: https://drive.google.com/open?id=1IqNpFwBLk5kL8-vJfn3w7ZrVvxcsXbPe

My process:

  • In blender/
    • make update
    • make full
  • In build_darwin_full/
    • ctest -C Debug/Release -V -V > testing.3.13.2019.13.02.00.log

Common message I see is:
“dyld: Library not loaded: @executable_path/…/Resources/lib/libomp.dylib”
But that file seems to be in the right place.

Screen grab of my dev directory structure is in folder linked above, with file hilighted

Any pointers on running tests correctly on MacOS?

System Specs:
OS: MacOS 10.14.3
CPU: 2.3 GHz Intel Core i5
Graphics: Intel Iris Plus Graphics 655 1536 MB
RAM: 16 GB 2133 MHz LPDDR3

did you actually run ctest -C Debug/Release -V -V &gt; testing.3.13.2019.13.02.00.log ?

cause you have to pick either Debug OR Release depending on the build you made, judging by the other commands you did, this should be Release

Made the adjustment, but same results.
73 out of 102 failing tests.
Log posted to: https://drive.google.com/open?id=1IqNpFwBLk5kL8-vJfn3w7ZrVvxcsXbPe

Updated process:

  • In blender/
    • make update
    • make full
  • In build_darwin_full/
    • make install
    • ctest -C Release -V -V > testing.3.13.2019.13.02.00.log