Now that blender 2.8 has gotten into a stabilizing phase, I’d like to once more bring attention to our unit tests.
Given the fast pace 2.8 had been moving in the past they have been neglected (and that is ok) but i feel the time has come to bring them back where they need to be.
I ran them earlier today (with latest updates from svn for the test files) on win64 and the results we somewhat horrific
77% tests passed, 23 tests failed out of 101
Cycles nearly 200 failures mostly small noise differences but also some larger looking disparities between the reference and the tests, and a single crash on one of the renders as well.
If there’s any windows specific issues I’d be happy to look at them but at this point I’m unable to tell what is actually broken due to windows/compiler oddities or what has just bit-rotted away.
I’ll make a blender2.7 branch for the Cycles tests, then we can update the reference images for master. Most of the failing tests are expected differences, though a few still need to be fixed.
Whoa, this message prompted me to go take a peek as I have a personal addon that sucks to test. Perhaps I’ll even add a few tests myself for blender proper along the way.
Today, in things that will nicely surprise you: I had no idea that bpy.types.Mesh.unit_test_compare existed! This will be useful
Today, in things that will completely terrify you: Looks like there’s some tests that fail, but don’t really fail. The very important new_from_object call fails but test is marked as Passed (happens in various other tests too; Needs Exception handling)
11/21 Testing: bmesh_split_faces
11/21 Test: bmesh_split_faces
Command: "C:/Users/redact/Source/Repos/blender-git/build_windows_Full_x64_vc15_Release/bin/Debug/blender.exe" "--background" "-noaudio" "--factory-startup" "--env-system-scripts" "C:/Users/redact/Source/Repos/blender-git/blender/release/scripts" "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend" "--python-text" "run_tests"
Directory: C:/Users/redact/Source/Repos/blender-git/build_windows_Full_x64_vc15_Release/tests/python
"bmesh_split_faces" start time: Mar 14 14:23 Pacific Daylight Time
Output:
----------------------------------------------------------
Blender 2.80 (sub 48) (hash 81a03e17f759 built Thu 03/14/2019 11:38 AM)
found bundled python: C:\Users\redact\Source\Repos\blender-git\build_windows_Full_x64_vc15_Release\bin\Debug\2.80\python
Read blend: C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend
WARN (bke.customdata): c:\users\redact\source\repos\blender-git\blender\source\blender\blenkernel\intern\customdata.c:3655 CustomData_verify_versions: .blend file read: removing a data layer that should not have been written
WARN (bke.customdata): c:\users\redact\source\repos\blender-git\blender\source\blender\blenkernel\intern\customdata.c:3655 CustomData_verify_versions: .blend file read: removing a data layer that should not have been written
Error: Engine 'CYCLES' not available for scene 'Manual' (an add-on may need to be installed or enabled)
[==========] Running 17 tests from 2 test case.
[----------] 15 tests from Manual
[ RUN ] SmoothSuzanne.30
Traceback (most recent call last):
File "C:/Users/redact/Source/Repos/blender-git/blender/release/scripts\modules\addon_utils.py", line 351, in enable
mod = __import__(module_name)
ModuleNotFoundError: No module named 'cycles'
Traceback (most recent call last):
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 192, in <module>
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 187, in main
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 160, in run_all_test_cases
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 125, in run_test_case
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 99, in run_test
File "C:/Users/redact/Source/Repos/blender-git/blender/../lib/tests/modeling/split_faces_test.blend\run_tests", line 52, in test_object
TypeError: BlendDataMeshes.new_from_object(): takes at most 4 arguments, got 5
Blender quit
<end of output>
Test time = 1.53 sec
----------------------------------------------------------
Test Passed.
"bmesh_split_faces" end time: Mar 14 14:23 Pacific Daylight Time
"bmesh_split_faces" time elapsed: 00:00:01
----------------------------------------------------------
I will look at making some of the non-cycles one pass. (I’d already done that for the bevel and boolean regression tests, which are good examples of using bpy.types.Mesh.unit_test_compare).
Cool. There’s another one though: modifier_stack/array_test is similarly broken with the same new_from_object call.
Now’s a good time to ask in case I do want to contribute in the future: What’s the tradeoff between external py files and embedding in the .blend like this? Obviously there’s this issue that makes it difficult to search etc. (as well as a bit of duplication in the script logic) Was embedding in the .blend done for another reason? Hypothetically, if one were to extract out those .py’s and offer that for a commit, would that be accepted?
Lucky find there! The test output for that one is different for me so I didn’t notice it was the same problem :-/ But yeah, seems to be the same issue of new_from_object
Took a quick peek at the crash, looks like there’s still some code paths inside BlenderSync::sync_curves that count on UV’s having float3’s. @StefanW mind taking a look there? Repro just render T50634_2.blend from the unit tests in a debug build and you’ll hit an assert.
