Hello all, I’m Ian. This is a proposal draft addressing improvements to regression testing for the compositor nodes. Please feel free to add any suggestions or improvements or criticisms. I’m all ears!
More detailed information is in this proposal
Synopsis
Regression testing is an important aspect of Blender’s development. Currently, the regression tests for the compositor cover all the nodes, but do not necessarily test the code of these nodes well. Many functions are not covered comprehensively, including some commonly used nodes. This may cause developers to miss bugs as changes are implemented. Improving code coverage will help reduce the occurrence of unseen bugs by allowing the regression tests to catch edge cases and broken code.
This project will propose a set of improvements for compositor regression testing to expand code coverage. The compositor is being actively developed, so ensuring that new changes do not introduce bugs is very important for users. By adding new tests either individually or through scripting for nodes with multiple cases, the quality of the compositor testing suite will be improved. The goal is to have a straightforward, simple, and useful feature implemented by the end of the program.
Deliverables
- Overview and audit of current code coverage, along with current regression tests
- Z-combine, mix_rgb, pixelate, and convert_color_space regression tests
- Scripts to generate regression tests for many cases, such as with color_ramp
- If time allows, converting some current regression tests to have a script as well, like with the mix node.
- End-user documentation
Survey of Current Coverage
To prepare for code improvement, I will conduct a comprehensive overview of the current code coverage to identify areas where coverage is lacking. This will help me prioritize certain commonly-used nodes that may need more attention before moving onto lesser used ones. Part of this process will include running coverage with different tools and machines from both Mac and Windows to get a good sense of areas of overlap and differences that hardware may cause.
I have begun a preliminary look into coverage from my machine, but some of these numbers may be incorrect due to lack of dedicated GPU hardware. Broadly speaking, the color and converter type nodes are most lacking in coverage support. The color_ramp/val_to_rgb node in particular is a commonly used node that is lacking in coverage.
As part of the survey, I will also study the current regression tests that are already implemented, firstly to prevent unnecessary redundancy as well as any scripts to reference and adapt for easier test generation. The node_vector_math_ADD.blend file is an example of scripting I can adapt for nodes with many parameters and cases to be tested, like the color_ramp node.
By the end, I should have a solid understanding of the compositor regression suite, as well as a preliminary list of nodes to add tests to, as well as an order of priority.
Add Individual Regression Tests
For nodes with particular edge cases or a small number of cases, I can add individual regression tests manually. This process will be straightforward, as I am mostly adding new .blend files with specific node setups. This project’s goal is primarily to expand test coverage, so this step is the most important. I will prioritize improving coverage with a wide breadth.
As of now, I am considering the mix_rgb, pixelate, z-combine, and convert_color_space nodes for adding individual regression tests:
-
The mix_rgb node has many parameters, but they are largely already implemented, so I do not expect many files to be needed. This node however may be a candidate to optimize by creating a script to generate the tests, like the vector_math nodes.
-
The convert_color_space node likewise also has many parameters, but from a brief look at the code coverage, it seems that much of the functions are related to cpu and gpu operations. I will work on adding individual tests to see more specifically which cases need to be triggered.
-
The z-combine node lacks coverage for settings like anti-aliasing, which should be a straightforward set of tests to add.
More tests may be approached in a similar way, depending on the depth of the survey and priorities.
Add Scripted Regression Tests
As mentioned before, it would make more sense for certain node tests to be generated with a python script to make modifications easier. A candidate for this is the color_ramp node. With this node in particular, I will adapt a script similar to the one used in node_vector_math_ADD.blend to generate each case with a different color mode and interpolation.
If time allows, the mix_rgb node is a candidate to optimize by creating a script to generate the tests, like the vector_math nodes. Currently, there are 18 different individual blend files the test each simple input. This is low on the priority list however.
Project Schedule
This is an easy size project (90 hours). From June 2 – Sep 1, there are ~13 weeks of the program. I plan to spend around 7-8 hours a week on each deliverable, with plenty of time for quality code review, bug-fixes, documentation, and other necessary adjustments if a deliverable requires more work.
| Date | Tasks |
|---|---|
| May 8 – June 1 (Bonding Period) | * Get to know mentors, continue to familiarize myself with Blender and the community, solidify proposal and deliverables, create outline for expectations + start coverage survey |
| June 2 – June 7 (1 week) | * Complete Z-combine regression test |
| June 9 – June 21 (2 weeks) | * Complete regression test |
| June 23 – July 5 (2 weeks) | * Complete regression test |
| July 7– July 19 (2 weeks) | * Complete regression test |
| July 21 – July 26 (1 week) | * Extra week for catch-up if behind, clean up and bug fixes, documentation |
| July 28 – Aug 9 (2 weeks) | * Complete regression test |
| Aug 11 – Aug 23 (2 weeks) | * Complete regression test |
| Aug 25 – Sep 1 (1 week) | * Final week for bug fixes, documentation, and catch-up if behind |
Particular areas I’m seeking some feedback are:
- Specific nodes or tests to be included or prioritized
- Suggestions on the project structure or an area I should consider revising
- Any other pressing improvements to the regression testing or extensions of its framework you would like me to consider!
Note: The schedule of the project is a little ambiguous at the moment because some tests will be very straightforward to add, while others may vary wildly depending on the type and unforeseen errors. I just put a deliverable of regression tests every other week for now but that is definitely going to change. + I will continue adding nodes to the list as I study their coverage flaws and get a general understanding of what to add