Hello everyone, I’m Ian and I’m excited to be working on adding new compositor regression tests for GSoC 2025!
Synopsis
This project will propose a set of improvements for compositor regression testing to expand code coverage. The compositor is being actively developed, so ensuring that new changes do not introduce bugs is very important for users. By adding new tests either individually or through scripting, the quality of the compositor testing suite will increase. The goal is to have a straightforward, simple, and useful feature implemented by the end of the program.
Benefits
Currently, the regression tests for the compositor cover all the nodes, but do not necessarily test the code of these nodes well. Many functions are not covered comprehensively, including some commonly used nodes. This may cause developers to miss bugs as changes are implemented. Improving code coverage will help reduce the occurrence of unseen bugs by allowing the regression tests to catch edge cases and broken code.
Deliverables
- Overview and audit of current code coverage, along with current regression tests
- New regression tests for multiple nodes
- Depending on the level of coverage needed, scripts to generate regression tests
- If time allows, converting some current regression tests to have a script as well, like with the mix node.
- End-user documentation
Project Details
Survey of Current Coverage
To prepare for code improvement, I will conduct a comprehensive overview of the current code coverage to identify areas where coverage is lacking. This will help me prioritize certain commonly-used nodes that may need more attention before moving onto lesser used ones. Part of this process will include running coverage with different tools and machines from both Mac and Windows to get a good sense of areas of overlap and differences that hardware may cause.
I have begun a preliminary look into coverage from my machine already and here are some of the findings:
Here they are in table form, sorted in order of function coverage:
| File Name | Function Coverage | Line Coverage | Region Coverage | Branch Coverage |
|---|---|---|---|---|
| node_composite_rgb.cc | 100.00% (4/4) | 100.00% (26/26) | 100.00% (6/6) | - (0/0) |
| node_composite_exposure.cc | 100.00% (5/5) | 100.00% (35/35) | 100.00% (7/7) | - (0/0) |
| node_composite_gamma.cc | 100.00% (5/5) | 100.00% (44/44) | 100.00% (7/7) | - (0/0) |
| node_composite_hue_sat_val.cc | 100.00% (5/5) | 100.00% (79/79) | 100.00% (8/8) | - (0/0) |
| node_composite_posterize.cc | 100.00% (5/5) | 100.00% (42/42) | 100.00% (7/7) | - (0/0) |
| node_composite_normal.cc | 100.00% (7/7) | 100.00% (65/65) | 100.00% (9/9) | - (0/0) |
| node_composite_normalize.cc | 100.00% (7/7) | 94.67% (71/75) | 88.89% (16/18) | 66.67% (4/6) |
| node_composite_mixrgb.cc | 11.11% (2/18) | 12.81% (31/242) | 5.26% (4/76) | 0.00% (0/58) |
| node_composite_map_value.cc | 11.76% (2/17) | 14.45% (25/173) | 12.50% (4/32) | 0.00% (0/6) |
| node_composite_convert_color_space.cc | 18.18% (2/11) | 17.56% (23/131) | 11.11% (4/36) | 0.00% (0/16) |
| node_composite_map_range.cc | 22.22% (2/9) | 36.52% (42/115) | 11.76% (4/34) | 0.00% (0/12) |
| node_composite_sepcomb_ycca.cc | 25.00% (4/16) | 24.90% (63/253) | 21.43% (12/56) | 0.00% (0/32) |
| node_composite_image_info.cc | 33.33% (2/6) | 21.21% (21/99) | 11.43% (4/35) | 0.00% (0/26) |
| node_composite_scene_time.cc | 33.33% (2/6) | 39.47% (15/38) | 28.57% (4/14) | 0.00% (0/4) |
| node_composite_sepcomb_hsva.cc | 40.00% (4/10) | 61.62% (61/99) | 72.73% (16/22) | - (0/0) |
| node_composite_sepcomb_rgba.cc | 40.00% (4/10) | 62.89% (61/97) | 72.73% (16/22) | - (0/0) |
| node_composite_sepcomb_xyz.cc | 40.00% (4/10) | 55.88% (38/68) | 57.14% (8/14) | - (0/0) |
| node_composite_sepcomb_yuva.cc | 40.00% (4/10) | 61.62% (61/99) | 66.67% (16/24) | - (0/0) |
| node_composite_bilateralblur.cc | 45.45% (5/11) | 31.65% (44/139) | 33.33% (11/33) | 14.29% (2/14) |
| node_composite_common.cc | 50.00% (1/2) | 67.86% (19/28) | 38.46% (5/13) | 16.67% (1/6) |
| node_composite_zcombine.cc | 50.00% (13/26) | 46.38% (160/345) | 48.39% (45/93) | 38.46% (20/52) |
| node_composite_value.cc | 50.00% (2/4) | 61.54% (16/26) | 66.67% (4/6) | - (0/0) |
| node_composite_pixelate.cc | 50.00% (5/10) | 35.05% (34/97) | 42.31% (11/26) | 20.00% (2/10) |
| node_composite_val_to_rgb.cc | 57.14% (8/14) | 47.80% (76/159) | 38.71% (12/31) | 0.00% (0/18) |
| node_composite_map_uv.cc | 58.82% (10/17) | 63.20% (146/231) | 59.57% (28/47) | 50.00% (11/22) |
| node_composite_colorbalance.cc | 60.00% (12/20) | 45.37% (142/313) | 27.59% (24/87) | 34.09% (15/44) |
| node_composite_invert.cc | 61.54% (8/13) | 76.99% (87/113) | 61.29% (19/31) | 33.33% (2/6) |
| node_composite_channel_matte.cc | 61.90% (13/21) | 48.29% (127/263) | 61.54% (32/52) | 55.56% (10/18) |
| node_composite_keyingscreen.cc | 63.64% (7/11) | 49.47% (47/95) | 54.55% (18/33) | 33.33% (4/12) |
| node_composite_cryptomatte.cc | 64.91% (37/57) | 57.47% (381/663) | 43.21% (156/361) | 41.67% (85/204) |
| node_composite_curves.cc | 65.38% (17/26) | 65.10% (194/298) | 68.63% (35/51) | 35.71% (5/14) |
| node_composite_alpha_over.cc | 66.67% (10/15) | 64.54% (91/141) | 69.81% (37/53) | 61.54% (16/26) |
| node_composite_distance_matte.cc | 66.67% (10/15) | 56.05% (88/157) | 54.84% (17/31) | 50.00% (8/16) |
| node_composite_crop.cc | 66.67% (12/18) | 54.23% (109/201) | 50.68% (37/73) | 34.21% (13/38) |
| node_composite_file_output.cc | 68.57% (24/35) | 48.17% (316/656) | 50.51% (148/293) | 56.91% (107/188) |
| node_composite_trackpos.cc | 70.00% (14/20) | 54.50% (115/211) | 51.19% (43/84) | 41.30% (19/46) |
| node_composite_brightness.cc | 70.00% (7/10) | 86.96% (80/92) | 81.82% (18/22) | 75.00% (3/4) |
| node_composite_bokehimage.cc | 71.43% (5/7) | 55.71% (39/70) | 77.78% (7/9) | - (0/0) |
| node_composite_rotate.cc | 71.43% (5/7) | 82.46% (47/57) | 47.37% (9/19) | 75.00% (6/8) |
| node_composite_image.cc | 72.22% (26/36) | 52.54% (290/552) | 52.86% (148/280) | 44.44% (72/162) |
| node_composite_colorcorrection.cc | 72.73% (8/11) | 48.10% (228/474) | 66.67% (36/54) | 94.44% (17/18) |
| node_composite_boxmask.cc | 73.68% (14/19) | 57.41% (151/263) | 58.33% (35/60) | 63.89% (23/36) |
| node_composite_math.cc | 75.00% (6/8) | 66.67% (50/75) | 41.38% (12/29) | 20.00% (2/10) |
| node_composite_sepcomb_color.cc | 76.92% (20/26) | 80.17% (372/464) | 58.76% (57/97) | 71.74% (66/92) |
| node_composite_composite.cc | 77.27% (17/22) | 72.69% (157/216) | 74.29% (52/70) | 70.59% (24/34) |
| node_composite_mask.cc | 77.78% (14/18) | 66.67% (72/108) | 69.57% (32/46) | 45.00% (9/20) |
| node_composite_antialiasing.cc | 77.78% (7/9) | 75.86% (44/58) | 52.94% (9/17) | - (0/0) |
| node_composite_setalpha.cc | 77.78% (7/9) | 86.76% (59/68) | 81.25% (13/16) | 83.33% (10/12) |
| node_composite_stabilize2d.cc | 77.78% (7/9) | 72.29% (60/83) | 54.84% (17/31) | 62.50% (10/16) |
| node_composite_sunbeams.cc | 77.78% (7/9) | 83.33% (105/126) | 87.50% (28/32) | 93.75% (15/16) |
| node_composite_ellipsemask.cc | 78.95% (15/19) | 67.55% (179/265) | 65.52% (38/58) | 71.88% (23/32) |
| node_composite_scale.cc | 79.17% (19/24) | 76.04% (165/217) | 70.45% (62/88) | 72.22% (26/36) |
| node_composite_viewer.cc | 8.70% (2/23) | 9.13% (22/241) | 4.88% (4/82) | 0.00% (0/42) |
| node_composite_denoise.cc | 80.00% (12/15) | 59.17% (129/218) | 48.48% (32/66) | 32.61% (15/46) |
| node_composite_switchview.cc | 80.00% (4/5) | 77.08% (37/48) | 87.50% (21/24) | 70.00% (7/10) |
| node_composite_diff_matte.cc | 80.00% (8/10) | 83.02% (88/106) | 86.67% (13/15) | 100.00% (2/2) |
| node_composite_luma_matte.cc | 80.00% (8/10) | 78.22% (79/101) | 83.33% (10/12) | - (0/0) |
| node_composite_translate.cc | 80.00% (8/10) | 81.40% (70/86) | 71.05% (27/38) | 70.00% (7/10) |
| node_composite_movieclip.cc | 81.25% (13/16) | 70.65% (142/201) | 68.06% (49/72) | 57.50% (23/40) |
| node_composite_despeckle.cc | 81.82% (9/11) | 87.12% (142/163) | 86.11% (31/36) | 81.25% (13/16) |
| node_composite_id_mask.cc | 81.82% (9/11) | 82.02% (73/89) | 73.33% (22/30) | 58.33% (7/12) |
| node_composite_chroma_matte.cc | 83.33% (10/12) | 84.48% (98/116) | 81.82% (18/22) | 100.00% (4/4) |
| node_composite_color_matte.cc | 83.33% (10/12) | 77.70% (108/139) | 91.30% (21/23) | 100.00% (10/10) |
| node_composite_levels.cc | 83.33% (10/12) | 61.70% (87/141) | 43.55% (27/62) | 60.87% (28/46) |
| node_composite_switch.cc | 83.33% (5/6) | 90.91% (30/33) | 81.82% (9/11) | 50.00% (1/2) |
| node_composite_transform.cc | 83.33% (5/6) | 93.06% (67/72) | 57.89% (11/19) | 87.50% (7/8) |
| node_composite_moviedistortion.cc | 84.62% (11/13) | 79.63% (86/108) | 75.76% (25/33) | 57.14% (8/14) |
| node_composite_split.cc | 84.62% (11/13) | 90.27% (102/113) | 93.94% (31/33) | 100.00% (12/12) |
| node_composite_color_spill.cc | 85.71% (12/14) | 65.09% (110/169) | 67.74% (21/31) | 40.00% (4/10) |
| node_composite_planetrackdeform.cc | 85.71% (18/21) | 68.94% (222/322) | 71.88% (69/96) | 60.00% (36/60) |
| node_composite_tonemap.cc | 85.71% (24/28) | 74.50% (222/298) | 67.57% (50/74) | 60.00% (18/30) |
| node_composite_texture.cc | 85.71% (6/7) | 77.92% (60/77) | 73.08% (19/26) | 50.00% (7/14) |
| node_composite_directionalblur.cc | 86.67% (13/15) | 81.72% (152/186) | 73.17% (30/41) | 64.29% (9/14) |
| node_composite_defocus.cc | 87.50% (28/32) | 75.20% (285/379) | 82.30% (93/113) | 70.00% (42/60) |
| node_composite_premulkey.cc | 87.50% (7/8) | 90.77% (59/65) | 87.50% (21/24) | 83.33% (10/12) |
| node_composite_flip.cc | 88.89% (8/9) | 92.13% (82/89) | 95.92% (47/49) | 87.50% (7/8) |
| node_composite_huecorrect.cc | 88.89% (8/9) | 89.11% (90/101) | 71.43% (10/14) | 0.00% (0/2) |
| node_composite_displace.cc | 90.00% (9/10) | 80.00% (164/205) | 79.07% (34/43) | 57.14% (16/28) |
| node_composite_bokehblur.cc | 91.67% (22/24) | 92.65% (290/313) | 91.67% (77/84) | 85.71% (36/42) |
| node_composite_cornerpin.cc | 92.31% (12/13) | 80.98% (166/205) | 81.13% (43/53) | 67.65% (23/34) |
| node_composite_filter.cc | 92.31% (12/13) | 79.70% (161/202) | 82.46% (47/57) | 71.15% (37/52) |
| node_composite_blur.cc | 92.31% (24/26) | 85.96% (355/413) | 79.69% (102/128) | 74.36% (58/78) |
| node_composite_kuwahara.cc | 93.10% (27/29) | 93.67% (622/664) | 85.14% (126/148) | 80.26% (61/76) |
| node_composite_double_edge_mask.cc | 93.33% (14/15) | 95.40% (228/239) | 93.94% (62/66) | 95.24% (40/42) |
| node_composite_dilate.cc | 93.33% (28/30) | 91.67% (319/348) | 84.16% (85/101) | 80.36% (45/56) |
| node_composite_glare.cc | 93.50% (115/123) | 94.62% (1724/1822) | 88.74% (331/373) | 87.37% (173/198) |
| node_composite_lensdist.cc | 93.55% (29/31) | 88.50% (254/287) | 84.38% (81/96) | 65.00% (26/40) |
| node_composite_vec_blur.cc | 93.55% (29/31) | 94.48% (462/489) | 92.73% (102/110) | 90.74% (49/54) |
| node_composite_keying.cc | 94.12% (32/34) | 93.27% (554/594) | 87.26% (137/157) | 79.35% (73/92) |
| node_composite_inpaint.cc | 94.44% (17/18) | 97.33% (255/262) | 96.55% (56/58) | 93.33% (28/30) |
As can be seen from this outline, broadly speaking, the color and converter type nodes are most lacking in coverage support. The color_ramp/val_to_rgb node in particular is a commonly used node that is lacking in coverage.
Whatever the case, the reasons for why coverage is lacking can be due to a variety of factors. Each node has different parameters, functions, and coverage holes. Some nodes have certain input combinations that are not tested. Others have functionality that depends on hardware, like functions that run for gpu, or in certain edge cases that haven’t been covered. Part of this survey will be understanding the reasons for lack of coverage for each node, meaning I will study the codebase of the compositor in-depth to identify the tests needed.
By the end, I should have a solid understanding of both the compositor code and the compositor regression tests, as well as a preliminary list of nodes to add tests to, along with an order of priority.
Add Individual Regression Tests
For nodes with particular edge cases or a small number of cases, I can add individual regression tests manually. This process will be straightforward, as I am mostly adding new .blend files with specific node setups. As this project’s goal is primarily to expand test coverage, this step is the most important. I will prioritize improving coverage with a wide breadth, rather than aiming to get 100% coverage for each node and risk not getting to as many as possible.
Nodes that are commonly used such as the mix and glare nodes, as well as new nodes like the texture node will be prioritized over less-used nodes.
Some examples of nodes I could improve are the mix_rgb, pixelate, and convert_color_space nodes. The z-combine node for example, lacks coverage for settings like anti-aliasing, which should be a straightforward set of tests to add.
Currently, there is an open issue: #134223 for compositor testing, which has many useful comments and discussion I plan to (and have already) engaged with to aid with this contribution. My PR addressing an improvement to the Glare Node regression tests 138587 has been merged as well. I have already started on adding a test for the Z-combine node as well.
Add Scripted Regression Tests
For certain nodes with many permutations of parameters, it may make more sense to generate tests with a python script to make modifications easier. A script is already used to generate tests for some nodes, like the one used in node_vector_math_ADD.blend. Thus, existing scripts can be easily adapted for similar cases.
I don’t believe there should be many nodes that require a large breadth of test combinations. If time allows, the mix_rgb node is one candidate I could optimize. Currently, there are 18 different individual blend files that test each simple input. However, this will largely depend on necessity and thus is low on the priority list for me. This can be a point of further discussion with mentors and their advice on the best direction for improving regression testing.
Project Schedule
This is an easy size project (90 hours). From June 2 – Sep 1, I plan to spend around 10 hours a week on each deliverable, with plenty of time for quality code review, bug-fixes, documentation, and other necessary adjustments if a test requires more work. 3 of the weeks will be set aside for my final exams and travel*
| Date | Tasks |
|---|---|
| May 8 – June 1 (Bonding Period) | * Meet with mentors to solidify proposal and deliverables, create outline for expectations |
| June 2 – June 21 (3 weeks) ~ 20 hours | * Complete survey and report findings to mentors, with a small roadmap of nodes that I will target. June 9 – 12 is finals week for me so I have set aside some time for exams |
| June 23 – June 28 (1 week) ~10 hours | * Add regression test(s) |
| June 30 – July 5 (1 week) ~ 10 hours | * Add regression test(s) |
| July 7 – July 12 (1 week) ~ 10 hours | * Add regression test(s) |
| July 14 – July 19 (1 week) ~ 10 hours | * Add regression test(s) or extra week for catch-up if behind, clean up and bug fixes, documentation. Meet with mentors about project direction, midterm evaluation |
| Aug 11 – Aug 16 (1 week) ~ 10 hours | * Add regression test(s) |
| Aug 18 – Aug 23 (1 week) ~ 10 hours | * Add regression test(s) |
| Aug 25 – Aug 30 (1 week) ~ 10 hours | * Final week for bug fixes, documentation, and catch-up if behind |