GSoC 2025: Regression testing (2x ideas)

Contributing to Blender Summer of Code
,
1

Hi,

in the scope of my GSoC 25 application I have some questions on the Regression Testing briefs. Mainly regarding the current testing setups and the roadmaps to improve them.

I am trying to weigh up the usage usage of .blend files against any alternatives.
What is regarded as improvement vs unnecessary files/ complexity etc.
And maybe; Are there different opinions (pros/ cons) on that ?

  1. .blend files are mostly preferred for test cases correct ?
    So its easier to add, edit, delete tests cases (?)

  2. Is there a limit of .blend files ? how many are enough ? When are there too many ?
    Is there a defined un-written limit of .blend files. e.G. when a good balance is reached between files added and cases covered for example ?
    Could it be limited by the time it takes to run the tests ?
    I was surprised how quickly a single headless Blender session, opens and executes a series of .blend files (*unless I got the wrong impression on how the setup works currently).
    Are you aware of any limit that would be some kind of ceiling to stay away from ?
    e.G. not more that 30 sec. to run on test module xyz. Or not more than 100 files etc. ?

  3. Is there motivation to move more towards a scripted setup ?
    either by having those tests generated ? Or having those test exist as python code for example ? (*links to 1st question in a way)
    I understand .blend case > .py case since its easier to add, edit, remove.
    Would that be true even if .blend(s) are harder to track, review and maybe run a little slower (file opening) ?
    Is there an unwritten .blend file when possible - scripted when necessary rule ?
    With scripted I mean either a script that generates more temporary (?) blend files or holds test cases in code form that are harder to replicated in blend files (e.G. Node creation for example)

2

@izo these mainly came up from going through #134223

@HooglyBoogly I’d think this is mostly similar to the Geometry Nodes Brief right ?

3

Too many questions to answer everything in detail, but in short we don’t have hard requirements about run time or blend file sizes. Tests should run as fast as possible and be as small as possible. Recently we agreed that a run time of 3s for ~10 new test cases for the file output node is reasonable.

  1. Is there motivation to move more towards a scripted setup ?

Regression tests are meant to detect unwanted changes only, therefore, the current approach is good enough. Other types of testing such as functional testing, where you typically test against known requirements, while allowing the end result to vary, can be implemented with Python scripts, google tests in C++ or other frameworks.

1 Like
4

okay, so for the compositor case, the GSoC’25 idea roughly would be;

  • primarily to add more test cases as .blend files.
  • optional add scripts that help add (generate) those .blend files
  • optional/ alternatively come up with different alternative other setups

*Thanks this helps with checking if I interpreted it correctly

5

Yes, those would be valid solutions. To avoid adding tests with no value, it’s important to add tests that cover typical user use cases (and not just arbitrary node/parameter combinations that improve code coverage by a fraction)

1 Like