Hello, we are a group of software engineering master’s students from LUT University in Finland. We are doing a project on maintaining software and we chose Blender as our topic. We have some questions we’d like to ask about the process of developing and maintaining Blender, mainly on testing activities and metrics.
Are there any testing requirements for plugins/libraries?
We noticed that automated testing is disabled on buildbot, why?
How often is manual testing done? What is the coverage policy for manual testing, full regression or just modified parts?
On metrics, we can analyze the development process by looking at Maniphest and Differential data, but is there data gathered and available from the software itself, i.e. analytics?
What lead to the major spike in development and reporting activity in 2018?
Again, thank you! We’re contrasting manual testing to automated testing, whether that means running specific unit tests manually or doing exploratory testing by using the modified/implemented feature. We are interested in if the code review process is enough for software of this scale to maintain acceptable stability.
Automated testing for Blender development should be improved a lot, and more effort on it will improve stability. It’s a matter of balancing priorities.
For manual testing, of course that happens when making changes to some area. You do some tests to see if everything is still working, on the modified parts and other areas that you think might be affected. “Full regression” is a bit unclear given that Blender is so huge, there’s only a tiny fraction that can be manually tested in a reasonable amount of time.