GSoC 2020: Regression Testing Framework - Weekly Reports

This will contain the weekly reports, I just came to know my branch is available publicly, it is called soc-2020-testing-frameworks and can be viewed here.

Week 1 (18-24 May):

  • Went through the Generate and Deform modifiers for surveying, as to how the framework should be built with special focus on helper objects and vertex groups.

  • Survey of mesh operators ( focus on user interaction )

  • Checked reproducible(ness) of the modifiers.

  • Support for other object types like Curves, surface, text.

  • Simplified inspection of Physics modifiers.

  • Build a working prototype for the automatic creation of blend objects.
    Sheet 3: Survey

Week 2 (June 1-7):
Plan of Action:

  • Complete the prototype to make it usable by fellow developers.
  • Plan next deliverable based on the survey.
5 Likes

Week 2 (June 1-7):

  • Finished with the prototype, the main aim for this week was to make it work, the improvements would be done in the following week.
  • Added support for generating vertex groups from the script.

Week 3 (June 8-14):
Plan of Action

  • Polish the current prototype
  • Identify tests by a unique name id.
4 Likes

It’s been a crazy week, I fell a little sick. Here are the updates-
Week 3 (June 8-14):

  • Made improvements to the prototype (can now be called a workable script), can be used by other developers.
  • Added the functionality to add unique names and check for unicity.

Week 4 (June 15-21):
Plan of Action

  • Add unique test names to modifiers and operators (working on it).
  • Add remaining tests for Modifiers based on the survey.
  • Add tests for Physics modifiers
3 Likes

This week was a bit slow.

Week 4 (June 15 - 21):

  • Added the test names to operators and modifiers
  • Meeting with mentors
  • Deliverable 2 started towards the end of this week.
  • Started adding tests for mesh modifiers (Skin,…to be continued)

Week 5 (June 22 - 28):

  • Finish the mesh modifiers
  • Start with Curves Modifiers
2 Likes

Week 5 (June 22-28):

  • Worked on including remaining mesh modifiers.
    The ones I was trying to test required “Object Operators”, so I added support for modifiers which require operators. e.g “Bind” for Surface Deform

  • Added default tests for Wave and Ocean, and dummy* tests for Surface deform and mesh deform.

*Dummy test - there is no actual deformation, hence the test cannot fail, which is necessary.

Week 6 (June 29-July 5):

  • Add support for imitating user input, using shape keys or animation.
  • Covering the remaining of mesh modifiers. (make real tests as opposed to dummy ones).
    Will cover Hook, Surface deform, Mesh deform, Warp, Laplacian Deform.

*Note 1- this does not involve Physics Modifiers, they would be covered separately.
*Note 2- Curves modifiers has been shifted to next week.

9 Likes

I just wanted to drop in and thank you for working on this, testing in general is so under appreciated! Keep up the good work :slight_smile:

Also apologies. I posted this in the wrong thread.

5 Likes

Week 6 (June 29-July 5):

  • Yayy! Finally done with mesh modifiers*. Now we can actually test the deformation using the animation and shape keys. So I have included the tests for the above-said modifiers. (look at the previous report)
    Idea credits @deadpin

Plan of Action

Week 7 (July 6-July 12):

  • This week will be dedicated to extend the framework for Curves (pretty much in sync with the timeline)
  • Improvements in the work done until now based on the review(s).

*Note- this doesn’t include Physics modifier, and there were some complications with Remesh and Smooth Corrective (they will either be skipped for now or worked upon in the background)

4 Likes

Week 7 (July 6-July 12):
It was a slow week, some college stuff came up and there were some personal issues. Also I got stuck on the review like having a coder’s block, I have discussed with the mentor, I did get some idea still got nothing working.
And I recently pushed the code to my remote branch.
Will update if something significant happens over the weekend.
Weekend-

  • added the run_test_using_name feature to the test cases and made some minor improvements as per the review.
  • as of now we can have tests that can imitate user input and support object operators, it is fully functional now (it is being reviewed again as I type this)

Next plan

  • Get started with building the framework for testing Curves Modifiers.
5 Likes

Week 8 (July 13-July 19):

  • Extended the framework, rather made it generic, to be able to test Curves
  • Added test cases for Curve Generate Modifiers
  • Updated the remote branch

Plan of Action:

  • add tests for the Deform Curves
  • start with Physics modifiers
  • reviewing/improving the newly added code
8 Likes

I have lot of things to take care of this week, so documenting alongside, this is the ongoing week.
Week 9 (July 20-July 26):

  • Tried testing Fluid modifiers, it is giving different “baked” result when testing inside blender as compared to testing it using a script. Found, it was because the test objects were produced in a different blender version and were being tested in a different blender version. Therefore the test was failing.
  • Adding Deform tests, Armature modifier for Curves cannot be assigned to Automatic weights (hence no vertex groups are created) so Bone envelopes is used.
  • New developments - Wave modifier is not regressable, it fails randomly. It passes after I freshly add it but fails on the next day or the day after. I suspect same thing as Fluid, different versions of Blender.
  • Worked with the Particle system, it is not reproducible as far as I tested, will give a final conclusion soon.

Plan of Action

  • The plan is to finish with Physics modifiers, firstly, I am planning to test Explode modifier, but it requires Particle system, so if that is reproducible, I will write an interface to interact with it, and write a test for it which will cover both I guess.
  • And I have some research paper work to take care of in the upcoming weeks, I will try my best to finish with the Deliverable 2 this week only and probably a meeting with the mentors(after finishing up I guess).
2 Likes

Hello Calra, I have a script available that can automate the process of rigging any arbitrary curve with an armature and envelope weights. In my addon the file ops_misc.py has a CreateBonesFromCurve operator that does this. When the addon is installed, this operator is available from the Add Armature menu when a curve is selected. Please feel free to use it if it will make creating the tests easier.

Thanks for making Blender better :slight_smile:

@Josephbburg I will have a look at it, if I would like to add more complicated tests. But I would like to keep it simple, and don’t mind a bad-rigged curve for testing :wink: , the modifier should work correctly, that’s what I am after. Thank you for both the things : )

1 Like

Week 10 (July 27-Aug 3):

  • Added a class and test for Particle systems, reproducible with a script.
  • Added a class and test for Dynamic Paint, its combined with Fluids.
  • Had a meeting with mentors, there have been changes with the next Deliverable and how to spend the next 3-4 weeks.
  • Had another meeting with Habib to discuss possible implementation of creating a generic class which can handle Physics modifiers as well.

Plan of Action for the upcoming week

  • To clean and polish the framework, refactoring the existing classes and combining all of them and creating a new Modifier Spec that can handle Physics modifiers.
  • Main focus would be on to solve the problem of parsing through a nested dictionary to apply the modifier parameters.
4 Likes

Week 11 (Aug 4- Aug 9):

  • Added a generic support for nested parameters.
  • Updated the bake method, and mer
  • Added test for Particle Instance
  • Updated my soc svn branch with updated blend files
  • Added documentation for testing here

Plan of Action

  • Add more tests for newly added features of modifiers, with more focus to Physics modifiers.
  • Code improvements, reviewing, and working on test object generator script
  • Merging to master
6 Likes

Week 12 (Aug 10- Aug 15):

  • Code improvements
  • Added tests for Cloth modifiers for newest features (pressure, improved self-collision, etc)

Next Week

  • Make final improvements based on review
  • Merge to master
  • Final Report
  • Improve User Documentation
  • Add more test cases
4 Likes

Week 13 (Aug 17- Aug 21):

  • Worked on code improvements based on review.
  • Improved User Documentation, made it more formal.
  • Finally worked on Object Generator Script, changed the style to Object Oriented, had difficulty including Vertex Group support, was unable to pinpoint the problem of “where did all the other objects go?” so that part has been removed for now.
  • The tests for Cloth Pressure and Self Collision were removed due to non-reproducibility on other systems.

Coming Week- The Finale Week

  • Have a summary meeting with mentors on how or when to make the merge.
  • Work on the review on both D8507 (Habib) and D8674 (Habib and Campbell)
  • D8507 is the Framework and Tests and D8674 is Test Object Generator Script
  • If extra time, add few more tests for Physics modifiers.
  • Work on the Work Project Submission (Final Report) which needs to be submitted to Google.
4 Likes

The Final Week: The End

  • Polished the framework, minor fixes and edits
  • Updated the operator tests, bevel and boolean for the 3rd time for the now new more generalized framework.
  • Submitted the final report, you can have a look here.

Upcoming weeks:

  • come to the rescue in case of test failures and if my patch errs.

Thanks for reading, this is all from my side!

11 Likes

Congratulations and Good Work @Calra!

1 Like