GSoC 2020: Regression Testing Framework - Weekly Reports

This will contain the weekly reports, I just came to know my branch is available publicly, it is called soc-2020-testing-frameworks and can be viewed here.

Week 1 (18-24 May):

  • Went through the Generate and Deform modifiers for surveying, as to how the framework should be built with special focus on helper objects and vertex groups.

  • Survey of mesh operators ( focus on user interaction )

  • Checked reproducible(ness) of the modifiers.

  • Support for other object types like Curves, surface, text.

  • Simplified inspection of Physics modifiers.

  • Build a working prototype for the automatic creation of blend objects.
    Sheet 3: Survey

Week 2 (June 1-7):
Plan of Action:

  • Complete the prototype to make it usable by fellow developers.
  • Plan next deliverable based on the survey.
5 Likes

Week 2 (June 1-7):

  • Finished with the prototype, the main aim for this week was to make it work, the improvements would be done in the following week.
  • Added support for generating vertex groups from the script.

Week 3 (June 8-14):
Plan of Action

  • Polish the current prototype
  • Identify tests by a unique name id.
4 Likes

It’s been a crazy week, I fell a little sick. Here are the updates-
Week 3 (June 8-14):

  • Made improvements to the prototype (can now be called a workable script), can be used by other developers.
  • Added the functionality to add unique names and check for unicity.

Week 4 (June 15-21):
Plan of Action

  • Add unique test names to modifiers and operators (working on it).
  • Add remaining tests for Modifiers based on the survey.
  • Add tests for Physics modifiers
3 Likes

This week was a bit slow.

Week 4 (June 15 - 21):

  • Added the test names to operators and modifiers
  • Meeting with mentors
  • Deliverable 2 started towards the end of this week.
  • Started adding tests for mesh modifiers (Skin,…to be continued)

Week 5 (June 22 - 28):

  • Finish the mesh modifiers
  • Start with Curves Modifiers
2 Likes

Week 5 (June 22-28):

  • Worked on including remaining mesh modifiers.
    The ones I was trying to test required “Object Operators”, so I added support for modifiers which require operators. e.g “Bind” for Surface Deform

  • Added default tests for Wave and Ocean, and dummy* tests for Surface deform and mesh deform.

*Dummy test - there is no actual deformation, hence the test cannot fail, which is necessary.

Week 6 (June 29-July 5):

  • Add support for imitating user input, using shape keys or animation.
  • Covering the remaining of mesh modifiers. (make real tests as opposed to dummy ones).
    Will cover Hook, Surface deform, Mesh deform, Warp, Laplacian Deform.

*Note 1- this does not involve Physics Modifiers, they would be covered separately.
*Note 2- Curves modifiers has been shifted to next week.

10 Likes

I just wanted to drop in and thank you for working on this, testing in general is so under appreciated! Keep up the good work :slight_smile:

Also apologies. I posted this in the wrong thread.

4 Likes

Week 6 (June 29-July 5):

  • Yayy! Finally done with mesh modifiers*. Now we can actually test the deformation using the animation and shape keys. So I have included the tests for the above-said modifiers. (look at the previous report)
    Idea credits @deadpin

Plan of Action

Week 7 (July 6-July 12):

  • This week will be dedicated to extend the framework for Curves (pretty much in sync with the timeline)
  • Improvements in the work done until now based on the review(s).

*Note- this doesn’t include Physics modifier, and there were some complications with Remesh and Smooth Corrective (they will either be skipped for now or worked upon in the background)

4 Likes

Week 7 (July 6-July 12):
It was a slow week, some college stuff came up and there were some personal issues. Also I got stuck on the review like having a coder’s block, I have discussed with the mentor, I did get some idea still got nothing working.
And I recently pushed the code to my remote branch.
Will update if something significant happens over the weekend.
Weekend-

  • added the run_test_using_name feature to the test cases and made some minor improvements as per the review.
  • as of now we can have tests that can imitate user input and support object operators, it is fully functional now (it is being reviewed again as I type this)

Next plan

  • Get started with building the framework for testing Curves Modifiers.
5 Likes

Week 8 (July 13-July 19):

  • Extended the framework, rather made it generic, to be able to test Curves
  • Added test cases for Curve Generate Modifiers
  • Updated the remote branch

Plan of Action:

  • add tests for the Deform Curves
  • start with Physics modifiers
  • reviewing/improving the newly added code
8 Likes

I have lot of things to take care of this week, so documenting alongside, this is the ongoing week.
Week 9 (July 20-July 26):

  • Tried testing Fluid modifiers, it is giving different “baked” result when testing inside blender as compared to testing it using a script. Found, it was because the test objects were produced in a different blender version and were being tested in a different blender version. Therefore the test was failing.
  • Adding Deform tests, Armature modifier for Curves cannot be assigned to Automatic weights (hence no vertex groups are created) so Bone envelopes is used.
  • New developments - Wave modifier is not regressable, it fails randomly. It passes after I freshly add it but fails on the next day or the day after. I suspect same thing as Fluid, different versions of Blender.
  • Worked with the Particle system, it is not reproducible as far as I tested, will give a final conclusion soon.

Plan of Action

  • The plan is to finish with Physics modifiers, firstly, I am planning to test Explode modifier, but it requires Particle system, so if that is reproducible, I will write an interface to interact with it, and write a test for it which will cover both I guess.
  • And I have some research paper work to take care of in the upcoming weeks, I will try my best to finish with the Deliverable 2 this week only and probably a meeting with the mentors(after finishing up I guess).
2 Likes

Hello Calra, I have a script available that can automate the process of rigging any arbitrary curve with an armature and envelope weights. In my addon the file ops_misc.py has a CreateBonesFromCurve operator that does this. When the addon is installed, this operator is available from the Add Armature menu when a curve is selected. Please feel free to use it if it will make creating the tests easier.

Thanks for making Blender better :slight_smile:

@Josephbburg I will have a look at it, if I would like to add more complicated tests. But I would like to keep it simple, and don’t mind a bad-rigged curve for testing :wink: , the modifier should work correctly, that’s what I am after. Thank you for both the things : )

1 Like

Week 10 (July 27-Aug 3):

  • Added a class and test for Particle systems, reproducible with a script.
  • Added a class and test for Dynamic Paint, its combined with Fluids.
  • Had a meeting with mentors, there have been changes with the next Deliverable and how to spend the next 3-4 weeks.
  • Had another meeting with Habib to discuss possible implementation of creating a generic class which can handle Physics modifiers as well.

Plan of Action for the upcoming week

  • To clean and polish the framework, refactoring the existing classes and combining all of them and creating a new Modifier Spec that can handle Physics modifiers.
  • Main focus would be on to solve the problem of parsing through a nested dictionary to apply the modifier parameters.
4 Likes

Week 11 (Aug 4- Aug 9):

  • Added a generic support for nested parameters.
  • Updated the bake method, and mer
  • Added test for Particle Instance
  • Updated my soc svn branch with updated blend files
  • Added documentation for testing here

Plan of Action

  • Add more tests for newly added features of modifiers, with more focus to Physics modifiers.
  • Code improvements, reviewing, and working on test object generator script
  • Merging to master
5 Likes