GSoC 2021: Regression Testing of Geometry Nodes: Weekly Reports

This post will be used for updating the progress on Regression of Geometry Nodes project.
Have a look at the proposal and daily logs.
For feedback, this thread itself can be used.
From last year’s experience, since this project doesn’t directly affect the users, feedback received is less.

The Geometry Nodes Survey Form has comments enabled, feel free to add your remarks or make corrections, point to resources or Youtube videos or attach a link to your blend files.
What I am looking for is simple blend files that can be used for testing the Geometry Nodes. Thanks!

Week 0: 17 May - 23 May

I got vaccinated for my first dose, so few days I was in recovery.

  • Less activity on bug tracker.
  • Talked to mentors : Jacques and Habib on blender.chat
  • Topics of discussion: Incorporate versioning of geometry nodes in testing.
  • Official kick-off starter meeting planned on Tuesday, 25th.

Plans for next week:

  • Study geometry nodes and discuss and think ways for how it can be tested.
  • Planning to go from the top and start with Attributes.
  • Do a literature survey by looking at Blender and Houdini documentation and how they work.
10 Likes

Week 1: 24 May - 30 May

  • Moderately active on Bug Tracker
  • Changes: I started with surveying Point Nodes
  • Went through sample files of GN.
  • Convex Hull operator test shifted to next week.
  • Link for Geometry Nodes Survey.

Plans for next week:

  • Will be less active on tracker.
  • More time with GN.
  • Look at Color and Material Nodes.
  • Another look at Geometry and Mesh nodes.
  • Look at LazyDodo’s Geometry Node builder.
5 Likes

Meeting was at Friday, 12 CEST.
Platform: Google Meet
Attended by: Habib Gahbiche, Himanshi Kalra

Meeting Agenda:

  • Run the Class design by the mentor.
  • Feedback on the current state of things.
  • Fixing a review day (if a review is due)
  • Rules for using an existing addon in the code or copying it.
  • Thought on Attributes and Utilities, can they be tested…
  • Open a TODO/Task for test/example case contributions for simple blend files for Geometry Nodes.

Issues

  • Python API for setting attributes of geometry nodes.
  • They are accessed by index[i].default value instead of name…
  • Discuss on the different types of attributes [IMP].
  • API for using existing addon “addon_utils”, perhaps.
  • versioning

Meeting Notes:

  • Discussed Class Design and made some changes (Adding Geometry Nodes Modifier as part of the Geo Node class)
  • Reviewing within fixed number of days instead of a specific day. (e.g. within 2 working days)
  • Discussed to look at an approach with loading a blend file with existing Geometry Nodes tree.
  • Code Specific Issues and Class - Differentiating between the 2 types of parameters in Geometry Nodes.
  • Versioning Releases (2.9 vs 3.0 etc): Regression Testing already covers testing amongst various git versions. (Clarify with Jacques)
  • Deciding on leaving Attribute Nodes for a later period, also talk with Simon (technical artist) to provide for meaningful test cases.
6 Likes

Week 2: 31 May - 6 June

  • Manual testing of some nodes. (Curve to Mesh)
  • Designing a raw prototype for the GeoNode Test class.
  • Meeting with mentor (Meeting Notes in the above post)

Plans for next week:

  • Try another approach for testing, blend file with pre-loaded modifiers.
  • Refine the design for the proposed solution as discussed in the meeting.
6 Likes

Week 3: 7 June - 13 June

  • The work on the script for testing blend files directly is almost complete.
  • Tried testing out some Geometry Nodes.

Plans for next week:

  • Shift the work to the soc branch.
  • Polish the script and put it up for review.
  • Add test cases for Mesh type Geometry Nodes.
  • Discuss Jacques’ new idea of regression testing.
5 Likes

Week 4: 14 June - 20 June

A little info on the script:
I am working on creating a Python script which can read the test and expected (reference) objects from the blend file, compare them and give the verdict.
I was more active in the first half of the week, was a bit stuck with design decisions at the end of the week.

  • Moved the script to soc branch.
  • Polished the script based on Jacques’ review.
    Added features:
  • Can add expected object automatically if it doesn’t exist with the BLENDER_TEST_UPDATE flag.
  • Makes adding test objects easier.

Plans for next week:
Mesh Test framework pretty much does what the script currently does. With the difference being, it creates the modifiers (applies the operator) from scratch.

  • Merge script’s common functionality with Mesh Test’s.
  • Refactor script to act more like a module.

Thanks for reading. Stay updated!

9 Likes

June 23:
Meeting Notes:

Attended by: Habib Gahbiche, Himanshi Kalra
At: 12 CEST
Platform: Google Meet

Meeting Agenda: Recap all the Approaches and discuss the current approach.

Meeting Notes

  • Himanshi summarized the various approaches attempted till now.
  • Discuss the current approach that uses separate module for common functions.
  • Mentor’s suggested solution: To create a static method for compare_meshes.
  • Finalized with a modification of Approach 5: Implementing an Abstract Base Class from which Mesh Test (existing framework) and Geometry Nodes Test (new one) can inherit common functionalities.

Doc Link: Different Approaches

Next Plans:

  1. UML diagram based on Approach 5 (common class).
  2. implement uml diagram only (functions don’t do anything, e.g. they contain ‘pass’)
  3. implement functions
5 Likes

Week 5: 21 June - 27 June

I created a module for common functionalities that are needed in executing a test. The current state of D11611. After discussion with the mentor, we decided on creating a common class for these re-usable code instead of a module.
This is the UML diagram of the classes:

Plans for next week:

  • Implement the UML diagram with pseudo code.
  • Implement the Base MeshTest class and its methods.
6 Likes

Week 6: 28 June - 4 July

My keyboard started acting up :confused: this week, some keys are not working.
Updates:

  • Mesh Test class has been fully implemented.
  • Tested Lukas’s Edge scrolling patch and gave feedback.
  • Read through some of the Attributes columns proposal.
  • Had a small meeting with Jacques regarding some implementation questions.

Plans for next week:

  • Implement SpecMeshTest- class for handling existing tests which requires specifications.
  • Implement BlendFileTest - class for handling new geo-node tests which load directly from blend file.

Here is the updated UML.


P.S. I updated to Windows 11 :slight_smile: and boy it has some issues…

4 Likes

Week 7: July 5 - 11 July

I bought a new keyboard, but now laptop keys are working fine.
Updates:

  • The framework is refactored and complete. Link to Diff
  • Added the “create an expected object” feature for existing tests.
  • Updated all tests in accordance to the new framework.

Plans for next week:

  • Add tests for the new nodes.
  • Help out with the patches listed under Testing/Feedback on the GN workboard.

Evaluations coming week! :slight_smile:
Meeting tomorrow.
Thank y’all for reading.

5 Likes

Week 8: July 12 - 18 July

I had my evaluations…
I passed :slight_smile:
I was working on 2 things this week.
First:

  • Polishing the framework. [All done]
  • Incorporating feedback.

There are some minor code-styles I am ready to overlook, e.g. condensing if-else etc.

Second:

  • Adding the Custom data properties to compare attributes and values of these attributes.

I have had been struggling with the second part, there has been less progress. Having trouble with understanding the C side of the code.

Plans for next week

  • Add tests for Curves.
  • Continuing from previous week on comparing custom data properties.
7 Likes

Week 9: July 19 - 25 July

This week was fast and slow. For the first half of the week, I was working with Curves. For the second half, I was working with comparing attributes.

  • Finished with adding test cases of Curves.
  • Reported some Assert Failures.
  • Worked on comparing custom data.

Plans for next week:

  • Merge the project with master. [Improve Commit message and make test size smaller]
  • Finish with adding the code for comparing custom data. […hopefully]
  • Start with tests for Volumes. [Discuss]
2 Likes

Week 10: July 26 - 1 Aug

Tootooroo… my project is merged in master :slight_smile:
Commit link

  • Clean up of the test files and patch.
  • Merging… (hah messed up the commit message ^^’)
  • Was working with both mentors to better under the custom data and attributes.

Plans for next week:
More info on what I am trying to implement:
Currently for regression testing, we compare mesh data such as vertex coordinates, edge positions, total number of edges, faces and so on.
There are also generic attributes which are stored in form of custom data(including the above), from my understanding*, they could be of any type and be created by user. It doesn’t necessarily come along when object is created. e.g. Rotation of points.
*Note: This is just my understanding of the concept, reach out to Jacques or Hans or Habib for clearer explanations.
So, the goal is to compare these from test object and expected object. This will help in better testing of Attribute Nodes.
Rolling over from last few weeks, been trying to handle custom data…in the original meeting Jacques said it was easy…
Had a debugging/discussion session with Habib on Saturday 10 CEST on the above topic, concluded with doing it in Python with a proof of concept dry-run in Blender Python console.

  • Discuss with Jacques about:
    • Using Python to compare attributes in Monday meeting or earlier.
    • When Built-in attributes will be exposed.
    • Benefits of doing it in C than in Python.
  • Add Volume tests on the side.
    Thanks for reading! Was rather a long post! Next time I will show how to run tests with a video :stuck_out_tongue:
13 Likes

Week 11: August 2 - 8

Discussed with Jacques to implement custom data comparison in Python, Jacques shared a boilerplate code to help me out in implementing in C. My attempt at editing and appending in the existing to compare the layers didn’t work out, as the layer order for generic attributes didn’t match for the two objects, hence I was stuck at it since the beginning.

Plans for next week:

  • Definitely add test cases for more Nodes.
  • Work on changing the threshold from absolute to relative.

Note for future me: Don’t use Point Instance Node in attribute tests as it doesn’t transfer attributes due to current limitations.

I will be traveling on Thursday and Friday, will be available on phone.

8 Likes

Week 12: August 9 - 15

I was traveling, I came back this evening. Worked only for 3 days from Monday to Wednesday, this week. Too tired to write a full weekly report. Will update it with more details tomorrow.

  • Added case for comparing custom color data type.
  • I added test cases!

Plans for next week
Coming is the final week.

  • I would like to focus on adding test cases. If time permits at the end, I will amend it from the relative threshold.
  • Upload a diff with test cases made last week (local on my computer).
  • Also some other pre-Gsoc patches need update, will update them this week.
  • Submit work product.
5 Likes

Ah I forgot, sorry, am late.

Final Week: August 16 - 22

I had a great time this summer. Thanks for your support!
Now you can try editing the geometry nodes code and then run the tests to check if you accidentally break something and easily fix it.

10 Likes