Developer Discussion: New Grease Pencil Data Structure Proposal

Technical Feedback
1

Introduction

When grease pencil was first introduced to Blender back in 2008(?) by Joshua Leung it was designed around drawing annotations into the viewport. Over a decade later, the community has built a feature-rich 2D animation tool out of the initial draft of grease pencil. Today, there are some challenges and limitations that we face with how grease pencil is structured. I’d like to summarize the issues and propose a way forward.

Current Code Related Problems

  • Inefficient data structures
    • Layers, Frames and Strokes are stored in linked-lists making them very slow to search and iterate through.
    • Large tree-like hierarchy: GPdata → Layers → Frames → Strokes → Point
      • E.g. to access a point, one needs to search through all the levels above to get to it.
  • Grease Pencil is separated from the rest of Blender. Integration with other modules is hard.
    • Some examples are:
      • Grease Pencil modifiers
      • Grease Pencil materials (will change with upcoming EEVEE rewrite)
      • Grease Pencil dopesheet
    • More internal examples:
      • Grease Pencil modes
      • Grease Pencil brush settings (might be necessary)
  • A large portion of the grease pencil code base is very old and also very verbose (not an issue exclusive to grease pencil in Blender of course).
    • It gets harder and harder to make bigger changes
    • New developers have a hard time contributing

Proposal

For the last couple of weeks I have been working together with the Grease Pencil module (Antonio, Daniel, Matias) as well as Hans (Geometry Nodes developer) and Amélie (Developer at Les Fées Spéciales) to work on a new proposal for grease pencil.

The main goals of this proposal are to solve the issues mentioned above and also open the door for features that are currently very hard to implement.

Here is a quick summary:

  • Move to C++ and a RAII idiom.
  • Use two separate arrays to store layers and frames.
  • Use CurvesGeometry to store the strokes in a frame.
  • Make use of attributes to store radius, opacity, vertex color, etc.

(For more details, take a look at the demo/test implementation.)

Some of the new possibilities with this proposal:

  • Overall improved performance (e.g. copy-on-write).
  • Grease Pencil fills with holes.
    • Idea would be to use an attribute on strokes to group them and then pass the whole group to the triangulation algorithm.
    • This could enable: boolean operation on fills, boolean eraser, etc.
  • Layer groups.
  • Stroke types (default: poly, but could also use e.g. bézier type).
  • Easier integration into Geometry Nodes (because the core structure would be based around the same as e.g. Curves).
  • More readable and concise code.

Test Implementation

To better understand how the new grease pencil structures could look like, how we would work with them, and how they could perform I created a branch gpencil-new-data-proposal that implements a test file with the new proposed structures. They are:

  • blender/source/blender/blenkernel/intern/gpencil_new_proposal_test.cc and
  • blender/source/blender/blenkernel/intern/gpencil_new_proposal.hh

Feedback on the code would be greatly appreciated!
Here are some first performance numbers that came out of this test:

Duplicate GP data-block
Layers: 10, Frames: 5K, Strokes: 500K, Points: 50M
New structure:  205ms
Old structure: 1410ms

Note: Since the new structure is still missing parts of the old structure, take these numbers with a grain of salt.

Discussion

This proposal would be a major rewrite of course. There is a good chance it will break python compatibility, etc. We would like to get opinions/suggestions/concerns from other developers on this.

Note: This thread is not meant for user feedback. Only for development related discussions on the proposal!

45 Likes
2

Generally speaking, using CurvesGeometry with attributes makes a lot of sense to me.

One thing that’s not an ideal fit is that the list of attributes would be duplicated per frame, and there would need to be some way to keep that in sync between frames. Maybe across layers too, depending if attributes are defined per layer or not.

Do you have an idea for how this would integrate with geometry nodes exactly? Would geometry nodes treat grease pencil data as its own geometry type that works directly with existing fields and curves nodes (applying the changes on all frames and layers)? Or would there be some conversion to and from a curves geometry to be able to use the curves nodes?

3 Likes
3

I am not sure if I understand exactly what you mean. If you mean that the CustomData layers would have to be allocated per frame, then yes that is true. But since every grease pencil frame can be entirely different from all other frames (they are just their own container of strokes after all) I don’t think that is a big issue. Maybe I am missing something.
EDIT: To clarify, with how grease pencil is currently used, it’s rare to modify multiple frames at the same time. The exception of course is multi-frame edit.

So from what I gathered so far, grease pencil would implement it’s own GeometryComponent. I don’t think we can fully reuse the curves geometry component code, since - as you mentioned - grease pencil works on frames and you would have to have some piece of code that extracts the actual CurvesGeometry that is then modified. I don’t know how this would work at the moment. Maybe you always use the data from the current scene frame, maybe there is a node that allows you to access a specific frame, or maybe you modify all frames by default with an optional selection of some sort. But once you get that curves geometry, you could make use of it’s attributes without much hassle I believe.

4

If you have a user interface with a list of attributes for the user to edit, then I think that would be expected to affect all frames. Adding, removing or renaming a color attribute again for every frame is inconvenient.

5

Ah yes, I suppose a panel in the object data properties with a list of attributes would affect all frames when modified.

6

I like the goal to make gpencil more integrated with the rest of blender and allow for geometry nodes and more stroke types.

However a lot of what you want to achieve can be done with less drastic measures. We can move away from linked lists to array lists and improve the current data structs to make them more compatible with the rest of blender. E.g. use float[3] and float3 for C++ instead of float x, y, z and improve the names of things like pressure to curve compatible names like radius. And when that is done, we can start to move it over to the new structs with less code changes.

This should already give us half of what you want before breaking changes come up (performance, cleaner code) and should not be controversial. Keep the patches small and concise and make the compatibility breaking change as delayed and lightweight as possible. We can assess each step individually, one at a time.

1 Like
7

There were some discussions on how we could move to this proposed new structure and I fully agree with you here:

I didn’t want to include a possible way of moving to this new structure precisely because this hasn’t been decided on and there are some open questions.
Switching over to arrays might be a way forward, though this change alone would have to touch pretty much all grease pencil related source files. The versioning part of the code would be straightforward so that’s good.
Overall I think I would personally prefer a “gradual” change (as much as possible), like you suggest. But I think this needs careful planning first.

5 Likes
8

my 5 cents from addon developer, that using gpencil from py. breaking changes are a bit tough thing, but if we get in exchange a chance for normal booleans (pixel-perfect eraser?) and geometry nodes support and some unification with other blender types (materials stuff, etc) - this is certainly worth rewriting the py code :slight_smile: hope to see this proposal go live some day!

8 Likes
9

From an animator’s standpoint, getting a possible future where grease pencil frames can live and be transformed in the same dope sheet is a huge win.

5 Likes
10

Look at this: ⚙ D15003 Add GP layers in main dopesheet (WIP)

3 Likes
11

I have to agree with you about curves. As fun as it may be for developers to re-write again for them. curves make a lot more sense.

There’s more of everything out there for curves.

Here is our dev doc on improving the curves in Blender. These documents suggest that there would be no need for Grease Pencil. The Ideal is to make Blender’s codebase smaller and more functional. Using curves means more developers can jump in and refine the current toolset.

No layers are needed moving forward this way, because the outliner handles all of it.
No Rigging and Parenting and Rigging Issues Fewer
Fewer geometry nodes with great functionality.

3 Likes
12

To be clear, I agreed with the proposal that keeps Grease Pencil a separate datablock but share data structures with Curves internally.

Replacing the Grease Pencil datablock with the Curves datablock entirely as you propose is quite different thing. Adding concepts like layers, frames and fill to the Curves datablock or even any geometry datablock would be a huge design change.

That’s interesting to consider, but there are many trade-offs and design challenges, that as far as I can tell your proposal does not go into.

1 Like
14

@AdamEarle Reminder that this topic is only meant for development discussions on the proposal as stated at the end of the post. I appreciate your concerns regarding curves and how to work with them. This is unfortunately not the place to talk about it.

15

I know it’s early days and a lot of planning hasn’t happened, but is there any kind of guess as to how long such a restructuring would take? I can see add-on developers weighing the trade-off between writing to the existing structure against waiting for the new one to be implemented. Not looking for fixed dates, but it would be useful to know if the time horizon for breaking changes is greater or less than a year.

1 Like
16

Any breaking changes to the python API would only be allowed for Blender 4.0 earliest I believe. That would be end of next year.

1 Like
17

We are building this idea out in Python first. In our minds curves will become more unified because of the viewport compositor, and real-time playback, Ai powered or not.

At the moment there are:

  1. 3D curves.
  2. 2D curves.
  3. Roto Curves.

2D curves and Roto Curves could be rolled into one curve, or adding to the curve object properties adding a check box roto curves to be available in the viewport when it comes time to compositing. Ideally, this eliminates a window in Blender and helps unify the experience.

This would mean “grease pencil” could act more like a modifier combined with lineart modifier rather than a separate tool. The curve could adopt grease pencils frame-by-frame approach (a Check box maybe?). This would tie in with the viewport compositor in a more functional way.

At this point Grease pencil heading in the direction of curves is more about maintaining the branding.

This helps simplify and reduces Blender’s code base. It also means geometry nodes for grease pencil really won’t need to be developed at all and removes an entire window from Blender. To me this not just about grease pencil this is about blenders’ future in “animation”.

How can we rif on this? SVG Animation Export?

8 Likes
18

I wouldn’t sit around waiting brother, just get to work on what it is you want to do. My guess is something like this is going to take a veryeeeeeee long time.

1 Like
19

Proposal Update

Since the time of this post, there have been some developments that I want to communicate.

Defining the goal

In order to ensure that we can prepare grease pencil for the future, we want to hear as many artists out there as possible and make sure their needs and vision for grease pencil are aligned with what the proposal would bring. There is now a questionnaire that we’re sending out (soon to the wider public). After we have gathered that information, we’ll see if the proposal could have some limitations that we would need to fix. We will try and take as much as possible into account.

Ongoing development on the proposal

In the past weeks, the proposal branch has seen some further development. We’re trying to refine the code and expand on the functionality. Other developers (Amélie, Antonio, Hans, Yimming) have been getting involved as well (through meetings, discussions, contributions, etc.). For now we’re doing this on the branch and mostly to test out code structures, performance etc. Nothing testable by a user yet.

Idea on how to get this to master

One of the big questions that remained was how do we actually get this into master. Here is my current idea:
When talking about the proposal with other developers, a question that came up was: How do we deal with annotations? Currently, grease pencil and annotations use the same underlying data. But since grease pencil became it’s own object, they have drifted further and further apart. I think it would be a good idea to make them two separate IDs and clearly define them as different. Since we can use the current grease pencil data (bGPdata) as a basis for the annotations, the idea for the new data would look something like this:

  1. Add a new grease pencil ID for the new data structure (maybe ID_GP ?).
  2. At some point: Rename the current grease pencil ID_GD to something like ID_AN for annotations. The data structure (bGPdata) remains in the code for annotations as well as conversion/versioning.
  3. Slowly move all the functionality to the new ID. Slowly deprecate functionality in the old ID that’s not needed for annotations.

Since the new development would happen on a new ID, it could be done in master. Maybe behind an experimental flag. Conversion code would then convert between the old and new ID to make sure everything keeps working while the new ID is introduced. Finally all .blend files should use the new ID for grease pencil data and the old ID for annotation data.

8 Likes
20

For Mesh we are moving to SoA incrementally, but if the grease pencil data structure is completely different then a new data structure may be best indeed.

How would functionality be slowly moved or deprecated in terms of integration with the rest of Blender? I guess you would make existing modifiers, operators and tools support both types? While some more internal things like data structures, depsgraph integration and draw caches would be completely duplicated?

If you’re taking the approach of a new data structure, then I guess practically grease pencil development should focus on getting this finished, and not do much work on new features and changes that will cause things to go out of sync.

I would expect splitting off annotations to be done as the last step, not sure how you could rename the datablock to annotations before that.

1 Like
21

Thank you for the quick response Brecht! :slight_smile:

IMO I would not try to reuse the code for operators, modifiers, and tools for the new ID. The implementations will be very different (also going from C to C++). If we wanted to use the new data internally, we could always do a conversion to the old data, run the operator or whatever, and then convert back. But the performance penalty would be pretty big. I don’t think we should have these regressions for potentially many releases. So I would rather expose the user to the old data, and work on getting feature parity with the new data behind an experimental flag. At some point, we can just convert the old IDs to the new (like I think it’s done with the new hair system for example).

The depsgraph would just treat them as two separate IDs I guess so, yes, we need the logic to be duplicated there. For the rendering, I think Clément is already prepared for this and it’s just a matter of passing the right data to the engine. That should not be a big problem and we won’t have to duplicate much I don’t think.

I agree. There is an idea floating around to do a workshop in Amsterdam so we can do a bigger leap in a shorter time.

Right, that might be better. I was thinking of an internal renaming so that all grease pencil objects are still exposed as Grease Pencil to the user, but seen as annotations in the code. That might not be beneficial, so maybe it’s better to just do it once there is feature parity.