Refactoring IDProperty Code

I’ve been researching IDProperties because we plan to use them to store values for node
trees in geometry nodes. I have found the system quite complicated, and there seems to be
some consensus that the way they are implemented is a bit of a hack. Maybe my perspective
is naive, but it’s also the perspective of someone encountering this for the first time
who would rather build on top of a solid base.

Mapping Between IDProperty and RNA Types for UI

Currently, in order to get information like RNA’s PropertySubType, min and max,
description, etc, the code actually builds a group of properties, with a parent group
property and a child group property for every value in the following chart. This means
the UI code has to do a string lookup in the group’s children for every one of these
values for every layout pass.

Value Type IDProperty Name IDProperty Type
Subtype subtype IDP_STRING
Description description IDP_STRING
Int Min min IDP_INT
Int Max max IDP_INT
Int Soft Min soft_min IDP_INT
Int Soft Max soft_max IDP_INT
Int Step step IDP_INT
Int Default default IDP_INT
Float Min min IDP_DOUBLE
Float Max max IDP_DOUBLE
Float Soft Min soft_min IDP_DOUBLE
Float Soft Max soft_max IDP_DOUBLE
Float Step step IDP_DOUBLE
Float Precision precision IDP_DOUBLE
Float Default default IDP_DOUBLE
String Default default IDP_STRING

While there’s an argument for doing it this way-- it reuses the existing “groups”,
I get the impression it causes more problems than it “fixes”. There’s a change I’m
missing something though, and there’s a good reason these values are defined this way.

Proposal

I propose adding these fields directly to the IDProperty struct. While they are maybe
not always needed, it would be a small price to pay for the convience of properly storing
these values.

I suggest storing these values in a “UI data” struct, with polymorphic type specific structs to store
the values unique to each property type.

Versioning would have to be considered with these changes. To simplify the process,
these changes could be implemented step by step.

Adding an IDProperty

Currently there is the large IDP_New function, which handles the creation of every type
of IDProperty. It takes a multi-purpose union as an argument. For obvious reasons this
is not so nice to use.

typedef union IDPropertyTemplate {
  int i;
  float f;
  double d;
  struct {
    const char *str;
    int len;
    char subtype;
  } string;
  struct ID *id;
  struct {
    int len;
    char type;
  } array;
  struct {
    int matvec_size;
    const float *example;
  } matrix_or_vector;
} IDPropertyTemplate;

Proposal

I propose splitting up IDP_new and instead using a separate function for each type.
There could be a template struct for each function as a clean way to pass arguments.
Along with the changes I propose in the previous section, a call to create a new int property would look more like this:

IDPropertyInt *prop = IDP_new_int(name, 
                                  &(IDPropertyTemplateInt){
                                      .rna_subtype = PROP_PIXEL,
                                      .min = 0,
                                      .max = 100,
                                      .soft_min = 0,
                                      .soft_min = 10,
                                      .default = 5,
                                      .step = 5,
                                      .decription = "A description",
                                  })

Obviously in many cases all of those values wouldn’t be defined, so it would be shorter.

The Name

I think the idea was that these properties were only supposed to be added to IDs, but that isn’t true anymore, and it just confuses the idea.

Proposal

I don’t have a better name idea right now. Though IMO it’s a lower priority change anyway.
The function names in BKE_idprop.h should also be changed to conform to the style guide.

3 Likes

Thanks for looking into this. I think it is important to discuss these different refactorings separately. I’ll only comment on the RNA type related proposal for now, because this is about the problem we set out to solve. The other refactorings can be discussed later on.


Generally, I agree. Adding properties like subtype, min and max directly to IDProperty makes more sense than storing them in separate id properties. This would make accessing the properties much easier and more efficient.

I’m not sure if you suggest to embed the properties directly in the IDProperty struct or to create a new struct that is referenced by the IDProperty struct. Possible names for this new struct could be IDPropertyUI, IDPropertyInfo or IDPropertyRNA. There could be subtypes like FloatIDPropertyInfo to store type specific information.

I think it makes sense to use a separate struct and only store a pointer within IDProperty. This has a couple of benefits:

  • We don’t have to store it for id properties that don’t need it.
  • Adding type specific information is possible without cluttering the IDProperty struct.
  • It can be added without breaking existing code. When the pointer is null, we can just fallback to old approach. We might want to remove the old solution eventually, but it’s not necessary to do this now.

One thing that is still missing from the proposal is how addons can edit this information. Currently, they have to edit the _RNA_UI property. We should provide a better alternative. The easiest solution is probably to add new methods to BPy_IDGroup_methods and pyrna_struct_methods. Below is an example of how the API could look like. Feel free to suggest alternatives.

# This returns the top level group IDProperty or None.
props = C.object.custom_properties() 

# Update the rna type information of one specific IDProperty that is a direct child of the group.
# The method takes a couple of keyword-only arguments for the different options.
props.update_rna("my custom property", min=0, max=1, description="My Description)
2 Likes

I don’t think every IDProperty needs the overhead of UI metadata. We also use IDProperty to store data where either this data is not needed or already provided because ID properties are just used as storage for RNA properties which already have it. So storage as described by @jacqueslucke makes sense to me.

The IDP_new API seems fine. However in the context of geometry nodes, my understanding is that this would be use for group node inputs only, not built-in or add-on generated nodes. And then you don’t really need a simpler API, though it doesn’t hurt to have either.

The name is indeed not ideal, but it is used in the Python API so changing it either means breaking compatibility or having a mismatch. So I wouldn’t bother right now. It still makes sense in that it is a property for datablocks / IDs, just not necessarily at the root level, but on also on subdata.

I agree that only storing this data on demand would be better.

The second two sections are much less important to me, in retrospect I shouldn’t have included them here.

Can the refactoring be used for storing, for example, min and max values of a property per an object or what is use of the Property UI manager? There is nothing about it in API 3.0 documentation now.

I noticed that many of my plugins stopped working in Blender 3.0. Unfortunately, this IDProperties refactoring seems to be the major cause.
The old, dictionary-like approach allowed plugins to add extra metadata to custom properties other than min, max, soft_min etc. Obviously one could qualify this as “misuse” of the old _RNA_UI implementation. However there are many reasons to store extra metadata on custom properties and the old implementation easily allowed for that.

Examples of extra metadata:

  • Type identifiers, e.g. “channel_type”: “body_morph”/“head_morph”/“expression”/“pose” to indicate that the custom property controls a body morph, head morph, facial expression or pose so they can be properly displayed and grouped in a custom UI;
  • Source identifiers, e.g. “origin_type”=“my_anim_plugin/main_speed_control” to indicate that the property was created by (or managed by) some plugin;
  • Import details, e.g. “my_daz_import/channel” to indicate that the property was originally a Channel imported from Daz;
  • UI Label, e.g. “label”: “Offset” for a property “someplugin_offset” or the other way around (e.g. “name”: “Offset” and “original_name”: “someplugin_offset”;
  • Unique IDs to keep track of properties even when their name was changed;
  • … and many many more use cases.

It seems that that when .blend files are converted to the new 3.0 implementation, all of my plugins’ extra metadata is ignored and, as far as I can see, destroyed.
This means that most of my plugins would need to implement some other way of storing the metadata in 2.93. Then, my plugin users would need to update their 2.93 plugins, then load all their .blend files in 2.93 in order to convert them , then save them in order to “prepare” them for opening in 3.0. Quite a hassle…

Is there any way to make Blender 3.0 import my custom properties as “python” type (that would seem helpful, at least for converting the existing metadata)?
What would be a good way to store metadata on custom properties? I guess we can’t have “custom properties on custom properties” which is a pun, but might actually be quite easy to implement?

1 Like

In addition to the above:

Previously, an IDPropertyGroup behaved like a dictionary and could be used to both set and get metadata, i.e.

# setting values
ui_data["max"]=0.5

# getting values
max=ui_data["max"]       # Or max=ui_data.get("max", 1.0)

The new IDPropertyUIManager has a new api for updating metadata, which is useful because it accepts only a limited amount of keys and actually “manages” the metadata, i.e. by keeping soft_min and soft_max within bounds and more:

ui_data.update(max=0.5)  # Also updates 'soft_max' if out of bounds.

However, this new class also introduces a new api for reading metadata: you must call as_dict() on it before you can read the values:

rna_data = ui_data.as_dict();   # Need to get a dictionary first
max=rna_data.get("max", 1.0)

# Should be:
max=ui_data["max"]       # Or max=ui_data.get("max", 1.0)

Code changes like these are fine to convert existing plugins to “blender-3.0-only” plugins, however for code that needs to run in both 2.* and 3.0, this introduces extra complexity not only for updating the metadata, but also for simply reading it. In Blender 2.*, ui_data behaves like a dictionary, but in Blender 3.0, you need to first call .as_dict() on it to get a dictionary.
That’s a lot of extra if bpy.app.version < (3, 0, 0): statements…

It would be much more convenient if the IDPropertyUIManager behaved like a dictionary (at least for getting data) so that you could simply use ui_data["max"] or ui_data.get("max", 1.0) on them (just like in Blender 2.*) without being forced to call as_dict() in newer Blender versions.