Render engine add-ons performance

Archive Python API
9

The biggest limiting factors for Python addon render engines are, in my opinion:

  • There’s no fast way to iterate depsgraph.object_instances and other potentially large depsgraph data structures. Currently they have to be iterated in Python, and there’s one element for each particle (easily millions). Best would be if such huge lists had a C++ API or would support the Python buffer interface, so they could be passed to a C++ Python module.
  • Threre’s no official fast way to read mesh data from a C++ module. Our exporter works around this by getting a pointer to the mesh with as_pointer() and re-interpreting it as a Blender mesh struct in C++, but because this is an internal Blender datastructure, it can change between Blender versions.
  • Hair data has similar problems: It can’t be accessed from C++ at all and requires a call to the Python function particle_system.co_hair(object=obj, particle_no=pindex, step=step) for each hair point (easily millions).

Images are actually the least of our troubles, because we read them directly from disk.
But I also have a wishlist for image pixel access, which is important for packed files and for image formats unsupported by our own image loading libs:
The new image.pixels.foreach_get() is nice, but if the image data is not yet loaded into Blender, it behaves as a blocking call and freezes everything until the image is loaded (which can take a few seconds for large images on slow storage). It would be great if there was an asynchronous API to avoid the freezing, for example:

  • image.pre_load() could be a non-blocking call that triggers image loading in a background thread and immediately returns. Maybe it would also make sense if we could pass a callback function to it that gets notified once the loading is done: image.pre_load(on_loading_done)
  • image.is_loaded() could return wether the image was loaded yet, so we can poll for it.
  • Once image.is_loaded() returns True or the callback is called, it would be safe to call image.pixels.foreach_get() and the only freeze happening would be the time it takes to copy the data.
11 Likes