Fast mesh IO for addon development

Hello, every one!
I’m making a simulation addon that needs to dump its computation result into Blender viewport, in the form of mesh (and particles), so that user can view it and render it.
My simulation part is written in C++ which is fast, but I have to convert the compat C++ data into low-efficient Python list in order to make foreach_set happy… making my simulation unable to be real-time when there are huge amount of vertices and faces…
So I wonder if there is a way to ‘dump’ mesh data into Blender mesh with less overhead, like using memcpy with as_pointer? But I heard that C++ API has been deprecated since Blender 3.4 according to another thread, not sure what I should do now.
Any information would be appreciated, thanks in advance!

2 Likes

foreach_set also accepts a numpy array (or other python type that provides the buffer interface, I believe), without having to form a Python list. You could wrap your lowlevel data in a numpy array

1 Like

You could write an extension module in C or C++ callable from Python, that does bulk creation of a Python list object.

1 Like

By the way, this doesn’t make sense as foreach_set does not take a Python list, it only takes sequence values (such as a numpy array or buffer). So it seems like you already tried passing the latter types of data, but it is still too slow? What exactly have you tried so far (e.g. do you have some code to show?). One thing I noticed in passing large meshes is that it is not so much the data copying that takes time, but internal processing done afterwards, which you cannot avoid.

2 Likes

Exactly, I have tried NumPy arrays, it did work, but not too much faster than raw Python list… here was my attempt:

So I was thinking if it actually silently convert NumPy array to Python list in foreach_get, maybe I was wrong.

Thank for your information! So Blender will build something like a lookup table for geometry processing, every time after I update vertices of it?
But actually I don’t expect user to be able to enter Object Mode (by pressing Tab) to edit the generated mesh… it will update every frame… Is it possible to dump a read-only mesh, to prevent Blender from building the lookup tables, like what the Blender built-in fluid solver does?

Your original post talked about foreach_set, but this example shows foreach_get? Plus, you’re still passing a Python list to foreach_get (the seq value) and not a numpy array, so that’s going to be slow. You can pass the numpy array directly:

# Python console in Blender
>>> import numpy
>>> o = C.active_object
>>> m = o.data
>>> a = numpy.empty(len(m.vertices)*3, 'float32')
>>> a
array([ 1.1210388e-43,  0.0000000e+00,  0.0000000e+00,  0.0000000e+00,
        0.0000000e+00,  0.0000000e+00,  1.4012985e-45,  1.4012985e-45,
        0.0000000e+00,  3.6433760e-44,  0.0000000e+00,  0.0000000e+00,
        0.0000000e+00,  0.0000000e+00,  0.0000000e+00,  0.0000000e+00,
       -4.5643964e-37,  3.0685634e-41,  1.7037363e+00,  4.5906538e-41,
        2.3024779e+23,  4.5906538e-41,  0.0000000e+00,  0.0000000e+00],
      dtype=float32)

>>> m.vertices.foreach_get('co', a)
>>> a
array([ 1.,  1.,  1.,  1.,  1., -1.,  1., -1.,  1.,  1., -1., -1., -1.,
        1.,  1., -1.,  1., -1., -1., -1.,  1., -1., -1., -1.],
      dtype=float32)

I think entering edit mode will generate even more internal data structures, the ones I was talking about are when setting the geometry. I don’t think there’s a way around it. Plus the overall time spent setting the data might not be an issue. However, it currently won’t be superfast. For example, here’s some test code I had lying around:

import bpy
import numpy, time

# Number of triangles to create
T = 1000000

bpy.ops.object.select_all(action='SELECT')
bpy.ops.object.delete()

num_vertices = T*3
vertices = numpy.random.normal(size=num_vertices*3).astype('float32')

vertex_index = numpy.arange(T*3, dtype=numpy.int32)

loop_start = numpy.arange(T*3, step=3, dtype=numpy.int32)
loop_total = numpy.repeat(3, T)

t0 = time.time()

mesh = bpy.data.meshes.new(name='created mesh')

mesh.vertices.add(num_vertices)
mesh.vertices.foreach_set("co", vertices)

mesh.loops.add(num_vertices)
mesh.loops.foreach_set("vertex_index", vertex_index)

mesh.polygons.add(T)
mesh.polygons.foreach_set("loop_start", loop_start)
mesh.polygons.foreach_set("loop_total", loop_total)

mesh.update()
mesh.validate()

t1 = time.time()
print('Mesh created in %.3fs' % (t1-t0))

With 2.93.1 on my system this takes around 1.25s.

1 Like