Super helpful input warning: Just try to make it work dude, what do you have to lose?
1 Like
exactly you need to isolate the export/import data into Blender and maybe create a kind of proxy or show a part of it in Blender at the time and just use the full sim for the final render. 5 to 9 million particles in Blender itself will be a hell of performance issue no matter what kind.
I need to mention I did all simulations on the CPU and will release the GPU and Network versions later.
Most of the developers do not know about the difference between developing normal tools even rendered with solvers that need to contract with UI and objects more than even usual, in MPM Iām using more than 30 subsamples sometimes 70 subsamples and this is a completely different behavior than normal addons.
I am a part of the LuxCore renderer development team and know all aspects of developing tools for Blender, 3dsMax, and Maya and using Python instead of C/C++ especially in a not thread-safe environment using only one thread will be a disaster in performance matter.
Iāll be glad to know more about solvers, I appreciate your suggestion. if you have a discord server or someplace to talk I believe we can help each other more.
1 Like
If you can, you could participate in the next meeting of the physics module, there I think you could have more concrete answers.
2 Likes
Can i ask how we should join this meeting? what platform are you using ?
Unfortunately, time.
Yes, I am interested to listen and perhaps contribute, if I can.
1 Like
If you hadnāt considered it, you could treat it as a test. If it works good, great. If you wind up going in another direction, you would be able to gain something hopefully. The things you did, or could not do, would contribute to trying again with something else. At least some of the work would be re-usable too.
Again, not exactly helpful to you, but something to take away from a less than perfect situation.
Check the header of the Node and physics module chat there is a Google Meet link
https://blender.chat/channel/nodes-physics-module
Just a reminder for everyone, although most if not all meetings are open for anyone to join, if all you plan to do is listen, then itās prefered that you donāt join the meeting and instead wait for the meeting notes to be released.
If you want to join the meeting to discuss ideas, but have no plan to contribute them to Blender (E.g. You want a node that does a specific thing, and you want the Blender developers to implement it for you because you have limited to no programming knownledge), then it is again advised that you donāt join the meeting, instead provide your feature requests somewhere else, like Right Click Select
If you want to discuss new large ideas at the meeting, itās best to talk about it a bit with the module on something like Blender chat before the meeting. This is to help co-ordianate things with regard to time, ensuring the right people are at the meeting, or even figuring out if the module meeting is the right place to have the discussion.
4 Likes
I agree, there is usually something to learn 
Hey, this is a developer of the FLIP Fluids addon. Iām glad to add some info on our experience with interfacing with Blender as a simulation addon.
How the FLIP Fluids addon interfaces with Blender
Our simulator is written in C++ and is built as a shared library. Currently we distribute the library for Windows/Linux x64, MacOS Intel, and MacOS Apple Silicon to match the current Blender architectures.
The Python ctypes module is used to load the library and call C methods. The C methods are bindings that then call C++ methods.
import ctypes
lib = ctypes.cddl.LoadLibrary("simulator.dll")
lib.hello_world()
We wanted Blender to be responsive and usable while the simulation is running, so we run the simulation Python script in a separate thread so that it does not block the UI. The Blender Python API is not thread safe, so we make sure that the simulation script has everything it needs to run the simulation and does not need to access the API. Before the simulation begins, we export simulation parameters and mesh data using the API and then pass this into the simulation script.
While the simulation is running, events are handled within modal operator. A basic dictionary is shared between the modal operator and simulation thread for communication, such as reporting progress, or telling the simulation thread to stop.
Basic simulation loop:
Initialize simulator
for each frame:
Set simulation parameters and mesh data
Run simulation frame
Retrieve and write simulation data
Simulation data is mostly written as binary data to a cache directory. When a user updates the timeline frame in Blender, a frame change handler is run which loads the simulation frame into the Blender scene. The Python struct module is used to unpack the binary data. The data is then formatted and passed to bpy.types.Mesh.from_pydata() to create a mesh and attributes are added using mesh AttributesGroups.
Sidenote on frame change handlers: A large pain point for ourselves and artists is that there can be issues with updating data on a frame change. There is an issue (#88811) that can result in frequent crashes or incorrect renders. Rendering from the command line completely works around this issue. Due to this issue, rendering from the command line is almost always necessary when rendering our simulations.
We try to make this easy for the artist by providing an operator to automatically launch a background instance of Blender to begin a command line render. However, itās easy for the artist to miss the UI warnings we display and our documentation on this issue. Commonly, artists will have a poor first experience when they render their first simulation and find that the result is incorrect or has terminated early.
Data Exchange and Render Performance
Loading and rendering large amounts of point data can be slow and can require patience. We sometimes see artists rendering particle amounts around 100M, but often not much higher. Around this amount I think that the loading and rendering times can become prohibitive.
Sidenote on Point Clouds: Point rendering performance has improved greatly with the introduction of the Point Cloud object type since Blender 3.1 and has been my favourite Blender advancement. Before point clouds, we needed to render points by instancing spheres over each vertex and was limited to around 6M - 8M points before we ran out of patience or system resources. Now with point clouds, we can rendering 100M particles in less time and with less resources.
Tip: vertex meshes can be converted to point clouds using the Geometry Nodes > Mesh to Points Node.
When loading 100M particles in the scene through Python, the majority of the time spent is within the Mesh.from_pydata() method. Reading the points from SSD and then extracting the binary data is relatively quick compared to the from_pydata call. For a viewport preview, I would suggest loading a small subset of particles to keep the loading times down.
During render, the initialization stage before the actual GPU render begins can take a significant amount of time when there are many particles. You may often spend more time on initialization compared to the GPU render. Rendering with motion blur can add extra time for loading (extra 3D velocity vector to load) and initialization.
Hereās a quick loading and render initialization test of 100M particles on an Intel i9-13900K CPU and RTX 4090GPU:
Without motion blur:
Particle Load: 1m31s, ~16 GB RAM
Render Initialization: 22s, Peak 6.1GB VRAM
Time before render begins: 1m53s
With motion blur:
Particle Load: 1m43s, ~24.8 GB RAM
Render Initialization: 4m18s, Peak 9.4 GB VRAM
Time before render begins: 6m01s
If you have a large amount of RAM/VRAM, something that can help maximize resource usage is to launch multiple render instances to work on different frames at the same time. This way, the CPU can work on initializing multiple frames at once. As far as I can tell, the initialization stage is largely single-threaded. However, this wonāt be useful if youāre using all of your RAM/VRAM for a single frame.
Hope this info helps!
13 Likes
Aswesome contribution. Specific and relevant, very much appreciated! Some expected patterns there, but especially render times and their parts are a very valuable bit of info, even though not all scenes are the same, I think we can infer some common ground and make educated assumptions.
So yes, this helps, at least in my process of figuring out a valid way forward. Thanks again !!
2 Likes
This post was flagged by the community and is temporarily hidden.
The topic of Blender needing an improvements to handle complex setups (in both of scene/rig configuration, but also handle high-poly meshes and point-clouds) is known by the developers. The reasons why they are not handle as quickly as one would hope comes from the developer time availability and prioritisation of topics for the small team of developers.
This forum is for communication about Blender development in terms of contributing to the Blender project, and this discussion does not fall under this definition and is to happen somewhere else.
Iāll close this thread. Youāre welcome to start a new one if youāll be picking up some Blender development and contributing to it.
4 Likes