Improve RAM usage in Blender

Archive User Feedback
1

I am currently evaluating Blender for a subsidiary of one of the largest automative manufacturers in Germany, who want to change their 3D visualization pipeline from Maya to Blender, with Eevee as output engine. They are dealing with huge datasets which they try to handle with collections, linked libraries etc.

Car models from CAD software are obviously rather dense, even if you export them already simplified.This becomes a problem when trying to display it in the viewport, because Blender uses a lot of RAM to display the geometry.

So far they were using Maya, which was able to handle their car configurations.
However, with Blender they are running into memory limitations.

Comparing memory usage

In order to compare the two programs and how they handle geometry, they have created a test configuration, which uses the same amount of geometry in Bender and in Maya.

The company has simulated a case with 517 files, containing 5 subdivided Suzannes with 62.976 triangles each.
All in all that is 162.792.960 triangles.

Maya

In Maya they import the objects from the 517 .mb files as references, which as far as I know is more or less the same as linked libraries in Blender. However, at first they do not display the 2585 Suzannes in the Viewport yet, they are hidden.
This takes about 12.2 GB of memory.
In the next part of the test they display all the objects in the viewport.
The RAM usage spikes up to about 25GB while Maya is “uploading” the objects to the viewport. After that however the memory drops to about 14GB.

Blender

For Blender they wrote a script that saves their basic test scene with the 5 Suzannes to 517 blendfiles.
In their file they are also testing the impact of nested parenting, but according to my tests this doesn’t impact memory usage much.
I wrote a script that links the 2585 subdivided Suzannes from the 517 blendfiles to an excluded collection in a new scene, so that the viewport does not have to display them. Once all libraries are linked Blender uses 10.4GB of RAM, this seems even better than in Maya.
Now I instance the collection containing all the linked libraries into the active scene. This takes a while, but once all 2585 Suzannes are visible in the viewport Blender takes, according to its own stats, 23.3GB RAM and 8GB of VRam. In the system monitor (Linux) it says 31.3GB.
If I save and reopen the scene now, memory usage is the same. It also doesn’t matter if the geometry is spread over many objects or just a few. The more triangles, the more memory is needed.

Testfiles

Here are the two files for testing:
https://www.dropbox.com/s/azrby05y47xh5p9/TestScene.blend?dl=0
https://www.dropbox.com/s/01ck37pb88monsz/import.blend?dl=0
The first one is “TestScene.blend”
It contains the 5 subdivided Suzannes and a script that will write 517 copies of itself to the folder “TestFiles”.
Once those files have been written you can open “import.blend”.
When you run the script it will open a UI in the sidebar of the 3D viewport on the left, where you could adjust the path to the folder with the 517 if needed. The field “Pattern” could be used if you wanted to import collections with a different name. The default of “Test” is enough for this test though.
When you click “Car import” it will link all the collections from “TestFiles” into a new scene.
Once the import is ready you can hit Shift-A in the viewport and add the collection “new_import”. This will make the Suzannes visible in the viewport and you can observe the RAM filling up.

So?

An entire car configuration has far more triangles than this testscene, so they are running into hardware limitations.

The question is, can we do something about that in Blender? Is it possible to have a way to use libraries more memory efficient? Maybe turn of some animation features or so? Like, tag a library on import as non-animatable or so in order to cut down the needed geometry? Or improve memory usage for geometry in general?

13 Likes
2

During a preliminary discussion I proposed some solutions, but they would need to be verified if any is the correct approach.

In CAD every object is unique (has to do with bill of materials and traceability) the actual geometry could still be the same. For example bolts could to be addressed individually but the geometry can still be shared.

Blender does cache GPU data per mesh even if they are identical. This could be limited by using instancing, but that is manual work ATM.

When focusing on linked static data we could reduce the memory by reusing identical buffers. this could be detected during loading. GPU buffers might also be shared in this case. When user adds an override the buffers should be re-loaded or duplicated.

In Eevee we could optimize this even further using instanced drawing.

CrowdSim3D does a similar approach where buffers and materials are reused when adding variations to a crowd.

13 Likes
3

I don’t have a clue if it’s BMW, MB or VW Group Data but it’s the same problem everywhere: you get so many parts that are not necessary or inside a door / engine bay that there is manual work required to clean up and delete all that parts. If you get access to Step-Data (Stepper) you can import that data directly and if parts are instanced inside the CAD file they remain instanced inside your Blend and share the same mesh data + they can be re-tesselated.
I have quite some experiments and one of the wonderful reasons why VRed was so fast was because not every single part got re-tesselated over and over again.
If you have a Mesh from that is already tesselated the only possible way remaining is culling everything that is not visible (we tried with baking and then a lot of python and it was SLOW) or getting proper groups in your import where stuff can be deleted due to proper naming conventions.

4

Oh! So Eevee treats all geometry like individual objects even if they are instances? That would explain why it chokes when I render a stadium with 70k people in it. Yep, that’s what I’m doing right now and it will be a kick ass shot for a big feature film. We use Eevee only for the volumetrics because Cycles just can’t handle all these lights in the stadium in the fog. But the rest is done in Cycles. That would greatly help us if Eevee would use instances. We have a few shows lined up where we will need to do crowds. All because of Covid.

5 Likes
5

No, it does not.

I think @JeroenBakker is talking about cases where the incoming data is not instanced, and Blender auto-detecting identical meshes.

4 Likes
6

Yes, Eevee uses instance drawing when it can be destilled from the data (for example vertex instancing).
Eevee does not use instance drawing when you use the same mesh in two different objects as it tried to minimize shader switching.

The case I tried to describe is the situation where the actual geometry is the same between two mesh objects. For example having two identical cubes but with different materials. They would not share the data or caches. When we know for sure that they are static (no deformation), identical and cannot be edited (linked) we can reduce RAM usage and reuse caches by sharing identical data.

I am not up to date with latest CAD approaches and will ask freecad for some feedback on the topic. Perhaps they have insight that we don’t.

7

Are you sure about that? If there are no modifiers on the object, they should share the same Mesh, and the batch caches are stored on the Mesh? That’s an optimization Sergey implemented at the depsgraph level so it would benefit both Cycles and Eevee.

1 Like
8

Not sure, was how I understood the system works. Will dive in the depsgraph to find out the details/boundaries.

1 Like
9

I don’t have exact insight into their pipeline, but they said they already cleaned everything up as good as possible, and that the remaining parts cannot be instanced or tesselated.
So right now it seems to be Maya vs. Blender in terms of geometry and memory usage… :slight_smile:

1 Like
10

You can find me on LinkedIn (Hartmut Nörenberg) - maybe we can have a not public talk about this topic.
Generally “cleaned” is not specifically good …

11

Hey, I suppose the initial steps to mesh editing performance optimization are still in progress, but I wanted to ask if there are any updates on this?

12

I have delved into this topic a little. and it seems that the necessary commits have been made almost recently, which solve the problem under discussion at least partially Custom Data: support implicit sharing for custom data layers · 7eee378ecc - blender - Blender Projects Geometry: Use implicit sharing for curve and mesh offsets · dcb3b1c1f9 - blender - Blender Projects. @sebastian_k, I think you can check it out

2 Likes
13

The recent move to primitive types for arrays also improves run-time memory usage, it should reduce edit-mode undo memory use too. See Mesh Struct of Arrays Refactor #95965.

3 Likes