Particle Nodes UI

[JuanGea]

Yes, that’s why I was mentioning it, because it also supports points, but in general when I go near a VFX artist and I ask for the best file format for particles, both in features and weight, they always mention PRT, in the past the best format i’ve worked with regarding particles is PRT, I tried to use Alembic for a sand simulation, forget it, 30 million particles meant around 1Gb per frame, while PRT was 240Mb or so.

BTW this is from one year ago, and I don’t have the caches anymore, but as I said, you can try it, maybe USD is great, but if it’s the same as Alembic, it’s not optimized for particles, but for general usage, even when particles may be rather important.

PRT can be used with any software in the 3d DCC Industry, Max, Maya, Houdini, also there is MagmaFlow to do special modifications to particles, it’s fast and solid, and the biggest benefit is, as I already said many times, performance.

[EloiAndaluz]

Hello! If it helps of anything, I have been working with particles in productions for over 8 years, particles in general for over 15 years:

The way to go when moving particles between apps that is reliable latelly is PRT, it has multiple advantages:

-Has been designed to work with particles from the ground.
-Available in multiple softwares: Max, Maya, cinema, Houdini, Storm, others.
-Open.
-Small cache sizes.
-PRT comes with Krakatoa in max/maya/cinema, and it comes with magmaflow. Magma its a nodal based modifier to access/modify all types of data stored on cache on a time independent way. Its extremelly useful to be able to do postcache tweaks or even effects.
If you implement a prt reader on everything nodes, and you can manipulate data directly even better.
-Partitioning over prts is integrated with krakatoa, allowing to change seeds for your particles and do multiple caches allowing for huge amount of particles to be created.

Alembic is Okish for particles, but the way to store multiple data on particles, and how to access this data is not the most optimal (at least in max). Its heavier as well.

[SteffenD]

I didn’t “vote” for Alembic / USD for particles
I’m also against it, because despite being (another) industry standard, especially when working with Max and Maya artists, the flaws in the Autodesk implementation of Alembic become very obvious. Try to import an Alembic containing only points to Maya → Empty result, no points.
You have to jump through hoops to get points from an Alembic to render in Maya.

The optimizations of Alembics only come to play when outputting monolithic ABC files containing the whole frame range. But this is very error prone when it comes to caching simulations. It’s a bit like rendering directly to a movie format which is dangerous because one single hickup on the server you’re writing to or a crash of Blender and the whole previously rendered frames are gone, the movie file is useless or at least heavily damaged. The same goes for Alembics.

That’s why you should render discrete frames and use a cache format that uses discrete files. You can always continue rendering / simulating at the frame that crashed before and continue writing.
You can of course also write discrete ABC files per frame, but as I said above, most of the optimizations aren’t working then.
PRT might be OK but now that we have a choice and want Blender to become an industry standard I would vote for a truly open standard cache format. If you ask around in a bigger studio, many artists still want “OBJs” or “FBX”, some haven’t even heard about Alembic. But I still cringe when I hear the word “OBJ”!!

It might be easy to allow the user to output whatever format he wants after the simulation is done, even Alembic (or OBJ sequences, cringe), but during the simulation I would really vote for OpenVDB points because they’re a documented open standard and they’re super light-weight, fast and support all kinds of attributes.

[JuanGea]

PRT is Open, and it’s a standard, I would not ask for it in a different situation

Regarding OpenVDB, as I said it can do that, but it’s not designed to do that, anyways, let me stress again that I’m not against supporting particles in ABC or USD or OpenVDB, I’m just saying that PRT it’¡s very efficient and would be great to support.

I won’t continue with this talk, I’m repeating myself and I don’t want to repeat myself… more hahaha

There is still time for all this, will see the outcome, I’m very excited with particle nodes implementation in master

[SteffenD]

No worries, I totally understand you.

Just one last addition from my side: PRT looks good (and quite open) and offers a lot of features, no questions asked.

The only thing I can’t agree with is that:

OpenVDB, as I said it can do that, but it’s not designed to do that

The VDB points stuff was added to do exactly that
https://www.openvdb.org/documentation/doxygen/points.html#secPtCompression

[JuanGea]

If I’m right, and please… take into consideration that I may be wrong, the points in OpenVDB are stored in a similar fashion to volumes, using voxel partitions, so in the end it’s not just points, it’s more like what mantaflow does, points inside voxels, and this comes with some limitation, like the fact that you cannot have a super big volume with 1 particle on each point, because you would be generating a super big volume, taking up memory and space.

Meanwhile PRT, while can work with Voxels, and has a similar mode to OpenVDB (it was created before OpenVDB was mainstream and what it’s now) has also a particle mode that can store particles no matter the position in space, and it won’t generate that amount of intermediate voxels or the required volume because it works directly with points, not with volumes filled with points.

Please I’ll repeat myself, I MAY be wrong, but it’s what I have understood about how OpenVDB works

[SteffenD]

I just fired up Houdini to do a little test. I created 10.000.000 points and exported them to different formats.
The first thing I noticed was that despite Houdini’s long list of formats PRT was missing. Googling it gave me several odforce forums links that all pointed to a PRT exporter SOP on github. This link was dead.
So no PRT im-/export in a vanilla Houdini 18.0.
I exported the points to 4 different formats:

Here are the file sizes (no special attributes, only point position):

Don’t ask me what “pc” is, IIRC it’s something Maya-ish but I might be wrong.

The export times were:
VDB - Unnoticeable, at first I thought the export didn’t work because it was instantaneous
ABC - 2 seconds
PC - 4 seconds
PLY - 20 seconds

If someone could convert any of these formats to PRT to get a comparison, I would provide the point cloud of course.

P.S.: I also exported it to bgeo.sc which is Houdini’s native geo format which is highly optimized and BLOSC-compressed (just like the VDB) and it came out at 111,5MB

[statixvfx]

And USD is 114mb pr frame with the same settings.

I agree on not rendering directly to monolithic formats (we used bgeo before publishing to abc) but where alembic shines is when you have non varying attributes pr frame. So things like point UVs and other static data are not duplicated across frames.

You should run another test with lots of attributes, static and varying across a framerange to get a more correct picture.

But good call on VDB points, I might have to reconsider my stance on using it as a particle container at some point (hehe).

The good thing about USD is that it can contain and reference both ABC and VDB files as well as store the data if needed.

[SteffenD]

I gave it another try and added an animated color (called “Cd” in Houdini) attribute to the points. Then I exported 10 frames.

The PLY version took ages to export and turned out to be 500MB per frame. UGH!
The PC version was faster and took “only” 225MB / frame.

The interesting part is the VDB vs. ABC:
I exported a monolithic Alembic which turned out to be 1.4GB in size. The VDB sequence was 1.5GB, not too bad (150MB/frame)
I also exported an ABC per frame and the accumulated file size for 10 frames went to 3.2GB (320MB/frame) because the compression tricks Alembic does aren’t working any more this way.

So for final export or transfer to another application ABC might be fine, but as a cache format with discrete files it’s not the best.

[JuanGea]

Very interesting tests, I don’t know where is the Houdini PRT plugin, could be interesting if you could test it too.

The thing is that in my experience the usual channels in particles are, as a minimum:

Position
Rotation
Scale / size
Velocity
Color
Custom channel 1 (because I don’t remember the channel itself, but it was a state channel, to detect when the particle was in one state or another, like active/inactive/inside/outside, etc…)

With all those channels the ABC file per frame was a bit bigger than yours, around 1Gb per frame, meanwhile PRT with those exact channels was around 250Mb per frame.

If we pick what you have there, the size of both full ABC and OpenVDB are not bad, but I suspect that PRT would be better, also keep in mind that such a big amount of particles cache is usually better cached as sequence, and not a full file, so the loaded file per frame is as small as possible (memory footprint, load times, etc…)

Could be awesome if you manage to test PRT.

BTW if you can, to get a similar solution, do a granular simulation, don’t export just the points, but do an actual granular simulation of 10 frames, just so the basic channels are generated, and check the size per frame.

Thanks for the tests, very interesting!

EDIT: thanks to @EloiAndaluz for the link, here is one Houdini PRT plugin, free and open source

Releases · flipswitchingmonkey/houdini_PRTROP

[SteffenD]

Preface:
I hope that in Blender’s future there is no such thing as “usual channels”. That’s what made Houdini what it is today. It’s completely agnostic to what data it gets fed and what data it outputs. It lets you add as many attributes as you want to do things.
I want to write and use stuff like “the distance to my neighbouring particle” as a float value or “the name of the rigid body that collided with my emitter to generate me” as a string in an attribute to do things with it afterwards.

If PRT allows me to do so, fine. With OpenVDB Points I know that it allows me to do so as it has already been working in Houdini like that for years.

I just want Blender to be as open-minded as possible when it comes to dealing with points / particles and especially when the “Everything Nodes”-era is coming! Can’t wait for it!

Thanks for giving me the link to the latest developments for PRT in Houdini. I’ll try them out and keep you informed!

EDIT: OK, I will not be able to try them out because they’re Windows only…

[Bobo_The_Imp]

@SteffenD

I hope that in Blender’s future there is no such thing as “usual channels”. That’s what made Houdini what it is today

I don’t mind if there are some standard use ones that are “on by default”. I realize we’re talking about import/export here, but just like the attributevop, its sometimes nice to have so default parameters for convenience.

That is, provided blender has the presence of mind to allow any attributes you want. So long as that is held, some defaults don’t really get in the way in my opinion. Worst case scenario you just check a couple of boxes off/delete them.

[SteffenD]

I totally and 100% agree!

I don’t want to reinvent the wheel every time but I want to have the power to use any attribute I want at any time!

[JuanGea]

@SteffenD

I hope that in Blender’s future there is no such thing as “usual channels”.

What I meant with “usual channels” is that those are the “usual channels” I use, not the “default” or limited channels

Channels must be totally arbitrary and tied to user definition and selection.

[megalomaniak]

@statixvfx

PRT not seeing an update during the last 5 years (and the houdini plugin havent been updated in 3) means its a dead end .Sorry to say.

Counter point: Wavefront .obj has been a dead format according to your criteria for decades now and yet we all still widely use it. There is a benefit to being a mature and stable format.

[statixvfx]

Counter point: Wavefront .obj has been a dead format according to your criteria for decades now and yet we all still widely use it. There is a benefit to being a mature and stable format.

You got that wrong. OBJ still persists BECAUSE it got widespread adoption and had momentum, unlike PRT.
OBJ IS a dead format, as you’ll never see updates to the OBJ specs and most sensible pipelines have moved to more modern formats.

If a project on Github haven’t seen any updates it doesn’t mean its a stable and finished project, it means its been abandoned by its creator. Even thinkbox themselves have ventured away from VFX software since it got acquired by Amazon. Its mostly Deadline and cloud related services.

I don’t think people realize just how volatile open source / FOSS projects are. They do live and die by their community, and in turn; by their adoption.

Also, if you’re still using OBJs in 2020 you should really consider some modern alternatives.

[JuanGea]

I beg to disagree with you about the non widespread format.

May not be the main format for apps, since it’s a format created by a third party, open source, and we all know how much companies love open source file formats (even Alembic was kind of forced into Max after many years of being available for other packages like maya)

But if you ask people in production, PRT is widely used because it’s stability and performance, @EloiAndaluz, the guy who spoke here before, is constantly doing vfx and particles in large feature film productions, I really trust his knowledge and experience, and they could be using USD or Alembic, but they prefer to use PRT for the many reasons we spoke about.

Once again, this is not to say that particles should be ignored or postponed in favour of PRT, not at all, but to give you a picture of the situation, STORM, the granular simulation software (used in Game Of Thrones for example) implemented both, Alembic and PRT as particle file formats, well it seems the dev has to fight to properly implement Alembic into the system, but the PRT implementation was a matter of half a day, so supporting PRT is probably not a hard or complex task

[EloiAndaluz]

@SteffenD. If you are looking for example files here you can download examples exported from storm; https://effectivetds.com/resources/fx-tools/storm/
You have alembic, prt and bin files with different custom data exported from storm.
In the other hand, my preoccupations with Thinkbox stopped developpement for PRT are minimal. What else do you want? prt its as basic as it gets, you save unlimited data for particles on a file. Its the fastest, smallest format already available. Also thinkbox tech support in all their products its still excellent.

[SteffenD]

Sorry for replying that late but I was away for a few days (some may call it “holidays” ).

I downloaded the PRT and ABC variants of the simulations you recommended and adapted my Houdini file accordingly. File sizes are 92,9MB for the ABC sequence and 68,5MB for the PRTs:

What I did was simply load the ABC sequence (not a monolithic file which makes sense for a simulation cache vs. a final export format) which gave me some points with the following attributes:

• Point position as a float vector
• Velocity as a float vector
• id as a 64-bits integer
• width as a float

The PRT sequence I couldn’t use because Houdini simply doesn’t support it, but the data coming from it should be exactly the same I get from the ABC sequence.

I then exported this sequence as VDB Points sequence and as a monolithic ABC (just for fun).
The results were:

• VDB Points sequence ~322kb / file resulting in 60,1MB (8,4MB less than the PRT sequence, over 14% less in size). Export time was not measurable ( = instant).
• Monolithic ABC 72,2MB. Smaller than the original per frame sequence but despite all optimizations larger than the PRT and the VDB. Export time was superfast as well.

[NizarAmous]

houdini prt exporter :

GitHub - ThinkboxSoftware/HoudiniPRTExporter: Python module for Houdini that allows users to...

You need to do it in the ROP not in sop

Btw we disgraced from the topic