Making sense of cryptomatte usage in third party programs

[txo]

Hello! First of all, let me say that I find it absolutely great that now cryptomatte is available in eevee! thanks a lot, @JeroenBakker! I think this is a good moment to try to make sense of how are cryptomatte stored and how it is supposed to be used in third party programs, e.g. Nuke, and maybe fixing the unconsistencies I found.
Note: all I talked about from now on is referring to rendering multilayer .exr, as we use in our productions

The output wasn’t consistent before in how the cryptomatte data is stored, depending on the method you used, the properties panel output or the compositor’s file output node. the thing is, now I find the same discrepancies between cycles and eevee, even when only using the properties panel.

How it works rendering via cycles-properties panel:

When opening the saved exr in nuke, the metadata information gives you this list of keys (with its associated values/hashes):

As you can see, the cryptomatte keys are stored individually (and separated from the rest of the channels), and so you can use them as cryptomatte layer selections (although even in this case is quite annoying to use since you have to always modify the metadata to change the . for a _ in the layer name). But in the end it works, so no unworkable problem.

How it works rendering via the compositor (file output node):

But then, when you use the file output node from the compositor, it gives you this metadata instead:

Here you can see that there is no cryptomatte key/value data. Instead, they are stored within the channel key, which seems to be a single list:
main.Combined.A:{1 0 1 1},main.Combined.B:{1 0 1 1},main.Combined.G:{1 0 1 1},main.Combined.R:{1 0 1 1},main.CryptoAsset00.A:{1 0 1 1},main.CryptoAsset00.B:{1 0 1 1},main.CryptoAsset00.G:{1 0 1 1},main.CryptoAsset00.R:{1 0 1 1},main.CryptoAsset01.A:{1 0 1 1},main.CryptoAsset01.B:{1 0 1 1},main.CryptoAsset01.G:{1 0 1 1},main.CryptoAsset01.R:{1 0 1 1},main.CryptoAsset02.A:{1 0 1 1},main.CryptoAsset02.B:{1 0 1 1},main.CryptoAsset02.G:{1 0 1 1},main.CryptoAsset02.R:{1 0 1 1},main.CryptoMaterial00.A:{1 0 1 1},main.CryptoMaterial00.B:{1 0 1 1},main.CryptoMaterial00.G:{1 0 1 1},main.CryptoMaterial00.R:{1 0 1 1},main.CryptoMaterial01.A:{1 0 1 1},main.CryptoMaterial01.B:{1 0 1 1},main.CryptoMaterial01.G:{1 0 1 1},main.CryptoMaterial01.R:{1 0 1 1},main.CryptoMaterial02.A:{1 0 1 1},main.CryptoMaterial02.B:{1 0 1 1},main.CryptoMaterial02.G:{1 0 1 1},main.CryptoMaterial02.R:{1 0 1 1},main.CryptoObject00.A:{1 0 1 1},main.CryptoObject00.B:{1 0 1 1},main.CryptoObject00.G:{1 0 1 1},main.CryptoObject00.R:{1 0 1 1},main.CryptoObject01.A:{1 0 1 1},main.CryptoObject01.B:{1 0 1 1},main.CryptoObject01.G:{1 0 1 1},main.CryptoObject01.R:{1 0 1 1},main.CryptoObject02.A:{1 0 1 1},main.CryptoObject02.B:{1 0 1 1},main.CryptoObject02.G:{1 0 1 1},main.CryptoObject02.R:{1 0 1 1},main.Depth.Z:{2 0 1 1}
So now there is no way to access them from the cryptomatte node (as far as I know, if I’m wrong please let me know), because they are mushed together with the rest of the passes.

How it works rendering via eevee-properties panel:

Ok, so that was the old inconsistency within Cycles. But now, the Eevee implementation seems to behave the same way the compositor does, storing all crypto data inside that single list, instead of independently in their own channels (key/value pairs):

Note that this last example looks the same as the previous one, even though one is rendered in cycles from the compositor, and the other is rendered in eevee from the properties panel. And the channel list looks the same as well, with the crypto data stored inside as list items.

Conclusion.

The file output node is a different conversation in itself, but I think these discrepancies in the way cryptomatte is stored should be addressed (in the very least, both rendering outputs from the properties panel should be stored the same way), and I believe now that a new implementation is being worked on it is a good moment to tackle it.
I don’t doubt there is some way to access that information through programming, but I think it should be available right from the start in a way that is more industry compatible.

Again, thanks for all the awesome work, you guys rock!

PS: I haven’t tested the fileOutput node version in eevee because for some reason the render doesn’t get written in disk in the latest version in 2.92

[JeroenBakker]

Thanks for the analysis. Be aware that the project isn’t finished and there are other patches that should land to get stuff better. From a developer perspective it is better to use multiple smaller patches.

• Store cryptomatte meta data
• New cryptomatte workflow

As you mentioned the file output node is a known issue as the compositor (including file output node) doesn’t take meta data into account. Somehow/somewhere a mapping needs to be made from the render result meta data and the meta data that is needed when using the file output node. The mapping is needed as the layer names could be totally different. I am not sure if this re-mapping is required by other tools. This is a specific topic as Blender supports any areas of a studio pipeline and the compositor is sometimes ‘mis-used’ as a pipeline integration tool. Many of these tasks should be made accessible via BPY or as regular output settings or via a nodal pipeline editor.

Cycles and EEVEE share the same properties and the goal is to have the implementation act the same (what will be done by the patches already available). About the API there is some discussion on the actual use case Cryptomatte BPY API If you could detail your use case there we should be able to design an API that suit the needs.

[statixvfx]

We are now doing a feature film and rendering with cycles and using cryptomatte in Nuke. However we need to render cryptomatte out seperately due to the file out node in the compositor doesnt store the metadata.

Since our compers are using crypto materials we need to re-render the beauty JUST to get the metadata in the file out panel instead of the using the compositor, landing us additional 20% rendering overhead pr shot for this. (we re-render the crypto pass with less samples)

Please, anything that can be done to store the same metadata the panel can in the compositors file out would be extremely useful.

[txo]

@JeroenBakker

Thanks for the analysis.Be aware that the project isn’t finished and there are other patches that should land to get stuff better. From a developer perspective it is better to use multiple smaller patches

Thanks for replying! I hope the analysis made some kind of sense…
I had seen the patches that you mentioned, but the thing that worried me (and why I made this post) is that the implementation that is being worked in eevee would work the same way as it does through the compositor, which is the less useful when trying to fit it into an Arnold-Nuke centered pipeline.

@JeroenBakker

Somehow/somewhere a mapping needs to be made from the render result meta data and the meta data that is needed when using the file output node. The mapping is needed as the layer names could be totally different.

I completely agree. I was just talking about the cryptomatte in here for conciseness, but indeed there is need for that in general. I actually made a post about it a little while ago, in which I talk a little more generally about the output. And I even think it should be encompassed by an overall design for a more structured renderLayers manager (I think now is kind of all over the place)

@JeroenBakker

The compositor is sometimes ‘mis-used’ as a pipeline integration tool.

The problem, in my opinion, is that there is no way for a flexible output (that can accomodate a studio’s needs regarding custom layer output, naming, etc) other than the compositor. I wouldn’t say it’s a misuse…I use it because I have no other option to render different multilayer exr files per view layer.

@statixvfx

Since our compers are using crypto materials we need to re-render the beauty JUST to get the metadata in the file out panel instead of the using the compositor, landing us additional 20% rendering overhead pr shot for this

Kind of the same happens in my case. If you have many viewlayers (e.g. main chars, secondary chars, some fg elements, and bg), that may not be even an option. You have to rerender every layer from the main output panel, otherwise if you use the “beauty”, the fg elements are overlapping your main chars, those are overlapping the bg and secondary chars, and so on. And then, weird artifacts appear at the borders when tryin to mask them and do any kind of heavy compositing.

[JeroenBakker]

The problem, in my opinion, is that there is no way for a flexible output (that can accomodate a studio’s needs regarding custom layer output, naming, etc) other than the compositor. I wouldn’t say it’s a misuse…I use it because I have no other option to render different multilayer exr files per view layer.

Kind of the same happens in my case. If you have many viewlayers (e.g. main chars, secondary chars, some fg elements, and bg), that may not be even an option. You have to rerender every layer from the main output panel, otherwise if you use the “beauty”, the fg elements are overlapping your main chars, those are overlapping the bg and secondary chars, and so on. And then, weird artifacts appear at the borders when tryin to mask them and do any kind of heavy compositing.

The render pipeline of Blender (the flow that happens when you press F12) is still hard-coded and only configurable via a few settings. Studios are working around it. I do believe that making the render pipeline more flexible would is the way forward. In this case the file output node wouldn’t be a part of the compositor anymore but the output-node of the pipeline. The compositor could then have multiple output sockets.
IIRC during Spring production (especially the volumetrics) multiple renderers (Cycles/Eevee/Workbench) were used to render and composited it in when needed. I will check with Andy on how it was composited. And we need to check what part will already be solved by View Layer overrides and what might shift into a more flexible pipeline.

[statixvfx]

We’re currently rendering the volumetrics as a linked scene with all the materials overridden by a single hold-out shader. We do this because we need to tweak the volumetric render settings quite a bit to get a clean and usable result using values that are not “compatible” with the main beauty render.
So we do each shot as 4 renders, where 3 of them are rendered in a batch. Beauty, Volumetric and Utility shaders are rendered, denoised and recomped into a single EXR. While our cryptomatte need is yet another Beauty render but with lower samples and saved via the panel instead of the compositor.

The speed vs noise difference between splitting up the beauty and volumetric is insane. A 2k frame this way takes about 20min now on a TR16c, and without splitting and just rendering combined takes well over an hour to get the same sampling convergence.

I digress, but both the EXR file outs should save the same type of data (metadata in this case)

[txo]

Ok, so here is one of my use cases:

Context

I work at a medium sized studio (currently around 70 people) that has a pipeline centered around maya(arnold)-nuke, and I am basically the only one of two people I know there that uses blender, at least on a regular basis.
We work in various tv series, so time and budget constraints are basically a law of nature, and in that scenario the studio cannot afford to take time from technicians and devs to deal with a software that is used marginally when there are lots of fires to put out already within the main pipeline environment. I imagine that the situation may not be some weird exception, given the gradual adoption Blender is experimenting throughout similar small-medium sized studios. What I meant is that I am basically working with what Blender offers out of the box, and some 3rd party addons, but no custom tools developed in-house.

Workflow example:

So, our renders coming out from Maya and into Nuke:
We usually separate the renders in 2 to 4 layers, depending the complexity of the shot. The composition usually follows this logic:

It is oversimplified a lot for the sake of clarity, but the green columns would be cryptomatte selection nodes (reading from the multilayer exr at the top) some of them with a selection already created in the preconfigured templates for the scene. Things like hair, shadows, AO, BG, characters, etc. all require different render settings regarding sampling, frame numbers, matte objects, shadow catching and casting, so they are renderered individually. Pretty universal basic stuff, really.

Besides that, often times there are more than one char layer when there are many of them in different terms of depth. One of the reasons is so the comp department can have better access to clean cryptomatte selection (to avoid edge artifacts that usually appear if some heavy compositing is done, due to AA differences in the borders between overlapping objects) , but also for scene optimisation. Whatever the reason, the cryptos have to be split too.

As you can see, such a workflow wouldn’t be feasible nowadays in blender, because you would need to have the renders separated via the compositor (which would mess with the metadata), and having it all inside a single .exr with A LOT of layers and passes would be a no go, for many reasons:if some layer fails all has to be rerendered, having to work with such all content of massive files every frame would kill the comp artists if they only wanted to preview the one layer, etc.

Another case:

Another problem is the incoherence between industry standard cryptomatte implementation and Blender’s. I have dealt with it several times when doing mattepainting work (for which I normally use Blender), because I have to provide a structure as close as possible to the one that Maya creates, so the comp department can work efficiently within a single comp file.

In the case of cryptos, it becomes an issue due to the compositor not recognising metadata. So I have to rerender manually (from the output panel) the minimum number of layers that I think we’ll need with cryptomatte, and create some custom masks if I need something else. In the image above, the matte input could be done in Blender, and maybe even the BG and FG if, for a particularly special shot, they are treated as a mattepainting. But chars would almost always rendered in Arnold, because they are integrated in the pipeline.

And, in terms of cryptomatte, it is kind of frustrating that being a industry standard, the implementation differs not only to that of maya in terms of . vs _, but within blender itself depending on from which panel the file gets outputted. Because all of that, I usually end up creating custom masks with emission shaders manually, but it can be a pain, honestly.

[statixvfx]

The cryptomatte manifesto not being written to disk through the compositor have been reported.
This is something we’re also struggling with, as it forces us to do a separate beauty render with lower samples just to spit out crypto from the panel instead of the compositor, landing us an additional 20% unnecessary rendering time pr shot.

That said, we have no issues with using 2.90 and 2.91 crypto output in Fusion or Nuke once its written correctly to disc.

Ps, deep support would be amazing, it would remove a lot of holdout / matting issues.

[txo]

@statixvfx

That said, we have no issues with using 2.90 and 2.91 crypto output in Fusion or Nuke once its written correctly to disc.

Do you know of any way to make the crypto passes be readable in nuke without adding a modifyMetadata Node inside Nuke? I’d love to know about that, because even though the crypto passes themselves are fine and the data is correct, in my experience it does not recognize the name of the pass until you change the dots for underscores, and so the crypto layers don’t appear in the popup menu.Or do you modify the metadata externally beforehand?

[statixvfx]

We updated to the latest version of Cryptomatte from PSYOPs github and we dont have any issues managing the layers or metadata manually.

Its just a readnode and cryptonode and its golden.

We’re on Nuke 11.2 i think.

[txo]

I hope it’s that easy. Thanks a lot!
Edit: just updated cryptomatte and you are absolutely right. It works like a charm. Thank you so much.
Although the other problems still apply, it really makes a difference.

[JeroenBakker]

Hi all!

Yesterday during the render & cycles meeting we discussed about the main issue when using cryptomatte + file output node and want to include a patch in 2.92 that solves this issue without overhauling the complete system.

I just added a patch that solve the basic workflow issue. Any testing/feedback is welcome https://developer.blender.org/D10016

This change supports the basic use-case where the compositor is
used to output cryptomatte layers with a different naming scheme to
support external compositors. In this case the Multilayered OpenEXR
files are used and the meta data is read from the render result.

Meta data is found when render layer node is connected with the
file output node without any other nodes in between. Redirects and empty
node groups are allowed.

[statix]

Fantastic Jeroen!

Does this also work if there are multiple renderlayer nodes connected to the file out ? (but only one of them have crypto enabled)

We often have several passes that needs to be rendered (volumes, light categories etc) and we compile them into the same multi layer exr, where the beauty pas is carrying the cryptolayers.

[txo]

sounds great, thanks! I will give it a try as soon as I can.

[machieb]

@JeroenBakker: Hello, great to see the improvment with cryptomattes and the file output node! I wrote the first bugreport about this, more than a year ago.
How can I test the patch? I´m not a developer and can´t compile blender.
Greetings Marcus

[JeroenBakker]

I updated the patch description with example files. In stead of testing it by building it you could also download the openexr file and see if it is can be use. https://developer.blender.org/D10016
Multiple viewlayer should also be supported. The system is setup generic so it could work with any render engine that support cryptomatte (RPR for example).

[machieb]

@JeroenBakker: Thank you very much!
I testet your *.exr file with the compositing app fusion 16 and with Photoshop (exr-io plugin).
It works as it should! Great that the file output node now supports cryptomatte metadata!

[statix]

Confirming that the EXR you posted works fine in Nuke. The file doesnt seem to contain the cryptoAsset layer but Material and Object works fine.

On a separate note, does this mean all of the metadata the panel offers will now run through the file out node ? Its missing lens, render time, node etc. All of the other metadata Blender writes when using the regular panel.

None the less, good job!

[txo]

@statix

Confirming that the EXR you posted works fine in Nuke. The file doesnt seem to contain the cryptoAsset layer but Material and Object works fine.

Yes, it works for me too. Just dragged it into Nuke, connected it to the crypto node and it works (asset is missing, but in the example above it can be seen that it is not connected to the output node, so all is good). Thanks a lot!

[machieb]

I can confirm that the *.exr also works with Adobe After Effects. I think so we tested the widely-used compositing applications.