Taking a look at OptiX 7.3 temporal denoising for Cycles

[deadpin]

The future looks to more adaptive sampling based at least as far as cycles-x is concerned. Maybe it would be good to try adaptive sampling and investigate at which threshold level the results from New Optix meets/exceeds that of the Old version.

E.g. you might find that Old needs 500 samples with a 0.002 threshold to get a decent result even though the denoiser is technically not stable. And New may only need 500 samples with a 0.005 threshold to be perceptibly the same but with a 15% render time reduction. Or something along those lines.

Alternatively try bumping those sample counts up to 10000 and let the threshold fully dictate the stopping condition etc. That way you can give a more accessible blanket statement at the end and it wouldn’t be tied to the scene as much: i.e. To use the AI temporal denoisers, you need to let things converge to the 0.005 threshold level by using a sufficiently high sample count regardless of scene.

[Alaska]

Sorry, it took so long, but I’ve now uploaded a video demonstrating OptiX temporal denoising in the classroom scene at 100 and 300 samples.

It can be found in the link below. It’s a 1920x1080 render, however I have upscaled it to 3840x2160 in my video editor so it can be watch in “4k” in YouTube which has a higher bandwidth and should thus reduce compression artifacts. 4k will probably not be available straight away, so you may need to wait a bit for that to be processed.

OptiX temporal denoising tests with Blender Cycles

[YAFU]

Good idea to go back and forth repeatedly in the video to better appreciate it!
Where the difference between standard and temporal is most noticeable is in the edges of the shadows on the desks and on the floor.
I think that for production with render at high samples rates, optix temporal denoising is doing an acceptable job.

We still have to be sure if we are doing things right. In the little demo on the nvidia site that example looks much more extreme than mine at 50 samples but with even better results. But this could be only under some special condition for advertising purposes.

But for now the results we have obtained I think are acceptable.

[Alaska]

@YAFU

Where the difference between standard and temporal is most noticeable is in the edges of the shadows on the desks and on the floor.

Another thing I noticed looking back over the un-compressed version of the video was that the ceiling with temporal denoising is much better. With standard OptiX we get the “moving blotches” effect and with temporal OptiX it’s close to stable (Even at 100 samples).
Technically yes, this part of the scene is easy to denoise. It’s a low contrast area with very little dis-occlusion. But it’s just something I noticed.

@YAFU

We still have to be sure if we are doing things right. In the little demo on the nvidia site that example looks much more extreme than mine at 50 samples but with even better results. But this could be only under some special condition for advertising purposes.

Link for reference: https://lh3.googleusercontent.com/71PXAbmvgoLVBMeoaQdhiu1uqdaDw5YkMCeaE4GFjGMg0vX2nSdmxtUF6qJK4vjqqm4QB8aSNskBNTujNWga_XnJH5HyBJXt4SEeo00lkCH4tgkfMmZ8IgnGCnKctQfZt1rH2ZC5
Looking back over the OptiX example scene shown on their website, it’s hard to make definitive calls on anything because the resolution is so low, but it still looks like that even with temporal denoising there are still a bunch of “moving blotches” (primarily around areas of detail). Everything else looks mostly fine, but that’s probably down to two simple factors.

1. It’s low resolution.
2. The areas that are fine are basically just flat colours. E.G.Solid grey wall. This is similar to the ceiling in the classroom.

I think we may be doing it right? Maybe not? I don’t have the technical knowledge to say anything with any certainty.

Maybe OptiX temporal denoising isn’t as good as I expected it to be. It’s just a little tool to do the final clean up on animations once its rendered to a almost perfect state. Either way, seeing this integrated into Blender would be nice, and if not, I’m sure a small GUI app can easily be made to make OptiX temporal denoising more user friendly.

I might continue experimentation? Try simple scenes with small amounts of detail? Try scenes with lots of detail? Try different sample counts? etc.
We’ll see what happens.

[Alaska]

Sorry, this is just me kind of going on a tangent, expressing my opinions, and hypothesizing.

To start off, OptiX temporal denoising is great. The temporal option does increase temporal stability, however, it doesn’t live up to my expectations. I’m just going to talk about where I think OptiX temporal denoising falls short.

1. I probably had too high of expectations. For almost everything I do in Blender, it’s denoised with ODIN. And I do this because from my personal experience, OIDN does a better job at denoising than OptiX. It produces fewer AI blotches, achieves a more accurate brightness (when rendered at a really low sample count), and seems to preserve more detail. And yes, I do have OptiX setup to use colour, albedo, and normals. Now, because of OIDN better image quality and my use of it for generally everything, I kind of had this expectation that OptiX would match that plus have temporal stability. This is not the case.
2. OptiX doesn’t appear to take enough temporal information into account. In the documentation for OptiX it has a small note about how the temporal denoiser works. It takes information the current frame, motion vectors, and either the previous or next frame and tries to produce a temporally stable result. This means OptiX has a temporal memory of 1 frame. This is better than 0 frames (no temporal denoising), but Disney has shown that scenes (even without motion) can benefit from 3 to 7 frames of temporal memory (Source: Check the video or Page 10 of the PDF). I personally believe OptiX needs a larger “temporal memory” to achieve a better result. And hopefully that will come in a future update.
3. OptiX can be trained with various different models to handle different scenarios. It’s possible the default model in OptiX isn’t well suited for the noise produce by Cycles, or that the temporal denoising model just isn’t mature enough to be of a high quality. This sort of thing will simply be fixed with future updates or by having someone train a new model (I’m fairly sure OptiX has an option to train and use alternative models). Also, AI is complex with many different settings you can adjust. OptiX has simplified this process down to “Input images and press denoise”. As a result we lose a lot of ability to tweak settings to get the perfect result for a certain scene, same goes for OIDN. But as a plus side, both are easy to use and aren’t intimidating to new users.

I don’t know, I’m just ranting about things.

As a side note, I’m currently running more tests with OptiX denoising. May post results soon?

[YAFU]

Yes, in my tests I also get better results with OIDN for still images, but it still has the same problems for animation as well. OptiX’s main advantage is that it is faster running on GPU.

Well, this is the first version of OptiX denoiser with temporal denoising feature. We could expect improvements from nvidia with subsequent releases, right?

[Alaska]

@YAFU

Well, this is the first version of OptiX denoiser with temporal denoising feature. We could expect improvements from nvidia with subsequent releases, right?

Most likely yes.

As a side note, I ran a test at 16, 50, 100, and 300 samples with the classroom scene where I used material overrides to replace everything with the “default principled shader”. As expected, the ability of the temporal denoiser to produce a clean and stable image is greatly increased as it’s not trying to retain texture detail. However, areas that still had detail (in the form of geometric detail) still had issues. This could just be a limitation of the temporal denoiser or the motion vector issue we’ve kind of been stepping around.

And doing some simple tests, it seems like it might be the motion vector issue… The motion vectors seem to be backwards… This all depends on how you interpret the data, but here’s what I found:

With my interpretation of the data, OptiX is expecting motion vectors that give you the direction the pixles have to move to get to the next frame. Cycles is giving you the direction the pixels have to move to get to the previous frame.

I could be wrong on this. But if this is true, it seems like this may be one of our major problems.

[lsscpp]

Nice tests @Alaska, thanks. My opinion:
I am quite disappointed from this Optix temporal feature, because it simply doesn’t clear the issue, it makes it just a bit less evident. The trembling pixels are still there and not only: the image pays a cost in terms of textures sharpness. Check the zoomed clips.
I’m speaking of 300 samples tests, because 100 are “unsellable”… and overall at 300 I almost prefer the noisy clip!

[Alaska]

It’s possible my tests were less than ideal due to an issue with Motion vectors. I’m testing a “fix” and if it’s any better I’ll upload a video demonstrating the results. If it’s not, then I’ll probably just make a comment about it.

But I generally agree with you, OptiX temporal denoising, as I’ve seen so far from my own (potentially flawed) tests, just isn’t that great. It’s better than nothing in cases like the 300 sample scene, but it’s just not as good as it could be. Hopefully this will be fixed with updates.

[YAFU]

Keep in mind that we still do not know if we are using this “flow” or vector pass correctly. It would be good if Brecht, Lukas or Stefan are reading this and have some time they could give us some clue about it.

[Alaska]

@deadpin

The future looks to more adaptive sampling based at least as far as cycles-x is concerned. Maybe it would be good to try adaptive sampling and investigate at which threshold level the results from New Optix meets/exceeds that of the Old version.

In general, adaptive sampling results aren’t going to be that different from non-adaptive sampled renders. The idea is that adaptive sampling is supposed to produce a image with a noise level similar to when the feature is turned off but with increased performance by not sampling where it’s not needed.

I could run tests with different noise thresholds in Blender, but at the moment results from OptiX temporal denoising aren’t that great, even with adaptive sampling turned off, so these tests may have to wait until later.

[Alaska]

In the meantime I’m running a test, I made small adjustments to the animation in the scene, and am rendering it normally, but rendering the “flow” pass with the animation reversed. I can then flip the flow pass naming scheme and in theory it should be what OptiX temporal denoising expects…

However, there could still be issues with the data being in the wrong format? Not sure. We’ll find out soon.

[YAFU]

I have found this documentation from nvidia about motion vectors or optical flow:
https://developer.nvidia.com/blog/tag/motion-vectors/

And this:

Optical flow generated from cycles render is not proper

I don’t know if it could be useful to you, I don’t understand a word

[deadpin]

Yes, it’s not really about that directly (though the use of pmj sampling may show more or less splotchy patterns; I haven’t been a fan of pmj really). The goal is to understand at what noise threshold the denoisers become… “acceptible”. Adaptive is the only tool we have right now to measure some form of matematical noise level during a render. And it may be the primary tool of the user in the future.

Having the raw sample counts from your experiments is still useful, but they’re not transferable to other scenes directly and it’s difficult to understand how much further you’d have to go to get acceptible results.

[Alaska]

Okay, that makes more sense. I’ll look into that once I’ve figured out this motion vector thing.

[Alaska]

That looks way more like what OptiX is expecting. I’ll investigate it further and see what I can get out of it.

[Alaska]

I’m going to be honest, I’m lost. Changing motion vectors to Optical flow is probably what we need to do to resolve this motion vector issue. And Tobias Weis has a guide for this (as YAFU pointed out), I just don’t know how to use the python snippet to process a image. Because I have very little knowledge of python.

[Alaska]

I just thought I’d add some information here that’s a combination of what I’ve found out myself and what I’ve read from others.

---

1. The motion vectors produced by Cycles contains two sets of motion vectors. This is mentioned in the Blender manual and Tobias Weis’ article on how to generate computer vision data from motion vectors from Cycles.

Vector:
Motion vectors for the Vector Blur node. The four components consist of 2D vectors giving the motion towards the next and previous frame position in pixel space.
–
Source: Blender manual https://docs.blender.org/manual/en/latest/render/layers/passes.html

Important information: The flow-values are saved in the R/G-channel of the output-image, where the R-channel contains the movement of each pixel in x-direction, the G-channel the y-direction. The offsets are encoded from the current to the previous frame, so the flow-information from the very first frame will not yield usable values. Also, in blender the y-axis points upwards. (The B/A-channels contain the offsets from the next to the current frame).
–
Source: Tobias Weis’ article http://www.tobias-weis.de/groundtruth-data-for-computer-vision-with-blender/

Looking at this in Blender this seems to be true. And just to simplify and explain things, here’s a quick summary:

The motion vectors generated by Blender are 2D motion vectors. The 2D motion vectors describe the motion of a pixel along the X and Y axis based on its speed. These X and Y axis are saved to the motion vector pass using colour channels.

Red=X axis (Horizontal motion)
Green=Y axis (Vertical motion)

Blue=X axis (Horizontal motion)
Alpha=Y axis (Vertical motion)

There are two sets of X and Y axis saved. One describes the motion vectors from the current frame to the previous frame and one describes the motion vectors from the next frame to the current frame.

Note: The motion vectors can be saved with negative values (E.G. -10) to describe movements in the opposite direction. Hence it is important to save the motion vector pass in a float image format, like an EXR.

---

2. OptiX temporal denoising expects a 2D motion vector pass and from looking at the example scene provided by Nvidia, the 2D vectors are saved to the Red and Green channel of the EXR files.

Red=X Axis (Horizontal motion)
Green=Y Axis (Vertical motion)

This lines up with Cycles. So why do the motion vectors behave differently?
Because of one thing. According to the OptiX documentation OptiX expects the motion vector pass to describe motion from the previous frame to the current frame. Cycles provides a motion vector that describes the motion from the current frame to the previous frame (plus extra for future frames). In theory the “plus extra” part (Blue+Alpha channels) should be ignored by OptiX, it only needs the Red and Green channel, so that should be a non- issue, but the fact the direction of the motion vectors is reversed is a issue. All the motion produced by Cycles is backwards (according to OptiX).

In temporal denoising modes, pixels in the two-dimensional flow image represents the motion from the previous to the current frame for that pixel. Pixels in the flow image can be in one of two formats:

• OPTIX_PIXEL_FORMAT_FLOAT2 (two floats, XY)
• OPTIX_PIXEL_FORMAT_HALF2 (two halfs, XY)
  –
  Source: Nvidia OptiX documentation: https://raytracing-docs.nvidia.com/optix7/guide/index.html#ai_denoiser#temporal-denoising-modes

So, how do we fix this issue? I believe it’s just as simple as flipping the direction of the X and Y values for the current frame to the previous frame motion vector and saving that. In the process, I’m also going to get rid of the motion vectors for the next frame as OptiX doesn’t need it and I want to be on the safe side. This can be done with a few approaches, but here’s two that appear to work:

The results from these appear to match the general format expected by OptiX. The last thing I’m unsure about is if the magnitude is correct, but I believe it is.

–

I believe I have run some denoising tests using these new motion vectors (I’ve done so many tests I can’t be 100% sure about it) and I didn’t notice much of a difference. Will give it another try some time soon.

On another side note, the OptiX documentation says this about temporal denoising

The OptiX SDK provides the OptixDenoiser sample which could be used to verify properly specified flow vectors. When invoked with -z, the tool looks up the position in the previous frame of each pixel using the flow vector. The pixel value from the previous frame position is looked up in the current frame position and written. When denoising a sequence with the -z option, frame N with motion applied should look similar to frame N+1 in the noisy input sequence. There should be no major jumps when comparing these images, just shading differences as well as differences due to disocclusion and so forth.
–
Source: Nvidia OptiX documentation: https://raytracing-docs.nvidia.com/optix7/guide/index.html#ai_denoiser#temporal-denoising-modes

But the most important part to me is this:

When denoising a sequence with the -z option, frame N with motion applied should look similar to frame N+1 in the noisy input sequence.

Does this mean the noisy input should have very little variance between the previous frame and the current frame? As in, animated seed should be turned off? Or does it mean that the overall noise should be low so the differences between the previous and current frame are small? If it’s the latter, then that would explain why we don’t get great results. 100 samples still produces a lot of noise which can cause issues with the denoiser deciding that the scene is too different and thus doesn’t apply the temporal information?

Or am I just mis-interpreting the wording and instead it’s basically trying to say “temporal stability will be increased when using temporal denoising”.

[Alaska]

@deadpin I’m currently working on rendering and testing scenes with various noise thresholds to see what setting is required to make the animation temporal stable with:

1. OptiX standard
2. OptiX temporal
3. OIDN

I will post results and probably a video when I’m finished.

[lsscpp]

I might save you some time: OIDN is terrible at temporal stability. I found myself reintroducing some of the original noise (RGBMix .6 or .7) to avoid the “frying-pan effect”