Thoughts on making Cycles into a spectral renderer

[pembem22]

@Eary

Is there some other things besides these, like Optix support? Is there a complete list anywhere? I would like to know the “road map” for the initial merge.

Here’s what I remember that needs to be done:

• fix render passes bug
• add spectral reconstruction
• fix Principled Hair shader
• fix color management issues
• test OptiX support (I don’t have OptiX compilation set up, so it would be great if someone can build and test how it works)

@kram1032

Maybe this should become like an exposed-to-experimental-mode parameter or something. Definitely not a thing most people want to mess around with, but for the purpose of testing it out, hiding it behind the Experimental Features checkmark might be a good idea.

That can be easily done since the SpectralColor data type is defined only in one place and can be changed to any float* data type:

  #ifdef __SPECTRAL_RENDERING__
- #  define SPECTRAL_COLOR_DATA_TYPE float4
- #  define CHANNELS_PER_RAY 4
+ #  define SPECTRAL_COLOR_DATA_TYPE float8
+ #  define CHANNELS_PER_RAY 8
#else
#  define SPECTRAL_COLOR_DATA_TYPE float3
#  define CHANNELS_PER_RAY 3
#endif

typedef SPECTRAL_COLOR_DATA_TYPE SpectralColor;

The problem is, you would need to include and compile two different kernels for each variation. This increases both compilation time and the size of the Blender package. CUDA and OptiX kernels alone take 107mb of storage when unpacked, and adding two more variations of kernels would triple the size.
But it definitely can be included as a build option.

[pembem22]

@Eary

But I still see this even after disabling Adaptive Sampling:

Just cannot get it to work

Do you have more details about that errors, maybe somewhere in the logs or in the console?

[Eary]

@pembem22

Do you have more details about that errors, maybe somewhere in the logs or in the console?

There are a ton of them, here:

click here to see error messages

Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
Invalid context in cuStreamCreate(&cuda_stream_, CU_STREAM_NON_BLOCKING) (C:\blender-git\blender\intern\cycles\device\cuda\queue.cpp:33)
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
System is out of GPU and shared host memory
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
System is out of GPU and shared host memory
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
System is out of GPU and shared host memory
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
System is out of GPU and shared host memory
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
System is out of GPU and shared host memory
Invalid context in cuCtxPopCurrent(NULL) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:31)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\util.cpp:26)
System is out of GPU and shared host memory
Invalid value in cuCtxDestroy_v2(cuContext) (C:\blender-git\blender\intern\cycles\device\cuda\device_impl.cpp:147)

Most of the lines are repeating so I picked the last few lines

[Eary]

@pembem22

Updated tests:

I noticed after the darkening problem is fixed the color in the spectral one looks colder now? Is this related to this?

@pembem22

add spectral reconstruction

EDIT: Just tried to compare the previous 2.93 build with the X Spectral buid on the principled hair.

Both are 4 samples with spectral checked. And yeah the X build’s hair on the right does look problematic.
Also it is very obvious in this comparison that the X build have a really strong green tint in the spectral mode as well, wonder why.

I also tried an older Spectral X build that was before the darkening fix:

Hair shader still problematic (32 samples). No tint difference, just slightly darker.

Is this intended? I am starting to think this might be intended, as wavelength importance sampling would mean certain wavelength gets sampled more, therefore it’s expected to see the difference in tint?

[pembem22]

@Eary

I noticed after the darkening problem is fixed the color in the spectral one looks colder now? Is this related to this?

@pembem22

add spectral reconstruction

The current spectral reconstruction method is not perfect and I think is the reason for those differences. It just wasn’t as noticeable before because of darkening issues. The new spectral reconstruction method should give much better results.

@Eary

Is this intended? I am starting to think this might be intended, as wavelength importance sampling would mean certain wavelength gets sampled more, therefore it’s expected to see the difference in tint?

That’s definitely not intended, all emissive materials must look identical, including background. When using importance sampling, if certain wavelengths are selected more often or rarely, they are weighted accordingly and multiplied by corresponding values to ensure correct results.

[kram1032]

what is the new reconstruction method? Is it this now RGL | A Low-Dimensional Function Space for Efficient Spectral Upsampling ?

EDIT: apparently there is a variant that can do even wider gamut spectral upsampling by going into flourescence. Not sure if that’s a thing we want to pursuit immediately but might as well share. It seems to be a cool option for sure

Wide Gamut Spectral Upsampling with Fluorescence

Jung2019Wide.pdf

Plus followup to that paper:

Improving Spectral Upsampling with Fluorescence

009-012.pdf

---

And another paper seems to even discuss spectral Rendering in both the Offline (Cycles) setting and Real Time so that may also be interesting to look into. Maybe some spectral stuff could, in fact, eventually be supported by Eevee! (Sounds like the spectral realtime engine they have built is currently fairly limited though)

Using Moments to Represent Bounded Signals for Spectral Rendering

Peters2019-CompactSpectra.pdf

Plus a follow-up to the above:

Spectral Rendering with the Bounded MESE and sRGB Data

001-005.pdf

[kram1032]

I don’t suppose it’s still possible to look into maybe implementing this?

Continuous multiple importance sampling

https://dl.acm.org/doi/abs/10.1145/3386569.3392436

cmis.pdf

One of the proposed applications is an improvement on Hero Wavelengths specifically for very spiky spectra.

And at least based on the video they provide as supplemental material, it looks like it wouldn’t even be that difficult to implement. It seems to amount to stochastically selecting multiple wavelengths rather than doing equally spaced ones around a single stochastically picked hero wavelength, and doing the appropriate weighting. I don’t actually know how difficult this would be though

[kram1032]

I just tried out the most recent build on graphicall and as long as I don’t activate CUDA it’s fine but with CUDA it eventually runs into this:

Is that a Cycles-X side bug or somehow related to spectral rendering?

The first few samples work just fine but then it just breaks. It seems unable to recover from this in any way other than restarting blender.

(Windows 10, GPU: NVidia GeForce GTX 1660 Super, CPU Intel Core i5-8400, I set it to use both devices)

[Eary]

@kram1032

I just tried out the most recent build on graphicall and as long as I don’t activate CUDA it’s fine but with CUDA it eventually runs into this

Got the same thing. That was what I post:

@Eary

But I still see this even after disabling Adaptive Sampling:

Just cannot get it to work

And recently I saw this in the GitHub:

Not sure whether this was the fix, the build is not updated yet so I cannot confirm…

[smilebags]

Sorry I haven’t had a chance to reply to these messages. @kram1032 while I can’t see anything in the stack trace which is spectral specific, it could very well be because of one of the changes the branch has. Hopefully it’ll be resolved upstream, if the issue is coming from Cycles-X, otherwise we (probably @pembem22) will look into it.

[pembem22]

That was an issue from cycles-x and it has been fixed. I’ve uploaded a new build, can you check if it works now?

[Eary]

no still the same

And regular Cycles X renders fine now, just the spectral branch that still doesn’t let me use GPU

EDIT: BTW just tested the new CleanAux with Prefiltering OIDN in Spectral mode, the new fast mode for prefiltering does not seem to work well with color noise, Accurate mode works so much better in spectral mode.

[lorenzoangeli]

Sorry if it has already been posted.
guess this fits here:

2021_spectral_imaging.pdf

[smilebags]

I haven’t actually read that, though it looks like a great resource, thanks for posting!

[smilebags]

I have now read up to chapter 4 of that paper and can say it is definitely the most comprehensive summary of all of the spectral upsampling methods I am aware of. It also covers a lot of other related topics in an approachable-as-possible manner; it’s a truly priceless resource for anyone who wants to learn about the topic.

[smilebags]

Reading the last chapter in that linked PDF, I came across a quote which made me smile. Although I’m sure they worked hard to ensure this was the case, I think this is a suitable reason to start the transition to a more spectral-aware workflow.

Finally, given that the overhead of rendering spectrally is essentially a rounding error
in the sort of scenes we render, why not use the most accurate colour representation
available?

This is referring to Manuka, Weta Digital’s in-house spectral renderer. The last line in the PDF then says:

The overhead of uplifting from RGB to spectral is negligible compared to shading and
light transport in a typical production scene, so we firmly believe the right question isn’t
“why should we go spectral?”, but “why not?”

Great read. Thanks again @lorenzoangeli for posting it.

[pembem22]

@Eary @kram1032 I’ve updated the build, can you check if the CUDA issue is fixed?

[Eary]

Most scenes can be rendered now, but somehow if the scene contains a portal light:

Or if your scene uses the sky texture node:

It would still throw errors to me:

[Eary]

About the greenish tint in the spectral result, I don’t know whether it is caused by some bug in wavelength important sampling or the currently not-so-perfect RGB to Spectra conversion, but I tried to match the result in the compositor with the “Difference” Blend mode on. I am not sure whether I am doing this correctly, if not, please tell me where I did it wrong.

I used two scenes in one file, one rendered with Spectral switch on and the other one off. The two scenes have nothing but pure white world background. Then I used the compositor to try matching them. I consciously chose to use divide, so that the result I get can be directly apply to the white level under the color management section. So I got this result:

I tried my best to match them, you can see the output is not pure black, but I think it is close enough. The result is [0.9614, 1.02, 1]. So for now, if I want to get rid of the spectral greenish tint, a temporary workaround is to either do this in the Color Management panel:

Or do it in the compositor, so that you can turn the “use nodes” on and off in a per render slot basis for comparison:

Again, I don’t know whether I did this correctly, I might be completely wrong.

Here is a test scene:
First the unadjusted Spectral result:

Then the RGB result:

Then the adjusted Spectral result:

I think the adjusted one gets pretty close now.

[smilebags]

Multiplying the channels isn’t quite the correct operation to be doing here but for now it’s probably close enough. If it reduces the difference in simple scenes, it’s useful as an interim method of comparing the two.