GSoC 2022: Many Lights Sampling in Cycles X (Feedback Thread)

Thx for your reply, by emissive bodies I meant objects with a bright emission color and strength well over 1, which also contribute “lights” in cycles.

I will get a test blend file back to you this week, knowing you are finishing up your work, I think it will serve as a good test case for next iteration. (or at least a good comparison of rendering performance when trying to emulate many lights scenario).

I just thought I’d report a issue I found.

I constructed a scene like this:

  1. There is a gound plane.
  2. 0.1m above the ground plane is 2500 point lights spaced 1 meter away from each other with a power of 1 watt.
  3. The camera is placed above these looking down.

The issue I found was that lights closer to the center of this group of lights will render incorrectly with Many Lights Sampling.

Note: I used a splitting threshold of 0 to remove that as a factor as splitting seems to be causing it’s own issues

I tested with both the latest commit of the Many Lights Sampling Branch, and the latest commit with my own modifications (Which I have left in comments on ⚙ D14664 Cycles: Many Lights Sampling for Cycles X [WIP]. And here is a patch file with all my modifications: ✎ P3178 Alaska's modifications to Many Lights Sampling)

Render show casing the issue.

These renders were done at roughly 2000 samples per pixel.

Here is the .blend file for the scene.

2 Likes

I have managed to fix (or at the very least improve) on this issue.

Here’s what I found to be the issue:

  1. Part of the PDF calculations relied on the bit trail for tree traversal, but didn’t make proper use of it. So I fixed that.
  2. The bit trail was incorrect. So I fixed that.

The changes I made can be found here: ⚙ D15935 Fix bit trail construction and bit trail PDF calulations

1 Like

Great work Alaska!
I’d like to ask if the GPU fix is in already with the latest build; if it is, can you kindly share a Windows build with all these latest fixes for testing? Thanks

Currently none of these changes are in the Many Lights Sampling branch. I am just maintaining them in my own personal build.

I have posted these changes as patches for Brecht to review and commit.

As for whether or not I can share a build with GPU rendering and the fixes I’ve made? There is currently a bug I am investigating. If it’s a simple fix, then I’ll fix it and share a new build with you. If it’s a complex issue, then I’ll probably give up after maybe a few hours and create a new build for you and detail what the bug is.


Some notes about GPU rendering:
On Windows, I can only build Blender with CUDA, OptiX, and OneAPI support. I can not build with HIP.

Along with that, I have not personally tested OneAPI, so I do not know if it works.

Another issue is that with my current build environment, CUDA support will be limited to “modern GPUs” (I believe it’s Maxwell and later - So GTX 900 series and later).

Also, in my previous testing, CUDA easily experiences “illegal address” errors when using Many Lights Sampling in the viewport. To “work around” this, you need to close Blender and reopen it again.

1 Like

Here you go.

For GPU rendering support, it supports CUDA (GTX 900 series and later), OptiX, and OneAPI (untested).
There is no HIP support.

As you will have noticed, there are two builds, what’s different about them?


12-09-2022 - Many Lights Sampling - Alaska Custom:

This build takes the latest commit of the Many Lights Sampling Branch, and applies these patches to it:

Enable GPU rendering support with Many Light Sampling - D15934
Fix bit trail construction and bit trail PDF calulations - D15935
Fix an incorrect merge with master - D15936

The issue I mentioned earlier is NOT fixed with this build.

What is the issue?
Mesh lights don’t render properly in the reflections of rough materials. See: GSoC 2022: Many Lights Sampling in Cycles X (Feedback Thread) - #99 by Alaska


12-09-2022 - Many Lights Sampling - Alaska Custom with P3181:

This is the exact same as 12-09-2022 - Many Lights Sampling - Alaska Custom, but with this patch applied to it:

https://developer.blender.org/P3181

It fixes the issue I described above with mesh lights not having correct reflections, however I do not know if it introduces other bugs. Hence I have kept this as a separate build.


I should note, the changes I have been making have been focused on fixing bugs for the case of Splitting threshold = 0.0. There are still issues when the splitting threshold is not 0 that I personally have not looked at.

6 Likes

For anyone wondering, all the changes have been commited to the Many Lights Sampling branch.
However, there was an issue with one of them being commited, so if you pull the latest Many Lights Sampling branch, you will have issues.

I have submitted a patch to fix it here: ⚙ D15951 Fix PDF calculations for Many Lights Sampling


Edit:

Here is a Windows build of the latest Many Lights Sampling branch with D15951 applied to it:

7 Likes

Hey Alaska, I noticed you’ve submitted some more patches for Many Lights Sampling, can you provide a Windows build for us to test?

Here you go:

Here’s some useful information about the build:

I also wanted to document an issue I discovered.
At this current point in time, the Progressive Multi-Jitter sampling pattern has some issues, and they can manifest as visible artifacts when using Many Lights Sampling in certain scenes. Please use Sobol-Burley where possible until this issue is fixed.

I have reported the issue here and someone will investigate it soon: ⚓ T101356 New PMJ does not converge to the same result as Sobol-Burley

6 Likes

EDIT: Some the the structure for Many Lights Sampling has changed in a drastic manner since I wrote this comment. Some of this information is obsolete.

The Blender foundation has employeed a new developer to help out with the development of Many Lights Sampling.

So I wanted to collate a list of bugs/issues/limitations with Many Lights Sampling here so the information is more organised and readily avaliable. Both for the new developer, and for users who are using the Many Lights Sampling branch.

Note: Whenever I include image comparisons in this post, I am comparing renders of equal sample counts.

Here is the list of bugs/issues/limitations:


Incorrect Spot Light Importance - RESOLVED BY SPLITTING

The issue:
When a spot light has a large radius, Many Lights Sampling will over sample the spot light (and under sample other lights) in areas outside the cone of influence of the spot light. This results in an increasse in noise around the spot light.

This issue is resolved by using splitting.

What’s causing it?
My assumption is that the bounding box representation for the spot light is less than ideal for a spotlight with a large radius. This results in the spot light still being assigned “high importance” in areas outside the cone of influence of the spotlight.


Incorrect Area light Importance - RESOLVED BY SPLITTING

The issue:
Area lights with a large size, and low “spread” parameter have the same issue as spot lights. Many Lights Sampling will dedicate samples to it outside the region of influence the area light has. This results in an increasse in noise around the area light.

This issue is resolved by using splitting.

What’s causing it?
My assumption is that the bounding box representation for the area light is less than ideal for a these types of area lights. This results in the area lights still being assigned “high importance” in areas outside the region of influence of the area light.


Incorrect light importance when using textures - MOSTLY RESOLVED BY SPLITTING

The issue:
When a light uses textures to dictate it’s output, the light can end up being treated as if it has a greater or lesser importance than it actually does in certain areas. This can lead to over sampling or under sampling of the given light and increase noise.

This issue is mostly resolved by using splitting.

What’s causing it?
In Many Lights Sampling we use a number of approximations. It seems that one of these approximation is assuming that lights are uniform in brightness, which isn’t true for textured lights and that’s what’s causing this issue.


Incorrect light importance with variable material visibility

The issue:
In Cycles we can adjust the material visibility of objects and lights. For example, we can make a light visible to reflections, but not diffuse materials.

At the moment Many Lights Sampling doesn’t handle this properly and as such still dedicates samples to lights that have no benefit for the current material.

Splitting can help to aleviate the issue, but it doesn’t bring the quality up to the same level as rendering without Many Lights Sampling.

What’s causing it?
Many Lights Sampling currently doesn’t handle the case where an object is not visible to certain materials properly.


Poor balancing between distant lights and local lights

The issue:
In Many Lights Sampling, the balancing between distant lights and local lights can sometimes be quite poor, resulting in over sampling or under sampling of certain lights and increasing noise.

What’s causing it?
When picking a light from the light tree, we start by deciding whether or not we want to pick a local light or a distant light. This is done by comparing the importance of an approximation of all distant lights and the importance of the root node of the light tree and deciding between the two based on that.

The issue with this is that the root node of the light tree is a poor representation of the contribution of local lights to localized areas of the scene. And as such, it can negatively impact the accuracy of the decision on whether or not we should sample a distant light or a local light.

The proposed solution I showed in the image above is this patch here: ⚙ D16179 Improve distant light balance by comparing distant and light tree samples
We pick a local light and a distant light then compare the two and make a decision on which to sample based on that.


Increased noise in interior scenes when using bright exterior lights - CAN SOMETIME BE REDUCED BY SPLITTING

The issue:
When rendering a interior scene with some interior lights and some bright exterior lights there can be an increase in noise compared to rendering without Many Lights Sampling.

Splitting can help aleviate this somewhat when using local lights. However it does not help in the case of using a combination of local and distant lights.

What’s causing it?
Cycles doesn’t know if there is a wall between the current shading point and the light it picked until the light has been sampled. As a result of this lack of knowledge about the occlusion of lights, Many Lights Sampling may dedicate higher numbers of samples to lights behind walls without know it’s a bad idea, and as a result it can increase noise.


Incorrect brightnesses on glass in certain scenes - NOW FIXED

EDIT: This issue has been fixed.

The issue:
Reflections of lights on glass materials in certain scenes/situations are incorrect (usually too dark).

Splitting can hide this issue in some scenes. But ideally the issue should be fixed at the source rather than relying on splitting.

A .blend file:
⇪ Glass rendering issue - MLS.blend

What’s causing it?
In Many Lights Sampling we disregard lights that are behind a surface if the surface is opaque. However, if the surface is transmissive (like glass), then we don’t apply this optimization. This is not what’s causing the issue.

The issue is that there is a mis-match between the definition of what a “transmissive surface” is in the functions light_tree_sample() and light_tree_pdf() in /intern/cycles/kernel/light/light_tree.h (the same applies for the respective distant light fucntions in the same file).

In light_tree_sample() a transmissive surface is a surface with a shader that contains a transmissive component. This is correct.

In light_tree_pdf() a transmissive surface is a surface that we previously transmitted through. This is incorrect. But why is it incorrect?

If a ray hits a glass material, it has two paths it can follow. A glossy path (bouncing off the surface). Or a transmissive path (passing through the surface). light_tree_pdf() should consider both of these paths as “we’ve just interacted with a transmissive surface” for light_tree_pdf() to be correct. But at the moment, only the transmissive path is considered for this and as a result the light_tree_pdf() calculations are wrong for reflections off of glass in certain scenes.

A comment about this can be found in the file /intern/cycles/kernel/light/light_tree.h

  /* TODO: this is wrong, needs to also be true if the previous bounce had a transmission BSDF but
   * sampled a reflection BSDF. */
  const bool has_transmission = (path_flag & PATH_RAY_TRANSMIT);

Weird patterns when there is a combination of lights in front and behind a opaque surface - NOW FIXED

EDIT: This issue has been fixed.

The issue:
In some scenes with some lights in front and some lights behind a surface, you may notice weird patterns of increased and decreased noise.

Splitting resolves this issue in some scenes.

What’s causing it?
In Many Lights Sampling we disregard lights that are behind a surface when the surface is opaque.

The issue is the calculation to figure out if a light is behind or in front of a surface is inaccurate when there are wide bounding boxes around the lights behind the surface, and it can end up being wrong a lot of the time resulting in lights behind the surface being sampled and thus increasing noise.

⚙ D16192 Use corners of bbox to figure out which side of surface the box is on atempts to fix this issue.


Lack of Many Lights Sampling improvements when rendering volumetric materials - NOW FIXED

Edit: Volumetric support has been added in a recent revision of the Many Lights Sampling branch. It’s not included in the build I Blender I have linked below.

The issue:
Volumetric materials do not benefit from Many Lights Sampling.

What’s causing it?
Support for volumetric materials to benefit from Many Lights Sampling has not been added yet.


Blender crashing when enabling Many Lights Samping in certain scenes - NOW REDUCED/FIXED

EDIT: The test scene I included in this description no longer crashes with the latest iteration of the branch. As such, this issue has been reduced. I say “reduced” because there may be more scenes that crash that I personally haven’t found yet.

The issue:
In certain scenes, Blender will crash when enabling Many Lights Sampling. Here is a file I personally experience crashes with.

⇪ Multiple Spot Light Crashing Issue - MLS.blend

Note about this scene:
It is a collection of 15 spot lights with a radius of 0. If the radius is greater than 0, I can not reproduce the crash. If the number of spot lights is less than 8, I also could not reproduce the issue.

What’s causing it?
I tried debugging it, but I couldn’t really figure it out. Something during the light tree construction goes wrong and Blender crashes.


Noise discontinuities when using splitting and low sample counts - A LIMITATION OF THE TECHNIQUE

The issue:
When rendering with Many Lights Sampling, and using a combination of splitting and a low sample count, you may notice weird circles in different areas of the scene. And the noise inside and outside of the ciricle is different. This is known as a “noise discontinuity” and according to authors of the original paper Many Lights Sampling is based on (Importance Sampling of Many Lights with Adaptive Tree Splitting), noise discontinuities are an expected side effect of splitting and usually disappears at high sample counts or when using something like adaptive sampling to target a certain noise threshold. It’s just that the effect is more noticable in Cycles due to how we do splitting.

What’s causing it?
When using splitting, there are regions of the scene that traverse down the light tree in different ways and sample different sets of lights. This results in differences in noise in different regions and is most noticable at low sample counts. This effect is pronounced due to the fact there are hard boundraies between different regions.


"Kernel assert" and "illegal address" issues - NOW REDUCED/FIXED

EDIT: Some values have been adjusted and the kernel asserts and illegal address issues have been reduced. I say “reduced” as most of these issues stem from precision issues with how computers, and it can be hard to figure out exactly how these precision issues may manifest in a wide range of different scenes, making it hard to write code to account for these issues without testing various scenes.

The issue:
When rendering with a GPU backend (like CUDA or OptiX), you may experience errors like “illegal address in…”. Or if you’re rendering with the CPU in a debug build of Blender, you may encounter “kernel_assert”.

I personally encounter some of these issues with a variant of the “Classroom” scene. Depending on whether or not I’m testing on Linux, or macOS, with CPU rendering in a debugging build, or GPU rendering in a release build, I can personally come across some of these kernel assert or illegal address issues in different places.

And you will likely encounter these issues too. If you are a community memeber and experience issues like these, sharing the scene you have issues with may be helpful in assisting development and fixing these issues.

⇪ Classroom.zip

What’s causing it?
Throughout the Cycles and Many Lights Sampling code, there are lines of code like this. kernel_assert(variable < 0);

These are primarily used for debugging. If the program reachs that point in the code, and the condition inside kernel_assert() is false, then the program will stop there and you can look at what happened with a debugger. Usually these are added to the code to catch situations where something hasn’t worked out as expected so developers can look into it and fix them.
Typically kernel_asserts are deactivated in release builds of Blender (what users are using), but from my testing, they continue to show up with GPU render as “illegal address” error messages. I’m not sure if that’s an issue on my part or not.


Rendering artifacts with PMJ

The issue:
When rendering with the PMJ sampler and complex scenes, artifacts can occur. See:
⚓ T101356 New PMJ does not converge to the same result as Sobol-Burley

I personally recommend that you use Sobol-Burley for complex scenes.

What’s causing it?
This topic is still a little bit confusing for me. There is discussion on the mater in the bug report for the issue:
⚓ T101356 New PMJ does not converge to the same result as Sobol-Burley

Brecht: With many light sampling, we’re currently using a single 2D pattern for sampling lights. First we use one dimension of the pattern for selecting the light, then a subset of the 2D pattern for sampling a position on a chosen light. With many lights, that’s really stretching the sample out a lot.

It’s probably better to use a 1D pattern for sampling the light tree, and 2D pattern for sampling the light here. Ideally this would be a 3D pattern to ensure each chosen light still gets a stratified pattern. We could always use a 3D Sobol-Burley pattern just for light sampling regardless of the setting, and leave PMJ for the others.

There’s some alternative solutions possible, increasing divisions along one dimension only, or splitting the pattern along both dimensions some now. But probably not worth spending time on.

22 Likes

Thanks for the detailed report. We’ll sort through the issues and update the task + add regression test files.

In general if splitting solves the problem, it’s not necessarily something we have to fix. This algorithm is meant to be used with splitting to get good importance estimates. It can indicate places where we can improve the estimates, but not necessarily in a way that’s practical.

“Incorrect brightnesses on glass in certain scenes” should be fixed now.

7 Likes

Thank you for the rundown. It is no wonder then why Many-Lights Sampling has not yet landed in master even though there were multiple GSoC projects dedicated to it, considering just how complex it is to make everything work in a way that is unbiased and without issues.

3 Likes

Blender crashing when enabling Many Lights Samping in certain scenes

Should be fixed.

“Kernel assert” and “illegal address” issues

I fixed the case I found in the classroom blend, not sure if there are others.

5 Likes

I have updating the Many Lights Sampling builds for community members to test. Keep in mind that these builds will be outdated as soon as something new is committed to the Many Lights Sampling branch.

I should also note that this build is just the Many Lights Sampling branch as of commit rB039eba790675.

Windows x86

Has GPU support for CUDA and OptiX on GTX 900 series and later. Sorry AMD and Intel users.
2022-10-21 - Many Lights Sampling - Windows x86.zip - Google Drive

Linux x86

Has GPU support for CUDA and OptiX on GTX 900 series and later, and oneAPI support for Intel ARC (I haven’t tested the oneAPI support). Sorry AMD users.
2022-10-21 - Many Lights Sampling - Linux x86.tar.xz - Google Drive

macOS ARM64

Comes with Metal GPU rendering support.
2022-10-21 - Many Lights Sampling - macOS ARM64.zip - Google Drive

4 Likes

@Alaska thank you for sharing these builds!

Will many light sampling move to ‘supported’ feature set instead of ‘experimental’ in the future??

I am not a Blender or Cycles developer. I can not give a reliable answer. However my expectation is that the Blender/Cycles developers want to make Many Lights Sampling feature complete and move it out of the “experimental feature set” at some point in the future. How far away is that future? I don’t know.

Edit: Brecht has shifted Many Lights Sampling into the “supported feature set” of Cycles with rB0610cdcb86f4.

2 Likes

Edit: The build bot has been setup to build the Many Lights Sampling branch. Downloads can be found here with the name soc-2022-many-lights-sampling: Blender Builds - blender.org

I should note that at this current point in time the Linux build of the Many Lights Sampling branch is failing to build and as such can not be found on the site linked above. The reason it’s failing is due to an issue when building the GPU kernels for Vega GPUs.

5 Likes

This video might be interesting to some people. It is a brief visual explanation of the light sampling techniques. The first half of the video outlines the “problem”, the second shows some solutions:

3 Likes

This branch is incredible! Anyone know whether IES lights are coming? Right now IES Textured lights don’t render.

EDIT: After cranking up the brightness etc. I noticed that IES kind of work. But they react totally different than when not using the light tree… Looks like I can’t connect them to the Strength input of an emission.

From my testing, IES lights do render. However, the importance calculation for IES lights is incorrect since Many Lights Sampling currently doesn’t take textures into proper consideration when calculating a light’s importances. (IES lights are textured lights). This issue can be partially resolved by using splitting, but a decision has been made to move away from splitting due to it’s impact on the reliability of calculating the PDF of lights for forward path tracing.

Presumably if/when textured lights are taken into proper consideration, this issue with IES lights will be fixed.

3 Likes