Cycles OSL Camera - Feedback

Hello everyone,

Over the last few weeks I’ve been working on Cycles support for writing custom cameras using OSL.
This can be used for e.g.:

  • Lens distortion
  • Realistic lens simulation
  • Panorama mappings
  • Custom camera projections
  • many more things, OSL is a powerful programming language after all

It should be pointed out that this is independent from OSL shading (and the associated performance hit), you can use the custom camera scripts without enabling that.

While there still are some limits/TODOs (see below), it’s already usable for initial testing, so I’d be happy to receive feedback on what can be improved to make this as useful and powerful as possible.

Current limits/TODOs

  • Camera scripts are only supported on CPU and OptiX currently
  • Ray differentials are broken, so e.g. the wireframe node will not work properly
  • Inverse mapping is broken, so e.g. the Vector pass will not work properly
  • Many of the builtin camera controls (e.g. sensor size, aperture, focus distance) are not yet exposed to the script, so you need to add your own parameters for those at the moment

How to test

  • Download build from Blender Builds - blender.org
  • Use CPU or OptiX
  • Set Camera to Panorama->Script mode
  • Select either an external .osl file or an internal Text datablock (like you would for the Script node)
  • Set parameters (if your script uses any)
  • Render

Here is a quick example of how to implement a perspective camera to get you started:

shader camera(float focal_length = 90.0,
              output point position = 0.0,
              output vector direction = 0.0,
              output color throughput = 1.0)
{
  vector sensor_size;
  getattribute("cam:sensor_size", sensor_size);

  point Pcam = camera_shader_raster_position() - point(0.5);
  Pcam *= sensor_size / focal_length;
  direction = normalize(vector(Pcam.x, Pcam.y, 1.0));
}

Essentially, the script’s job is to turn a sensor position into an outgoing ray.
The sensor position (from 0 to 1) is available in camera_shader_raster_position(). If you need random numbers (e.g. for aperture sampling), you can find two uniform 0-1 variables in camera_shader_random_sample().
The shader is expected to output a position (named position) and direction (named direction) for the outgoing ray, as well as a transmission intensity (named throughput). If you set throughput to zero, the ray is skipped.
The script can have arbitrary input parameters, which will be displayed in the camera properties. Note that OSL parameter metadata is supported to set UI name, description, slider limits etc. for them.

46 Likes

Pretty cool! Is it possible to sample camera position/direction itself? or this is purely for sensor mapping? With camera data it would be ideal way to perform some screen-space segmentation into different parts of scene without tons of tedious setups

Awesome work you’re doing!

I have been testing the blender-4.5.0-alpha+main-PR129495.e5b8a11b3f70-windows.amd64-version and I have some notes:

When you rendering with lenses it is natural that some pixels become occluded. This is an issue with some AOV’s:

It seems like some AOV’s are sampled by the first sample, like Depth and Position. The Depth pass for example; if the first sample happens to be occluded (T = 0.0), the depth value will be registered as infinity, making the whole pass quite useless for compositing.

Passes that takes the average sample value is also affected by the discarded rays, like Normal, Mist and the Alpha channel. The Mist pass for example will be darkened towards the circle of illumination making it quite unusable for creating mist when the optical vignetting is strong.

Is it possible for the AOV’s to ignore the discarded rays and only account for the rays that hits the scene? The occluding is of course a natural effect when tracing the lens, but for artistic purposes having usable AOV’s like the Depth, Position, Mist etc. is quite important.

When resetting a custom parameter to default by hovering over the parameter and pressing the backspace key, it always sets it to 0 instead of following the default script value. For example; float MyParm = 3.0; should have 3.0 as the default value, not 0.0.

Also, just to have it written somewhere, I said in the meeting we had that it would be nice if the Script Parameters did not trigger the shader to be recompiled when values are edited.

You have also mentioned that there will be plans to support an aperture texture, how will textures be implemented? Can we have more than one texture, like the “type texture (string filename, float s, float t, …params…)” described in the OSL documentation? This would be really handy for custom distortion maps etc.

Some technical issues:

Update:
Changing the “bpy.data.cameras[“Camera”].custom_shader” will compile the shader, no need to execute the button.

The error message is not present in the latest build. Ignore the text under.

What is the correct way to force the Update Camera Script button in python?
This is how I do it now:

def reload_osl_shader():
    context = bpy.context
    for window in context.window_manager.windows:
        screen = window.screen
        for area in screen.areas:
            with context.temp_override(window=window, area=area):
                area.type = 'PROPERTIES'
                area.spaces[0].context = "DATA"
                bpy.ops.cycles.camera_script_update()

This feels a bit hacky and it requires the camera to be selected. It would be great to be able to execute a function that takes in the camera object instead. Like this for example:

bpy.data.cameras["Camera"].cycles.script.update()

Trying to set the internal camera script will work but it gives an error, same error for both lines of code:

bpy.data.objects["Camera"].data.cycles.script_mode = "INTERNAL"
bpy.data.objects["Camera"].data.cycles.script = bpy.data.texts["osl_camera_script"]

Error:

AttributeError: 'Context' object has no attribute 'camera'
File "E:\blender-4.5.0-alpha+main-PR129495.e5b8a11b3f70-windows.amd64-release\4.5\scripts\addons_core\cycles\properties.py", line 375, in update_camera_script
Traceback (most recent call last):
  File "E:\blender-4.5.0-alpha+main-PR129495.e5b8a11b3f70-windows.amd64-release\4.5\scripts\addons_core\cycles\properties.py", line 381, in update_camera_script
    osl.update_camera_script(context.camera, report)
                             ^^^^^^^^^^^^^^
AttributeError: 'Context' object has no attribute 'camera'
File "E:\blender-4.5.0-alpha+main-PR129495.e5b8a11b3f70-windows.amd64-release\4.5\scripts\addons_core\cycles\properties.py", line 375, in update_camera_script
1 Like

OK, I quickly tried the OSL camera in current main. Nice work!!

The 3 OSL scripts coming with it (in the Blender folder in /4.5/scripts/templates_osl/)

basic_camera.osl
advanced_camera.osl
cubemap_camera.osl

are working fine. The one lens in the initial PR gives me a black output.

I also tried an OSL script from an Otoy forum (Fisheye OSL lens - Page 2 - OTOY Forums) which compiles fine but also gives me complete blackness.
Any ideas?

2 Likes

I followed everything reported, creating an internal text block, copied the shader and connected to the camera but it doesn’t work, everything is black to me. Is there anyone who has the same problem?

I’m on windows 10, I used both cpu (i7 11700k) and gpu (rtx 3060), I updated dirver but nothing.

blender 4.5.0 alpha PR129495

I had to use Cpu/choose Optix as the render device. Also T(transmittance/throughput) was changed from float to color.
Try this 35mm example copy & pate it in the txt editor and rename to camera.osl

shader perspective (
    float fov = 54.4322,    //Field of View in degrees -- 35mm                
    output vector dir = 0.0, 
    output color T = 1.0 )

{
    float w = tan(0.5 * fov * (M_PI / 180.0)); //width
    float h = w / (16.0 / 9.0);                //height

    dir = normalize(vector(2.0 * (P.x - 0.5) * w, 2.0 * (P.y - 0.5) * h,  1.0));
}

The perspective example here just renders all black on the CPU, and throws this OptiX error on the GPU:

The camera example from the pull request seems to work on the CPU, but throws this CUDA error (render device is still set to OptiX) on the GPU:
image
and after doing anything the error changes to this:
image
then Cycles will be completely bricked until Blender is restarted (doing anything like switching to CPU going into and back from solid view does not fix it).

I am not sure if the provided shader is broken or if the feature is, so for now I will only make an issue on the GPU problems.

1 Like

Got te same issue :frowning: Though the cubemap camera is working as intended so it might be the perspective cameras themselves that are broken.

This issue has been reported on the bug tracker. #138185 - Cycles OptiX OSL camera errors with basic camera example - blender - Blender Projects

Just a reminder to everyone. If a issue is obviously a bug, then it’s best to report it on the Blender bug tracker rather than here in the feedback thread.

The easiest way to report a bug is:

  • Open Blender
  • Select Help -> Report a bug
  • Then fill out the form with all the relevant information.
1 Like

hello @lukasstockner97 is it possible to get printf functionality for GPU?

It would be nice if a exact replica of the standard blender camera would be provided in the OSL camera examples. The “Advanced Camera” seems to be doing that, only with a couple of extra features.

Maybe just add support for depth of field the “Basic Camera” example, that seems to replicate the standard camera. Or add a “Standard Camera” example that will replicate the standard camera exact:

shader camera(float focal_length = 90.0,
              output point position = 0.0,
              output vector direction = 0.0,
              output color throughput = 1.0)
{
  vector sensor_size;
  getattribute("cam:sensor_size", sensor_size);

  point Pcam = camera_shader_raster_position() - point(0.5);
  Pcam *= sensor_size / focal_length;
  direction = normalize(vector(Pcam.x, Pcam.y, 1.0));
  
  float focal_distance;
  getattribute("cam:focal_distance", focal_distance);
  getattribute("cam:aperture_position", position);

  point Pfocus = direction * focal_distance / direction.z;
  direction = normalize(Pfocus - position);
}

Also, I guess this has been discussed before; The “Focal Length” parameter is hidden when the camera Type is set to Custom, but it’s still used to determine the viewport field of view. This also applies to the other camera types except from the Orthographic type.

I don’t think its a good idea to just hide it, maybe bring that parameter back? Maybe it would fit into the “Camera” panel instead, to make a distinction between parameters that changes the OSL rays directly and those who don’t.

Shift X/Y and the Clip Start/End parameter seems to have an influence on the OSL shader without it having to implement it in the code, so it make sense to have that in the Lens panel. While the Sensor Height for example needs to be implemented for it to have an influence on the render. So for me it makes sense to move the Focal Length parameter over there instead.

It would also be great to be able to reference it with the getattribute function like this:

getattribute("cam:focal_length", focal_length);
4 Likes

Any chance we could get script to generate lens profiles?

7 Likes

Hi,
Looking forward to trying this out !
Has a timeline been discussed for the listed “Future improvements” ?
From a usability standpoint, providing a library of presets and supporting other GPU vendors by generating rays on CPU seem like low hanging fruits that would make the feature much more accessible.

4 Likes

I was very excited to try this but I guess I won’t have that luxury. Ran into the same issue as this. It won’t work on pretty much all of my blend files it looks like.

Strange, that problem should no longer be an issue since its decupled from the OSL shading.

It might also be a temporary issue with the lastest build, they mentioned it being broken in this post:

OSL camera is fixed by an upcoming library updates.

2 Likes

Oops… I misunderstood how it works and since it’s an OSL shader I thought I had to enable it. Well, I’m getting back to it and to see if I can figure out why the output is now black :sweat_smile:

1 Like

it was black for me when i used the example given in the opening post rather than the ones in the blender text editor templates

1 Like

Nice that you got it working! I see, the pos and T output parameter has changed since this tread was started, that’s probably why the old example failed.

Edit: Thought Calibrator was answering as I was responding on my phone and didn’t do a propper look at the username :sweat_smile:

1 Like

Damn, that’s rough cause with those I get an error or a crash like Alaska mentioned a while ago

hi @Alaska. I’m noticing that bump maps rendered with the OSL camera appear darker compared to the same setup using the default camera.
Could this be a bug, or is there a specific setting I might be overlooking?

(Left)OSL; (Right)Default - Bump vs Bump