Brush cursor implementation

Hi!

I’m trying to implement a brush cursor for sculpting and vertex painting with normal and vertex preview, similar to this:

As a first step, I’m trying to modify the function sculpt_stroke_get_location from sculpt.c to make it also return the stroke normal. It seems that I can obtain the normal from sculpt_pbvh_calc_area_normal. Is there any better way to do it?

Thank you.

4 Likes

That seems to be the right function to use, it makes sense for the preview to use the same code to display the normal as the actual sculpting.

The main concern with this feature is performance I guess, computing the normal for every mouse move can be slow with a big brush over a high poly mesh. So perhaps in the end the code can be optimized in some way if it turns out to be needed (caching average normal per node, sampling the normal from a random subset of vertices, …).

Try sampling the normal from a set evenly spaced vertices starting from the center

Finding a set of evenly spaced vertices is quite expensive itself. The idea with a random subset is that you would avoid looping over all the vertices and only pick 1000 or so at random, with the expectation that it is a close approximation to the average normal over all vertices.

Regardless, it’s best to get the basics working, test the performance, and then see if any optimization is actually needed.

Maybe check Retopoflow 2.0 github repository, they have made such cursor.

Thank you for your response.

When I try to get the nodes under the cursor with sculpt_pbvh_gather_generic() it seems it is assuming that ss->cache is initialized, and it isn’t. Where should I initialize it before drawing the cursor without needing to make a stroke?

It may be best to not create ss->cache, and instead make computing the area normal work without a stroke cache as well. The view normal can be stored in SculptThreadedTaskData, and the (ss->cache->original) test can be replaced by (ss->cache && ss->cache->original).

Ok, I think it is working and I’m getting the correct normals. I’m going to start the drawing code. Should I use the gpu_immediate functions or is there a better drawing code in the 2.8 branch? If so, how can I draw a circle in 3D space?

It’s fine to use immediate mode, the existing paint cursor drawing does as well. Probably the tricky part is getting the transform right, since paint cursor has 2D view and projection matrices set up by default. They are in rv3d->winmat and rv3d->viewmat, you’ll either need to set them to OpenGL before drawing or transform the coordinates with them yourself.

Everything is working now. There is just one problem, how can I set the value for the radius that is used to sample the vertices? If it is set too low it works fine on high poly meshes, but it doesn’t work on low poly meshes because it doesn’t sample anything. If it is set too high it goes crazy and it doesn’t calculate the normal correctly.

com-video-to-gif

4 Likes

I’m not sure why that problem would happen. I would expect the code to give the same normal for drawing the brush and doing the actual sculpting, so you’ll need to figure out what exactly is happening differently in those situations?

Sometimes you have a brush with a small size that it is not affecting any vertices despite being over the mesh. When you try to sculpt with it nothing happens, but if the cursor it’s not displayed on mouse hover it feels broken.

Ok, in that case I think setting the radius is not a great solution. You can set it some arbitrary amount higher but it still will not fix all the cases without breaking others.

Maybe the most correct solution in that case would be to get the normal from the face that is below the cursor. sculpt_stroke_get_location uses raycast to compute the position on the surface, this could optionally return the normal as well.

A simpler solution could be to remember the last valid normal, not sure if that would give some unexpected behavior or not.

Now I’m using the face normal as a fallback and it works as expected. I’m going to fix the code style and I’ll upload the patch as soon as I can.
Thank you for your help.
com-video-to-gif(1)

14 Likes

I believe it’s a little difficult to handle it, I had to change the code and still a couldn’t handle the cursor well

1 Like