I can’t really find any information on this when searching, so I figured this is probably a decent place to ask.
I need to create my own custom data overlay for my add-on.
What is the “Best Practice” approach? since there is no data on it?
So far, I’m just gpu batching what I need. this creates obvious issues like having to manually determine backface culling etc. for something that blender can already do WITH a data overlay (vertex normal).
Is it possible to hook up directly into the existing functions, and force a re-draw with a different length/color per vertex?
Perhaps a better question… where could I find the functions that generate the overlays in the source? (so as to maybe pull and push a change?)
Additionally. since i’m not even sure what I’m looking for… how do I get batch_for_shader LINES to not always draw above the object? there should/must be an easy way to do this…
Quick update on this.
I decided to force a redraw to happen only on right click. The lag is still fairly bad even with a proper vertex buffer and using mathutils to update. anything above 150,000 verts causes a slight lag for the re-draw process.
Currently the best practice for rendering overlays is to change the overlay engine. But this is something that is done in C and won’t help you when making the overlay part of an add-on. For reference the overlay engine can be found in /source/blender/draw/engines/overlay.
The batches for objects aren’t available outside the draw manager (/source/blender/draw) One of the reasons is that we still want to be able to change these batches for performance reasons and adding these to an API would reduce the freedom and flexibility in this.
Not sure what you exactly needs, but a work-around is to store the data in a custom data layer and let the workbench engine render the vertex colors.
When drawing batches always make sure that the correct depth test is enabled.
Performance is tricky when using only python. Profile and see where you can improve.
Thanks for the response and pointers.
Realistically, my issue is that there seems to be no reliable way to select/get the index of the last updated vertex.
You’d be like “what about raycast”. well, ray-cast just can’t keep up with vertex paint. Even on an optimized BVH the time it takes to “find” the vertex is more than it takes for the vertex paint to already have colored in another 3 or 4 vertices…
IF something like a “last updated vertex index” variable/pointer were to be made available for use (or is and it’s currently undocumented, or I didn’t dig under the correct rock for it?), then the batch change wouldn’t cost a thing since changing 1 vertex of the batch at a time is near instant.
Currently the reason performance is poor is that I’m forced to re-iterate the whole mesh to get the changes in.
On large meshes this is very problematic…
You can get an idea of what I’m doing and why here:
Having a lastest updated vertex index won’t be a solution as a single paint update can effect multiple vertices at the same time so it will already be a collection. Or a min/max vertex index.
Currently drawing and editing/painting are two separate system with as least as possible interaction between the systems. This has some performance penalties already and I would like there to be a smarter solution to identify what needs to be updated. (partial batch updating). Looking at your requirements there could be multiple systems interested in such mechanism. perhaps a pub/sub pattern would be a solution, but that needs to be build from scratch.
The obvious solution would be to just keep a specially sorted array that’s populated with a somewhat accurate (because the paint can color multiple at the same time) order of what was affected last.
The changes from a Modal operator can’t really happen as fast as the paint anyway, so being provided with a buffer in the order of “last edited” would effectively empower any python scripting to get and update the batch info…
I’d have to actually look at the source to know if that’s actually possible given the current code, but it would not be a horribly idea 
