I think it might be possible to improve performance if you did the internal calculations as vertex colors or bmesh custom data and then copied them to weight paint just in time. That way, you’d only have to do the slow conversion when you need it, instead of every frame. You could keep the interactivity in vertex-paint mode, which would convert to weight paint (you would only have to convert once instead of twice, maybe). I’m not sure if this is possible or not for your case. But either way, here’s a stack exchange page with code for copying vertex colors to weight paint (look at the second answer for my code, which can do the exchange back-and-forth. I need to turn this into an addon at some point). https://blender.stackexchange.com/questions/15172/turn-weight-paintvertex-groups-into-vertex-paint
As for using C, maybe using Cython could make it easier? Cython is a Pyhon-to-C conversion language (more-or-less), but it relies on Python which can vary from system to system or from Blender version. So apparently that comes with difficulties in distribution. Take a look at this devtalk page, which also discusses threading some: Noob question - Does Cython brings similar performance to pure compiled C code?
It looks like the excellent performance that Animation Nodes provides is at least partially because of C code replacing pure Python, but this requires some trickery at the setup phase which I think is in the init.py of that addon (I looked a long time ago, out of curiosity).
I wish I could help more with a ‘conventional’ solution, but I don’t know if there is one.