Is there a general BLI function for feathering/blurring a mask?
Not in BLI.
The sequencer has a glow effect which might be similar, see: SEQ_TYPE_GLOW
.
Implementations of inpaint/blur/glow tend to depend on use cases (image buffer / compositor, color handling etc), so not sure this can be a generic utility function.