Very slow deletion of linked libraries in blender

I hope this is the right spot for this question, because i’m kinda stuck. I have a scene of about 600mb with lots of objects in it, when I link this file it takes approx 3 min. Which is to be expected, but when I want to delete this library it takes 10min+.

So far I have tested this in blender 2.91 and also 2.83.
Linking using python and through the ui yield in same results
turning off undo didnt have any effect.
I also tested lighter files. I didnt take that long but always a serious multiple of loading the file.

Unfortunaly I’m not able to share the file, but does anyone know what blender might be doing on deleting a linked library?