Plugin Hot-Reload by Cleaning sys.modules?

Hello

When writing a multi-module plugin, it can be tricky to implement a hot reload function that will automatically reload modules when we enabled/disable the plugin

Of course we can always shut down blender and restart but well… that’s annoying

Initially it seems that the “standard method” is by using importlibs.reload()
But this method can cause issues as if the modules we want to reload contains other modules, those modules are not reloaded recursively, so depending on the exact code you can wind up in a rather broken state

Which was my case these last 24hs, i was stuck on an import reload similar to the link above
i had constant value error where sub properties straight up refuse tor register after an importlib.reload()
(yes i know, i should prolly have a better global/local space and module organisation to avoid such i guess? but it’s quite a hard fix when un-experienced)

But why bother to do all this? Why do we need them at first place? well that’s because plugins modules are still loaded in the python interpreter after disabling operation.
if we inspect sys.modules we can still see traces of our plugin that we just disabled.
And why so? That’s the root of our hot-reload problem isn’t it?

Therefore i just corrected this behavior in my main __init__.unregister() function that clean up all traces of my plugin in sys.modules

In other words, whenever you use an import statement, the first thing Python does is check if the dictionary that sys.modules references already has an entry for that module and proceed with the next step (binding names in the current namespace) without loading the module first. If you delete entries from sys.modules , then Python won’t find the already-loaded module and loads again.

In the example below, this hotreload implementation is working fine
Why is this not a more popular solution? is there something I’m missing perhaps?

def cleanse_modules():
    """search for your plugin modules in blender python sys.modules and remove them"""

    import sys

    all_modules = sys.modules 
    all_modules = dict(sorted(all_modules.items(),key= lambda x:x[0])) #sort them
   
    for k,v in all_modules.items():
        if k.startswith(__name__):
            del sys.modules[k]

    return None 

def register():
    for m in main_modules:
        m.register()
    return

def unregister():
    for m in reversed(main_modules):
        m.unregister()
    cleanse_modules()
    return
4 Likes

I also ran into issues with the “standard” way to reload addons in the past. My vscode extension (which I don’t have the time to support anymore unfortunately…) uses the approach you describe here. It seems to work well in practice, at least I did not have any issues with it.

See blender_vscode/addon_update.py at master · JacquesLucke/blender_vscode · GitHub.

3 Likes

Thanks for the info

If this technique is not causing issues at first glance
then we should probably cleanse sys.modules within PREFERENCES_OT_addon_disable

One thing I find is it’s easier if your addon is all in one file. I have seen addons split across, say, half a dozen files with only a few hundred lines in each, and I feel this is excessive. I see no big deal with a single addon file being, say, a few thousand lines in size – it’s still faster to load than multiple split files that total to the same size

And this way, Blender can reload the addon more reliably without quitting.

1 Like

Simplified version of cleanse module function, works great so far tested on multiple projects

def cleanse_modules():
    """search for your plugin modules in blender python sys.modules and remove them"""

    for module_name in sorted(sys.modules.keys()):
        if module_name.startswith(__name__):
            del sys.modules[module_name]

Carreful, modules in sys might not be in the correct order, it might create dependencies issues when deleting them, i remember i had issues… that’s why i sorted them in the more “complex” function

1 Like

Ah right :man_facepalming:, I have to edit it

Related insights (Re)Importing in python - don't touch sys.modules

1 Like

Thanks for that link.

The moral of the story is, there is no fully reliable way to dynamically reload a module in Python. importlib.reload() is the least bad way, since that updates the module object in place, but

  • names which have disappeared from the module will not be deleted.
  • references of the form from module import name will not have those name references updated.

Seems like the first point is a bug that could be fixed, but the second one is a pretty fundamental consequence of the design of Python itself: every name is a variable, and every form of name definition (including import statements) is just another kind of variable assignment.

1 Like

Right, something fundamental to Python itself

I think maybe we can think about this from another direction
and prevent the issue from happening in first place

that is we ditch the import statement and load the file ourselves
can help: https://stackoverflow.com/a/67692/8094047

I still have to try this later

Question: why the cleansing is not done upon unreg automatically by blender?

that is we ditch the import statement and load the file ourselves

IMO, that’s like re-inventing the wheel
cleansing is much cleaner

1 Like

My caveman solution: Restart Blender and recover the session.
I put this in an operator and bound it to F5, so I can restart Blender quickly with one button press.

Of course you lose all addon state that’s not saved in properties, but for me it achieves the goal to continue from where I was with updated addon code. E.g. reproduce a bug, fix it, press F5 to load the new code, test again to confirm the fix is working.

You can find the code here: https://github.com/LuxCoreRender/BlendLuxCore/blob/d435660c798451c0de3f023d07b0721ddbeccdf5/operators/debug.py#L23

class LUXCORE_OT_debug_restart(bpy.types.Operator):
    bl_idname = "luxcore.debug_restart"
    bl_label = "LuxCore Debug Restart"
    bl_description = "Restart Blender and recover session"

    def execute(self, context):
        blender_exe = bpy.app.binary_path
        import subprocess
        subprocess.Popen([blender_exe, "-con", "--python-expr", "import bpy; bpy.ops.wm.recover_last_session()"])
        bpy.ops.wm.quit_blender()
        return {"FINISHED"}
1 Like

right, cleansing is much simpler

interesting solution :ok_hand: