Plugin Hot-Reload by Cleaning sys.modules?

Archive Python API
1

Hello

When writing a multi-module plugin, it can be tricky to implement a hot reload function that will automatically reload modules when we enabled/disable the plugin

Of course we can always shut down blender and restart but well… that’s annoying

Initially it seems that the “standard method” is by using importlibs.reload()
But this method can cause issues as if the modules we want to reload contains other modules, those modules are not reloaded recursively, so depending on the exact code you can wind up in a rather broken state

Which was my case these last 24hs, i was stuck on an import reload similar to the link above
i had constant value error where sub properties straight up refuse tor register after an importlib.reload()
(yes i know, i should prolly have a better global/local space and module organisation to avoid such i guess? but it’s quite a hard fix when un-experienced)

But why bother to do all this? Why do we need them at first place? well that’s because plugins modules are still loaded in the python interpreter after disabling operation.
if we inspect sys.modules we can still see traces of our plugin that we just disabled.
And why so? That’s the root of our hot-reload problem isn’t it?

Therefore i just corrected this behavior in my main __init__.unregister() function that clean up all traces of my plugin in sys.modules

In other words, whenever you use an import statement, the first thing Python does is check if the dictionary that sys.modules references already has an entry for that module and proceed with the next step (binding names in the current namespace) without loading the module first. If you delete entries from sys.modules , then Python won’t find the already-loaded module and loads again.

In the example below, this hotreload implementation is working fine
Why is this not a more popular solution? is there something I’m missing perhaps?

def cleanse_modules():
    """search for your plugin modules in blender python sys.modules and remove them"""

    import sys

    all_modules = sys.modules 
    all_modules = dict(sorted(all_modules.items(),key= lambda x:x[0])) #sort them
   
    for k,v in all_modules.items():
        if k.startswith(__name__):
            del sys.modules[k]

    return None 

def register():
    for m in main_modules:
        m.register()
    return

def unregister():
    for m in reversed(main_modules):
        m.unregister()
    cleanse_modules()
    return
5 Likes
2

I also ran into issues with the “standard” way to reload addons in the past. My vscode extension (which I don’t have the time to support anymore unfortunately…) uses the approach you describe here. It seems to work well in practice, at least I did not have any issues with it.

See blender_vscode/addon_update.py at master · JacquesLucke/blender_vscode · GitHub.

3 Likes
3

Thanks for the info

If this technique is not causing issues at first glance
then we should probably cleanse sys.modules within PREFERENCES_OT_addon_disable

4

One thing I find is it’s easier if your addon is all in one file. I have seen addons split across, say, half a dozen files with only a few hundred lines in each, and I feel this is excessive. I see no big deal with a single addon file being, say, a few thousand lines in size – it’s still faster to load than multiple split files that total to the same size

And this way, Blender can reload the addon more reliably without quitting.

1 Like
5

Simplified version of cleanse module function, works great so far tested on multiple projects

def cleanse_modules():
    """search for your plugin modules in blender python sys.modules and remove them"""

    for module_name in sorted(sys.modules.keys()):
        if module_name.startswith(__name__):
            del sys.modules[module_name]
6

Carreful, modules in sys might not be in the correct order, it might create dependencies issues when deleting them, i remember i had issues… that’s why i sorted them in the more “complex” function

1 Like
7

Ah right :man_facepalming:, I have to edit it

8

Related insights (Re)Importing in python - don't touch sys.modules

1 Like
9

Thanks for that link.

The moral of the story is, there is no fully reliable way to dynamically reload a module in Python. importlib.reload() is the least bad way, since that updates the module object in place, but

  • names which have disappeared from the module will not be deleted.
  • references of the form from module import name will not have those name references updated.

Seems like the first point is a bug that could be fixed, but the second one is a pretty fundamental consequence of the design of Python itself: every name is a variable, and every form of name definition (including import statements) is just another kind of variable assignment.

1 Like
10

Right, something fundamental to Python itself

11

I think maybe we can think about this from another direction
and prevent the issue from happening in first place

that is we ditch the import statement and load the file ourselves
can help: https://stackoverflow.com/a/67692/8094047

I still have to try this later

12

Question: why the cleansing is not done upon unreg automatically by blender?

that is we ditch the import statement and load the file ourselves

IMO, that’s like re-inventing the wheel
cleansing is much cleaner

1 Like
13

My caveman solution: Restart Blender and recover the session.
I put this in an operator and bound it to F5, so I can restart Blender quickly with one button press.

Of course you lose all addon state that’s not saved in properties, but for me it achieves the goal to continue from where I was with updated addon code. E.g. reproduce a bug, fix it, press F5 to load the new code, test again to confirm the fix is working.

You can find the code here: https://github.com/LuxCoreRender/BlendLuxCore/blob/d435660c798451c0de3f023d07b0721ddbeccdf5/operators/debug.py#L23

class LUXCORE_OT_debug_restart(bpy.types.Operator):
    bl_idname = "luxcore.debug_restart"
    bl_label = "LuxCore Debug Restart"
    bl_description = "Restart Blender and recover session"

    def execute(self, context):
        blender_exe = bpy.app.binary_path
        import subprocess
        subprocess.Popen([blender_exe, "-con", "--python-expr", "import bpy; bpy.ops.wm.recover_last_session()"])
        bpy.ops.wm.quit_blender()
        return {"FINISHED"}
3 Likes
14

right, cleansing is much simpler

15

interesting solution :ok_hand:

16

I have to admit I was wrong,
messing with sys.modules is not a good idea,
it can cause crashes and makes blender test suite fail (if you get your plugin added to Blender),
a better solution might be to use importlib.reload

17

This require to reload modules one by one
too much work depending on how large your project is IMO

it can cause crashes and makes blender test suite fail (if you get your plugin added to Blender),

Got ±9 thousand users that have been using a plugin with this hot reload implementation and never got any crash report :thinking: did you had issues?

18

I have been using sys modules my self for a long time too,
the problem is that messing with sys.modules is basically undefined behavior,
and it fails the blender test suite, there are so many things wrong about it,

I agree with this answer now Plugin Hot-Reload by Cleaning sys.modules? - #9 by ldo

we can automate the import lib reload just like we do a for loop for modifying sys modules

19

also a bit more info, it seems the “correct” way to modify sys.modules is already provided by

bpy.utils.register_submodule_factory

    # Snippet
    def unregister():
        from sys import modules
        for mod in reversed(submodules):
            mod.unregister()
            name = mod.__name__
            delattr(module, name.partition(".")[2])
            del modules[name]
        submodules.clear()

    return register, unregister
20

Hmmm
and how do you make sure that the modules are reloaded in the correct order?

see the initial post

But this method can cause issues as if the modules we want to reload contains other modules, those modules are not reloaded recursively, so depending on the exact code you can wind up in a rather broken state

1 Like