Getting around invalidation of objects after user preforms an undo

I had a simiar issue and the workaroud I found was to add view layer update inside redo handler

    @persistent
    def redo_update_handler_post(scene):
        # fix crash in redo
        bpy.context.view_layer.update()

For UNDO crashing, adding an undo check point with

bpy.ops.ed.undo_push()

fixes the problem in some cases
More info here Addon Operators and Undo support