Blend-O-Matic Motion Tracking & ML Video Labeling

Contributing to Blender
, ,
1

Hey there,

I’m working on a project to seamlessly integrate Blender’s motion tracking with ML video annotation.

Overview:
Harnessing tracking markers as primary annotation points, we aim for dynamic annotations that adjust with tracked objects, optimizing the video annotation process for ML.

Technical Approach:

  • Tracking Markers: Direct utilization of MovieClipTrackingTrack and MovieClipTrackingMarker.
  • Custom Properties: Embedding annotation metadata directly on markers.
  • Visualization: Real-time annotation visualization tools in the Movie Clip Editor.
  • Data Management: Systems for I/O of tracking and annotation data, targeting Pascal VOC format compatibility.

Aspirations:

  • ML Model Comparison: Load and juxtapose overlays from different ML training sessions.
  • ML Output Refinement: In-Blender tools for editing and refining ML model outputs.
  • Interactive Annotation: Exploit motion tracking for on-the-fly annotation adjustments.
  • Collaborative Annotation: Features to bolster team-based annotation workflows.
  • Object & Pose Integration: Seamlessly import and refine object detection and pose detection results.
  • Segment Anything: integrating automatic object segmentation and pytorch with Blender.
  • Pie Menus: Intuitive pie menus for rapid label management, editing, and selection.
  • Streamlined UI: A user interface tailored for efficiency, minimizing clicks and maximizing productivity.
  • Extensibility: A modular design to allow future enhancements and integration with other ML tools.

Seeking Feedback On:

  • Challenges: Potential pitfalls or challenges with this approach?
  • Overlap: Know of any Blender projects or tools that might intersect with this?
  • Technical Recommendations: Pointers on APIs, data structures, or best practices, especially around motion tracking?

Your insights would be invaluable. Thanks for your time – Blend on!

-Don

1 Like