GSoC - ideas for the Video Sequence Editor

[tintwotin]

Some ideas for GSoC VSE projects - we should aim for small well defined ideas - so which idea would be the best suited?

https://blender.chat/channel/vse

First I want share, what I see as the top 3 of challenges for the VSE, basically I think the VSE should become the hub for realtime story development and virtual storytelling inside Blender:

The biggest all time challenge is INTEGRATION, simply because without it, the VSE go no raison d’etre. Or in other words if the VSE is directly integrated in the 3D View(In and Out), meaning you could use the 3D View for previewing the video VSE contents and control the 3D contents from the VSE(ex. camera and scene switching etc.) with the lightning preview speeds of eevee, no one would argue that during the storytelling process, you should render everything and edit it in an external NLE, at least not until you would have to assemble high quality renders, which could be argued, could be better done in ex. Davinci Resolve. With better VSE integration, comes also round-tripping of strips in the Compositor, but basically make sequences and strips available on a frame change basis from/to the other Blender Editors. In order to improve the integration a necessary step is to move sequences out of scenes, so changing the scene doesn’t change the contents of the sequencer.

The second biggest challenge is PERFORMANCE which concerns speed of playback(in 3D View, preview of video, scene strips(playbasts), multithreaded, cache optimized, GPU usage, fast image scaling, fast and invisible proxy handling/cache to disk) and quality(unbroken color-manegement, 10+ bit color, high quality image scaling etc.). This being second, since the final export quality could be handled externally(but would need a proper export option like OTIO), but it is very close to the number one, simply because the slow playback of ex. video on a first run will scare off many first time users.

The third biggest challenge is the TOOL SET & THE UI that comes with it. With the toolbar finally integrated it is now possible for proper icon based tools, but it needs serious planning to make it well designed, user friendly, UI integrated(on strip drawing/interaction) and (splitscreen)preview of ex. trim and slip/slide operations etc.

Integration:

• A better integration of the VSE with the Compositor, like roundtripping strips in the Node Editor, is one of the most requested topics in the Sequencer tagged posts on Right Click Select.
• A better integration of the 3D View with the Sequencer for ex. a storyboarding workflow, has been discussed much. There are multiple ways to address this ex. the playback performance of Scene Strips is very slow(a eevee rendered scene running at 30 fps in 3d view runs at 0.3 fps as eevee rendered Scene Strip) and/or live camera and scene switching in the 3d view from the sequencer(3d View Strips?), which would need a separation of sequences from scenes.
• Allow VSE to control frame handling of media assets elsewhere in Blender, like Movie Clip or Image Sequence inputs of texture or compositor. This allows you to time clip start/stop and offset in context with other animation.
• Develop for a seamless from storyboard/previz(or even screenplay: https://github.com/tin2tin/Blender_Screenwriter ) to final film workflow, without rendering to file until everything is finished.

Performance:

• There is a general need to improve the playback performance of the VSE, but can it be boiled down to a GSoC suggestion?

The VSE UI/toolset:

• On strip drawn operations like fade, volume rubberband, crossfade, thumbnails, and modernize the strip and handle drawing?
• Channel headers with options to mute/hide/lock(like NLA).
• Move Strip Effects into Modifiers Tab.
• Populate the newly implemented Toolbar with essential tools.
• Integrate basic image asset manipulation like scale, translate, rotate and boundary clip/mask.

Editing workflow:

• Industry standard 3 point editing with source, rec, in, out, overwrite, insert, lift, remove and channel headers to select/deselect what channels to operate in(would also need a separation of sequences from scenes(for ex. more playhead positions and sequences in the same window)).
• Auto preview of trim operations(including dual screen), strip parenting and other strip editing related operations(populate the toolbar), overhaul of the VSE API.

Assert Manager:

• A bin system based on the File Browser as Assert Manager(or is better to wait for a general Assert Manager solution to be implemented?).

I/O:

• Integration of the OTIO library which will allow the Sequencer to import and export in industry standart formats like aaf.

Proxy and ffmpeg:

• An overhaul of export, proxy and ffmpeg operations(including exposing ffmpeg to the API). Currently, export and proxies are generated single threaded(very slow) and the UI is not user friendly.

Plugins:

• Implement support of VST, LV2 and Open FX plugins?

Color handling:

• Color handling overhaul. Currently the color handling seems broken and slow.
• 10, 12 etc. bit video?

General VSE ideas: devtalk.blender.org/t/video-sequence-editor-a-list-of-ideas/10343
VSE Quick Tasks: devtalk.blender.org/t/quick-hacks-for-the-vse-a-list-of-suggestions/10341

[Davidmcsween]

Asset Manager:

• Allow local addition of metadata fields. For keywords and script integration or version control and favouriting etc.

VSE Toolset:

• Allow channel metadata for labelling and soloing etc. This could allow more flexible addon development.
• integrate basic image asset manipulation like scale, translate, rotate and boundary clip/mask.

Editing Workflow:

• Allow VSE to control frame handling of media assets elsewhere in Blender, like Movie Clip or Image Sequence inputs of texture or compositor. This allows you to time clip start/stop and offset in context with other animation.

[Bachadam]

I share an addon Yesterday on gumraod which content some feature that your talking about. Such as strip compositing and strip tracking etc…
The addon is call BACHADAM VFX SETUP.
After this expérience i undestand why it is important for blender to separate séquence editor from scene.
The sequencer must be the principal scene from where people will open 3d scene.
I publish a video showing some feature of m’y addon on tweeter today.
Thanks to take VSE improvement important.

[tintwotin]

I’ve added your image manipulation toolset suggestion. Could you write the meta data handling up as one project suggestion? As you know, currently, there is no VSE assert manager, could this metadata project still work without it?

[tintwotin]

Yes, there is so much great functionality to be unleashed if sequences are moved out of the scene data-structure. Looking forward to see your add-on.

[Andrea_Monzini]

I would like to see improved Color Management, better integration with 3D ( realtime EEVEE editing ) and with Compositor ( several nodes are not scene referred ), better cache performance ( to edit OpenEXR image sequences avoiding proxy ) , sub-frame audio editing, audio meters and LV2 audio plugins.

[tintwotin]

@Andrea_Monzini

I would like to see improved Color Management

It’s on the list.

better integration with 3D ( realtime EEVEE editing ) and with Compositor ( several nodes are not scene referred )

Also on the list - is there something you want me to add?

better cache performance ( to edit OpenEXR image sequences avoiding proxy )

That’s Richard’s area. He’s employed as a Blender dev now.

sub-frame audio editing, audio meters and LV2 audio plugins.

There is an audio project among the official ideas:
https://wiki.blender.org/wiki/GSoC/Ideas_Suggestions

[Andrea_Monzini]

@tintwotin

Also on the list - is there something you want me to add?

@tintwotin

Some ideas for GSoC VSE projects - we should aim for small well defined ideas - so which idea would be the best suited?

Maybe i misunderstood the question.
I wrote, in my opinion, what would be best suited to focus on starting from your list.
In particular about integrate the VSE in a correct scene referred workflow.

[MikeFutcher]

A proposal to link the VSE and compositing…

Upsides:
This would allow you to have multiple composites within a single edits, each using multiple clips that you can time on the VSE timeline.

Downsides:
While this would use all the same nodes as compositing, this is infact a new nodes view - thus allowing you to have several node trees within the same scene.

[MikeFutcher]

A revision - showing how you could have multiple comps in one edit.

[MikeFutcher]

A simple one…

Chroma keying within VSE. (please)

[tintwotin]

@MikeFutcher Already on the list, but strip based:

@tintwotin

A better integration of the VSE with the Compositor, like roundtripping strips in the Node Editor, is one of the most requested topics in the Sequencer tagged posts on Right Click Select.

NB. this is not a request list, but a list of projects which potentially could work as Google Summer of Code projects.

[MikeFutcher]

I apologise. My misunderstanding. With that in mind I will suggest a chroma key effect for the VSE - and sugest whoever creates it has a look at this: https://www.youtube.com/watch?v=UqGmRe53KdE

[BlenderFrenzy]

I absolutely love this idea and have wanted it for a long time! Because in order to do this currently, you would have to render out the video 3 times: First render it from the VSE, then import it into the compositor and render it there, then import that video back into the VSE and render it again. This is a huge headache and waste of space and time.

Being able to have the VSE, 3D Editor, Movie Clip Editor, and Compositor all work seamlessly together without having to render out multiple videos would be a game changer. You’d be able to edit in the VSE, see the preview result in the compositor, then go back and make more edits/changes in the VSE and see the updates in the compositor without having to render anything.

[tintwotin]

The Gsoc 2021 is open for applications the 29. january. In the light of the successful Outliner project which combined UI and new functionality, here are some elements I think could be developed into VSE Gsoc projects(first most simple > last most difficult):

1. POPULATE THE SEQUENCER PREVIEW TOOLBAR:

Add: Select, Cursor placement(pivot), Move, Rotate, Scale, Crop, Transform(multi), Text.

• The transform operations should include X, Y and number entry - like the UV Editor.
  (Consult the Transformation Tools add-on)
• Add: Insert keyframe menu(I).
• The Text option should include ex. bounding box editing, alignment, multiline text entry with carriage return.

2. POPULATE THE SEQUENCER TOOLBAR AND HEADER:

Select Tool with box select(handles?).

Trim Tool would need to be able to detect, change mouse cursor, and highlight handles to switch between trim modes, and it would also need split screen preview. It could contain the following functions, depending of mouse cursor position when click(CHRISTOPHER DOBEY):

• Roll by click drag on an adjoined cut.
• Slip(move inside) by click drag in the center of a strip.
• Slide(move position in timeline) by click drag in the center of a strip.
• Ripple by click drag on either side of a cut(or could be determined by toggle button in header).

Blade Tool. Patch is existing.

Audio Tool to draw volume levels on strips etc. (part of VSEQF: GitHub - snuq/VSEQF: VSE Quick Functions Addon For Blender. Improve the Blender video editor with new workflows and functions.) Context menu for audio operations only?

Header buttons to work with Tools.

• Ripple Edit toggle button. Patch is existing, however the functionality can be expanded.
• Snap Toggle.
• Parenting Toggle would have to be implemented from scratch, but some work was done in the Terrible Consequencer branch.
• Preview trim operations toggle - if it is too slow to be on at all times.
• Selected channels/All channels toggle(work on…) - or implement channel headers.

3. THREE POINT EDIT

Add: In and out markers.
Add: Channel Headers with insert, mute, lock, solo, name options(channel headers are already implemented in the NLA)(big enough for a gsoc project of it’s own?).

Add: View Type: Sequencer to File Browser, with import options in sidebar and import(multiple files) button.
Add: Insert, Overwrite, Append, Playhead in File Browser sidebar.
Add: Insert, Overwrite, Append as paste options.
Figure out how to a workspace with two different playheads, maybe by using a speed-up version of the clip editor as source player?)

4. CONVERT THE SEQUENCER CONTENT INTO DATA-BLOCK
(This will require a deep understanding of the data-structure of Blender, and will integrate the vse in all other Blender tools)

Convert the sequencer content into data-block, so it can be:

• Accessed from any editor(ex. as node, texture, mask, sequencer + seq preview etc.)
• Switched and overridden independantly of scene switches.
• Saved, loaded, new’ed.
• Have it’s own playhead position, range etc.

https://devtalk.blender.org/t/proposal-make-the-sequencer-contents-into-data-block-which-can-override-the-switched-to-scene-sequencer-contents/16562

[TheRedWaxPolice]

I want to be able to apply LUTs in the sequencer… hehehe…
Or is there a hidden trick to do that already?

[tintwotin]

I want to be able to apply LUTs in the sequencer… hehehe…
Or is there a hidden trick to do that already?

We can hope for a native solution when OCIO 2.0 will be implemented, until then, you can try this add-on for the image editor: GitHub - Suemura/blender_luts_importer: import&apply luts to images!