GSoC - ideas for the Video Sequence Editor

Some ideas for GSoC VSE projects - we should aim for small well defined ideas - so which idea would be the best suited?

https://blender.chat/channel/vse

First I want share, what I see as the top 3 of challenges for the VSE, basically I think the VSE should become the hub for realtime story development and virtual storytelling inside Blender:

The biggest all time challenge is INTEGRATION, simply because without it, the VSE go no raison d’etre. Or in other words if the VSE is directly integrated in the 3D View(In and Out), meaning you could use the 3D View for previewing the video VSE contents and control the 3D contents from the VSE(ex. camera and scene switching etc.) with the lightning preview speeds of eevee, no one would argue that during the storytelling process, you should render everything and edit it in an external NLE, at least not until you would have to assemble high quality renders, which could be argued, could be better done in ex. Davinci Resolve. With better VSE integration, comes also round-tripping of strips in the Compositor, but basically make sequences and strips available on a frame change basis from/to the other Blender Editors. In order to improve the integration a necessary step is to move sequences out of scenes, so changing the scene doesn’t change the contents of the sequencer.

The second biggest challenge is PERFORMANCE which concerns speed of playback(in 3D View, preview of video, scene strips(playbasts), multithreaded, cache optimized, GPU usage, fast image scaling, fast and invisible proxy handling/cache to disk) and quality(unbroken color-manegement, 10+ bit color, high quality image scaling etc.). This being second, since the final export quality could be handled externally(but would need a proper export option like OTIO), but it is very close to the number one, simply because the slow playback of ex. video on a first run will scare off many first time users.

The third biggest challenge is the TOOL SET & THE UI that comes with it. With the toolbar finally integrated it is now possible for proper icon based tools, but it needs serious planning to make it well designed, user friendly, UI integrated(on strip drawing/interaction) and (splitscreen)preview of ex. trim and slip/slide operations etc.

Integration:
The seems to be a wide support for a better integration of the VSE into the other editors:

  • A better integration of the VSE with the Compositor, like roundtripping strips in the Node Editor, is one of the most requested topics in the Sequencer tagged posts on Right Click Select.
  • A better integration of the 3D View with the Sequencer for ex. a storyboarding workflow, has been discussed much. There are multiple ways to address this ex. the playback performance of Scene Strips is very slow(a eevee rendered scene running at 30 fps in 3d view runs at 0.3 fps as eevee rendered Scene Strip) and/or live camera and scene switching in the 3d view from the sequencer(3d View Strips?), which would need a separation of sequences from scenes.
  • Allow VSE to control frame handling of media assets elsewhere in Blender, like Movie Clip or Image Sequence inputs of texture or compositor. This allows you to time clip start/stop and offset in context with other animation.
  • Develop for a seamless from storyboard/previz(or even screenplay: https://github.com/tin2tin/Blender_Screenwriter ) to final film workflow, without rendering to file until everything is finished.

Performance:

  • There is a general need to improve the playback performance of the VSE, but can it be boiled down to a GSoC suggestion?

The VSE UI/toolset:

  • On strip drawn operations like fade, volume rubberband, crossfade, thumbnails, and modernize the strip and handle drawing?
  • Channel headers with options to mute/hide/lock(like NLA).
  • Move Strip Effects into Modifiers Tab.
  • Populate the newly implemented Toolbar with essential tools.
  • Integrate basic image asset manipulation like scale, translate, rotate and boundary clip/mask.

Editing workflow:

  • Industry standard 3 point editing with source, rec, in, out, overwrite, insert, lift, remove and channel headers to select/deselect what channels to operate in(would also need a separation of sequences from scenes(for ex. more playhead positions and sequences in the same window)).
  • Auto preview of trim operations(including dual screen), strip parenting and other strip editing related operations(populate the toolbar), overhaul of the VSE API.

Assert Manager:

  • A bin system based on the File Browser as Assert Manager(or is better to wait for a general Assert Manager solution to be implemented?).

I/O:

  • Integration of the OTIO library which will allow the Sequencer to import and export in industry standart formats like aaf.

Proxy and ffmpeg:

  • An overhaul of export, proxy and ffmpeg operations(including exposing ffmpeg to the API). Currently, export and proxies are generated single threaded(very slow) and the UI is not user friendly.

Plugins:

  • Implement support of VST, LV2 and Open FX plugins?

Color handling:

  • Color handling overhaul. Currently the color handling seems broken and slow.
  • 10, 12 etc. bit video?

General VSE ideas: devtalk.blender.org/t/video-sequence-editor-a-list-of-ideas/10343
VSE Quick Tasks: devtalk.blender.org/t/quick-hacks-for-the-vse-a-list-of-suggestions/10341

1 Like

Asset Manager:

  • Allow local addition of metadata fields. For keywords and script integration or version control and favouriting etc.

VSE Toolset:

  • Allow channel metadata for labelling and soloing etc. This could allow more flexible addon development.
  • integrate basic image asset manipulation like scale, translate, rotate and boundary clip/mask.

Editing Workflow:

  • Allow VSE to control frame handling of media assets elsewhere in Blender, like Movie Clip or Image Sequence inputs of texture or compositor. This allows you to time clip start/stop and offset in context with other animation.
1 Like

I share an addon Yesterday on gumraod which content some feature that your talking about. Such as strip compositing and strip tracking etc…
The addon is call BACHADAM VFX SETUP.
After this expérience i undestand why it is important for blender to separate séquence editor from scene.
The sequencer must be the principal scene from where people will open 3d scene.
I publish a video showing some feature of m’y addon on tweeter today.
Thanks to take VSE improvement important.

1 Like

I’ve added your image manipulation toolset suggestion. Could you write the meta data handling up as one project suggestion? As you know, currently, there is no VSE assert manager, could this metadata project still work without it?

Yes, there is so much great functionality to be unleashed if sequences are moved out of the scene data-structure. Looking forward to see your add-on.

I would like to see improved Color Management, better integration with 3D ( realtime EEVEE editing ) and with Compositor ( several nodes are not scene referred ), better cache performance ( to edit OpenEXR image sequences avoiding proxy ) , sub-frame audio editing, audio meters and LV2 audio plugins.

1 Like

It’s on the list.

Also on the list - is there something you want me to add?

That’s Richard’s area. He’s employed as a Blender dev now.

There is an audio project among the official ideas:
https://wiki.blender.org/wiki/GSoC/Ideas_Suggestions

1 Like

Maybe i misunderstood the question.
I wrote, in my opinion, what would be best suited to focus on starting from your list.
In particular about integrate the VSE in a correct scene referred workflow.

1 Like

A proposal to link the VSE and compositing…

Upsides:
This would allow you to have multiple composites within a single edits, each using multiple clips that you can time on the VSE timeline.

Downsides:
While this would use all the same nodes as compositing, this is infact a new nodes view - thus allowing you to have several node trees within the same scene.

2 Likes

A revision - showing how you could have multiple comps in one edit.

1 Like

A simple one…

Chroma keying within VSE. (please)

@MikeFutcher Already on the list, but strip based:

NB. this is not a request list, but a list of projects which potentially could work as Google Summer of Code projects.

I apologise. My misunderstanding. With that in mind I will suggest a chroma key effect for the VSE - and sugest whoever creates it has a look at this: https://www.youtube.com/watch?v=UqGmRe53KdE

I absolutely love this idea and have wanted it for a long time! Because in order to do this currently, you would have to render out the video 3 times: First render it from the VSE, then import it into the compositor and render it there, then import that video back into the VSE and render it again. This is a huge headache and waste of space and time.

Being able to have the VSE, 3D Editor, Movie Clip Editor, and Compositor all work seamlessly together without having to render out multiple videos would be a game changer. You’d be able to edit in the VSE, see the preview result in the compositor, then go back and make more edits/changes in the VSE and see the updates in the compositor without having to render anything.

1 Like