It’s been over 20 years since post-production software started treating 3D scenes as just another node, with the possibility of mixing them and controlling render parameters from another node.
I think VSE should be the root, the Compositor a modified Strip, and 3D scenes objects with their own timeline.
This image shows how rendering could be controlled from the Compositor.
As nickberckley says, the audio section also needs more care.
Thanks aras_p for all the improvements added.
Feedback:
About the “pinned” sequencer timeline, I have been using code from the Spas Studios branch adapted to 4.3, using the hack with another custom fork of Blender.
The “3D Sequencer” when using the full feature set and “switching” scenes from a single timeline is… a godsend and I can’t live without it. A more performant or elegant solution I would be totally happy to work in! Working with the adapted 3D Sequencer from Spa Studio has saved me so much creative time and upped my game hands down.
Well, that’s disappointing, and quite concerning as well. It seems like BF is not aware of how their userbase actually use their software. Very interesting indeed.
If the sequencer content edit was moved out of the scene data structure becoming a data-block, and the individual strips also could become data-blocks, maybe the current node systems only would need sequencer-data-block input and output nodes (single strips as data-blocks could almost work out of the box)?
Imo, it’s better to integrate the VSE in the existing editors, instead of making new editors (ex. a node editor) in a VSE friendly version. And paradoxically, this is properly best done, by moving the sequencer out of the scene data structure. Ex. if the sequencer edits and individual strips were data-blocks, they could be listed, organized and imported into the Asset Browser, and trimmed here before 3-point-edit adding them to the sequence. And the same way could a strip, as data-block become a texture in the 3d space etc. Or a text/image-data block from the text/image editor could become a text/image strip.
So, another way of thinking about the VSE, is as an editor(hub/assembler) of data-blocks(potentially created or edited in all the other editors of Blender), and the edit of the sequencer itself should be a data-block, independent of the scene, so that could be used anywhere in Blender which takes a data-block as input. That would make the VSE an extremely versatile tool…
and basically become…
wait for it…
The blender of Blender!
That sounds like a beautiful dream if it becomes true.
Well, the Asset browser handling image, video and sound data-blocks including thumbnails and drag and drop into the VSE was working, until it was removed, that is.
It wasn’t removed. When the Asset Browser was first made all datablocks were allowed, but due to time constraints only a few were implemented “properly” over time (Material, Collection, Geometry Nodes, World, Pose).
You can allow all data-blocks to be assets by enabling “Extended Asset Browser” under Experimental features:
That’s fantastic to know! 
Thank you! I hope they can be properly implemented in the not-so-distant future 
So, it is still somehow floating around on the river Styx. Any plans on what coast it is going to end up at? For using the Asset Browser to organize VSE footage, afair only thumbnails for movie strips, were missing (and of course an option to either import A/V directly into the Asset Browser, or mark the VSE imported footage as data-blocks(which I did an add-on for)). I actually did a patch for movie strips, but it was undecided whether to commit it or not(so that is long dead and gone):
D’n d of movies worked too:
But it still serves as a good example of something as simple as having VSE strip content marked as data-block will make the VSE much more integrated in Blender.
i just want vse to use the gpu. glad that’s finally getting talked about more.
give us gpu support and we’re good. I could honestly care less about all the other features.
i just want this thing to run above 5fps. It’s not usable rn.
Im hoping you guys have the funding required to make these changes… hopefully soon.
I want Blender VSE to FULLY replace Davinci Resolve for me. For 1 reason.
Interface Customization. That and it’s just way easier to edit videos in Blender, aaaand ive been using blender for years. All we need now is GPU support for VSE to actually be usable. In my case, I’m getting 5fps sometimes 0fps in the sequencer with my core i9 CPU and 96 Gigabytes of RAM. Can’t wait for blender to update VSE with GPU support so we can actually use VSE for real. It’s a brick right now.
I think you want performance, not necessarily “gpu” (why? because there are other factors that make VSE slower than it could be, not just “lack of gpu whatever”).
It would really help have test cases where you get 5 or 1 FPS playback, just to see which parts are slow! Right now I am testing on VSE cases as used/needed by Blender Studio, but none of them are as slow as 5 fps. So yours presumably uses some different features or different settings, or something else from VSE that make it way slower. It would be good to know what is slow, i.e. have some blend+media files that roughly show your setup and are reproducibly slow.
Today VSE is roughly two people part time, so about “one person” working on it. Just to set expectations.
Ah… wasn’t aware there’s just 2 people working on this… damn. I might as well move back to davinci and just try to deal with the forced layout. or just wait a while for vse update.
I believe im getting 5fps due to my color data. It’s HDR video at 3440x1440 60fps footage. But VSE can’t seem to run it reliably.
If the vse used the gpu we’d be set rn. No more performance issues I’d imagine. As far as I know, current vse just uses a single core on my cpu… So yeah, I just want better performance to be honest, VSE is good as is rn. All it needs is decent performance.
If you filed an issue with your blend file and your footage (even if it’s a small sample of it), it would be way easier to investigate where the slowdown is.
Not necessarily, but hard to say without having actual files to look at.
That’s incorrect knowledge; VSE does use multiple CPU cores.
Global goal: story telling from the sequencer. Ideally: be able to do layout work, storyboarding, grease pencil directly from the sequencer space. Without the need to have a secondary window; perhaps even no dedicated 3d viewport; no extra clicks. Just timeline, and preview with powerful tools.
Wow, this is really impressive. I used to think that while the VSE was nice to have, it wasn’t necessarily essential, especially since its performance doesn’t match up to other editing programs.
However, my perspective has completely changed. If the VSE becomes fully integrated into Blender as a completely different storytelling tool, distinct from traditional editing software, it would be revolutionary.
I don’t know when this will happen, but you have my full support. 
how do we donate to vse workshop. I dont want to just donate to blender, I want to SPECIFICALLY donate to the people working on VSE. Because that’s the feature I care about the most.
Unless you’re barely using Resolve… Not going to happen. VSE doesn’t quite even belong in the same conversation.
Interesting talk about the Video Sequence Editor
2D/3D Hybrid Storyboarding with VISION - Maciej Gliwa - (BCON ->LA)
Anime Studios have long been exploring various methods to speed up pre-production work by using Grease Pencil and VSE in storyboards, animatics and previs (min 56).
Exactly, Resolve has superior color management, professional color grading, great media and bin management etc. VSE is awesome but I don’t see how it can catch up with Blackmagic tools anytime soon.
Because I was on vacation, I missed the workshop sadly.
I do think that most of the things that were discussed sounds great.
However, there are a few head scratchers that I would like to discuss:
If the whole pixel processing stack is faster (via SIMD/GPU, and evaluated at preview resolution), and video decoding has no stalls (via ffmpeg caching, GPU decoding etc.), then maybe “final frame” caches are not needed at all?
We should absolutely make everything as fast as possible! 
But I don’t think we can get rid of final frame caches for a few reasons:
Even if we make decoding and compositing really fast, people will be able to bring it to a crawn with enough stacked videos or with enough high definition video (for example 3 4k 60fps HDR strips on top of each other with filters applied on top)
Caching helps with slow network drives or can also help when running on under powered computers
Not needlessly wasting compute resources. With a cache we don’t need to re-evaluate the same frames all the time.
By the same thought, maybe “prefetch” should be only concerned with prefetching/preloading source media files, not rendering the whole frame?
I think we still need prefetch to render the whole frame as sometimes the actual composition of the final image is what is slow and not the reading/decoding of the source media files themselves.
Similar to trying to pre-render scene strips: if they are fast enough, there feels little point in pre-rendering them. If they are slow, then trying to pre-render them in the background will slow everything down anyway.
I don’t really understand what the argument is here?
“It is slow to render, so don’t bother pre-render them?”
To me it is a bit like saying “Cycles is slow to render, so don’t bother pre-render it to a video file. Even if viewing the resulting video file will be fast, the creation of it is slow, so just don’t do it.”.
This is the absolute best case scenario for pre-fretching and caching!
You can’t playback in real time, so instead you pre-render it (slowly) so that you can then watch it in real time later.
The whole point of pre-fretching and caching is that you spend time up front to get good performance instead of doing it in real time but with horrible performance.
There might still be a need for a very small/simple “only during one frame” cache of intermediate steps. Imagine being on one frame, and dragging some slider that controls parameters of an effect; might want to cache everything below the effect for more interactive tweaking.
This is exactly what our “preprocessed” and “composite” caches tries to solve. So It feels a bit weird that we want to remove them if we want features like this?
I do agree that we could probably limit these caches to just one frame though.
Flexibility of “anything can go into a channel” is not often used in actual production
We talked about this already in a google doc that Aras created. I’m a bit sad that it doesn’t seem like any of the feedback John Kiril Swenson and I posted there made it into the meeting about this topic. Why was this the case or was it simply ignored?
I’ll post mine and John’s comments here for clarity:
Mine:
I’ve talked to multiple users and most are either indifferent or think that it is a great feature that you can mix and not have strongly typed channels.
The reason why users think it is a great feature is that you can easily group strips together so it is easy to work on a “focused blob”. If the channels only could contain certain types, then the end user is forced into a certain way to organize their timeline.
Here in Blender they can instead organize in ways that makes sense to them.
(And of course this ongoing organization effort does sometimes conclude into channels only for audio or video etc)
John:
After Effects works like this (with un-typed channels) and it’s very successful software. I think the A/V split paradigm, where some channels are dedicated to audio and some to video et. al., only really accommodates the very specific (albeit common) use-case of having to edit simple 2-channel footage.
We could think about introducing features to help users out in these cases, like, maybe while you’re dragging strips, scrolling up increases the distance between your audio and video strips, so that if you want to stack video on video and audio on audio then you still can. Or maybe a setting to switch between “channel display types”.
But it’s very flexible to allow any channel to have any kind of strip. Confusion can be completely avoided if the UI makes it obvious which strip is which type (e.g. audio strips have a line down the middle displaying their waveform).