This workshop took place at the Blender headquarters in Amsterdam. Attendees:
Aras Pranckevičius
Dalai Felinto
Falk David
Sergey Sharybin
These are the meeting notes we took during the workshop. There is also a blog post summarising the workshop.
Performance
Primary goal is to make preview smoother, secondary is making rendering (esp “daily playblast”) faster.
Ffmpeg context creation/destruction is very costly
Richard has a WIP PR that tries to pre-create and keep them around in the background. Might need help ensuring it is robust.
SIMD
“How to do SIMD across Blender?” is an unsolved topic, needs more discussions with relevant teams (Compositor, Geometry Nodes).
There are “easy but does not take advantage of AVX2/AVX512” ways, i.e. building things based on float4, and mapping that to RGBA of a pixel within VSE.
For VSE specifically, could experiment with OpenImageIO simd.h, given that OIIO is a hard dependency nowadays.
A more scalable way is mapping one SIMD lane to a work item, similar to ISPC. Dragging in the whole ISPC as a hard dependency is not a trivial decision though.
Libraries like Enoki or CppSPMD might be worth looking into.
Targeting multiple SIMD variants based on actual CPU capabilities is very hard to do at scale, without resorting to “build a whole binary several times” (like what Cycles does).
Maybe Intel oneAPI / SYCL would work at some point, if it gets more platform coverage?
GPU
Makes a lot of sense to move “pixel processing” to be GPU (with possible CPU fallback for GPU-less cases e.g. render farms).
Ffmpeg video decoding / encoding might benefit from GPU acceleration too.
Need to figure out how exactly that is done on the ffmpeg API level.
Will need ffmpeg rebuild, today none of HW accelerated parts are included into Blender’s build.
Possible quality reduction might be an issue (is it only for encoding? Or decoding too?)
Preview
Preview window should render/evaluate everything at preview resolution, not full movie resolution!
This ties into proxies and scene strips as well.
Proxies
Whole point of proxies is to make source media that is fast to seek and fast to decode/load. They do this by potentially taking up a lot of storage space.
Current design of explicit proxy resolution and needing to pick the correct resolution for preview does not make much sense. From the user’s POV, they should have control of how much quality vs. storage space they use. If proxies need to internally build several resolutions (like mipmaps), that should just happen automatically.
Proxies should be evaluated at preview resolution, together with everything else.
Proxies stored next to the original movie file make no sense for scenarios where the movie file is on a network drive! Should perhaps be a local cache folder.
Cache
If the whole pixel processing stack is faster (via SIMD/GPU, and evaluated at preview resolution), and video decoding has no stalls (via ffmpeg caching, GPU decoding etc.), then maybe “final frame” caches are not needed at all?
By the same thought, maybe “prefetch” should be only concerned with prefetching/preloading source media files, not rendering the whole frame?
Similar to trying to pre-render scene strips: if they are fast enough, there feels little point in pre-rendering them. If they are slow, then trying to pre-render them in the background will slow everything down anyway.
How about complex post-processing configurations in the future with things like Kuwahara?
Feels like “preprocessed” and “composite” caches are not needed (they are also off by default) – remove.
Per-strip cache control: feels like it should be removed too.
There might still be a need for a very small/simple “only during one frame” cache of intermediate steps. Imagine being on one frame, and dragging some slider that controls parameters of an effect; might want to cache everything below the effect for more interactive tweaking.
Generally cache should be robust enough to be able to trust it, e.g. not always throw it out during rendering!
Scene Strips
Similar to above, for Preview evaluate them at preview resolution!
Today they use “Render” settings (subd levels, AA samples etc.), for preview should default to “Viewport” settings.
Re-use GPU context instead of re-creating it for each frame.
Re-use evaluated depsgraph somehow.
Re-use other GPU resources (Draw Manager, Eevee)?
Eventually do incremental render updates, similar to viewport
Timeline
Aras working on making thumbnail cache faster, then thumbnail drawing faster. Then making waveform / fcurve overlays to have lower overhead. Not much to discuss, just needs to get done.
Larger Picture
Global goal: story telling from the sequencer. Ideally: be able to do layout work, storyboarding, grease pencil directly from the sequencer space. Without the need to have a secondary window; perhaps even no dedicated 3d viewport; no extra clicks. Just timeline, and preview with powerful tools.
One of the large issues is that Blender can display one scene across the whole application (the “current scene”), but VSE data lives in a scene! Which means you can not edit the VSE timeline, and edit some scene (used by a scene strip) at the same time. BCON22 talk is one example of how people approach solving this.
Falk presented several possible solutions / approaches on how to address the above.
Do nothing and just use two Blender instances for editing. Pros: possible today. Cons: everything else.
Have a “scene override” within the VSE timeline window, similar to BCON22 talk. Not hard to do, is a bit of a hack and might feel weird.
Move the VSE timeline to live outside of the scene, into an object called “Movie” or “Sequence”. This has far-reaching implications and is a large project. This new top-level object would probably have to have render settings moved out of the scene and into it.
Have “active strip” as a more prominent feature and move all strip properties to be part of the regular Properties window.
Various cons and pros were discussed during the workshop, as well as an attempt to make a quick prototype. Unfortunately, there was not enough time to finish the prototype, or to make a final decision on the design. This is something the team will come back to.
Various other bits
Nodes
Generally it feels like VSE should “use nodes somehow”, details handwavy.
Most of the nodes you’d want would be very similar to Compositor nodes, maybe with just several that are different due to different color space expectations (normally in Compositor you’re in linear space, whereas in VSE you’re in display space).
As usual, naming is hard. Compositing nodes? Grading nodes? etc.
EXR images vs proxies
Today proxies are JPG images, they cannot represent EXRs correctly. Need to add capability for EXR proxies. Figure out optimal EXR settings for those wrt space and performance (e.g. possibly never use 32 bit floats). Primary issue at the studio is that EXRs are on network, and they are huge.
Improved Text strips
4.2 added text outline and configurable shadow
Richard starts to work on on-preview text editing, plus better text alignment tools
Compliant media
Need indication / confirmation when adding a movie strip that does not match project frame rate. Discuss/design with UI module.
Strongly typed channels
Flexibility of “anything can go into a channel” is not often used in actual production
If channels were strongly typed, it would be easier to do UI parts like master volume or metering for audio channels, etc.
Media Manager / Bins
A wanted feature, majority of work is probably within actual UI/UX design
“Render Animation” vs “Sequence Render Animation” operators
Mostly a historical artifact, from the days when regular “render” could not use GPU
Today in many cases “sequence render animation” is slower!
Need to figure out whether there are some actual useful behavior differences between them, and fold them into one operator.
GSoC 24
Connected Strips are in! Merged during the workshop.
John Kiril intends to work on improving “channels” (active channel and/or logic where strips are placed) after GSoC ends.
OpenTimelineIO
Seems to be interest from some groups of people
Priority unknown, perhaps low?
Maybe it should be an extension, made by said interested people?
“Interlaced” video option
At least the tooltip is confusing (says “removes fields”)
Produces garbage on some (not all!) videos when toggled on
Need to double check whether it actually does anything useful on actual interlaced videos these days
Old tasks / designs on Gitea: there are many designs/tasks that are almost or partially done, or outdated. Someone should have a pass over them and close irrelevant ones, or update the still-relevant ones to reflect current reality.
Great workshop ! thank you for the detailed report !
My feedback:
Very curious the workaround to use two Blender instances to edit the VSE timeline, while editing the same scene ( in another Blender instance ). I need to test it.
Probably in long term move the VSE timeline outside the scene would be the best approach for the “story telling sequencer” and work with realtime ( EEVEE ) scenes.
Working with OpenEXR I try to avoid jpeg proxies so yes to EXR proxies !
A better integration with VSE and the now realtime GPU Compositor nodes would be crucial:
Example for Color grading / Color compositor there are two main workflow: using the integrated but limited VSE strip modifier or use the Compositor nodes ( adavanced but complicated workflow ).
Very interesting also the Intel oneAPI / SYCL for performance improvent.
please do not not forget also the Audio ! Very pleased to see the subframe audio editing for 4.3.
Basic audio meters would be important for example.
Anyway i see a great vision and useful improvements to enhance the VSE !
thank you !
I understand complexity of this project, but can’t emphasize how much this would improve everything. As someone who uses VSE mostly alongside 3D scenes, there have been so many times were I wished this was true and Sequence was not tied to scene. Besides making storyboarding easy (and honestly it will be easiest and most feature-heavy storyboarding tool I’ve seen), it will lift multiple other limitations, as well as open door for new possibilites: like having multiple sequences, like other video-editing softwares do, like import-export for sequences and possible compatibility with other software.
About nodes, design that was discussed in chat multiple times: having VSE modifiers as compositor node groups (similar to GN modifiers) is the best it can possibly be from users point of view. Having yet another node editor is scary.
To provide some examples where I wanted Sequence separate from the scene:
I make videos that showcase animations, and put text and image overlays on top (like here https://www.youtube.com/shorts/shT4oFKWqN0). Currently I have to either put texts and images as 3D objects, align them and all that, or put those overlays on rendered video. I do both currently.
Ideal behavior would be to put 3D scene as a strip in sequencer, and simply put text strips and image strips on top of it. But now its impossible because you can’t put current scene on timeline. I can make separate scene just for render (and I have done that), but workflow gets so uncomfortable that I preferred putting them over rendered video.
Having the ability to overlay 2D items on current scenes render opens up possibilities to use VSE in more "“after-effecty” fashion and combine 2D-3D workflows much better.
I put reference videos in sequencer for animation, as well as audio I’m animating to. And when I want to render it renders sequence strips (reference videos) and not the 3D scene. Its SO annoying. I go in there and hide/mute strips > Render > 3D scene still not there. I have to disable Sequencer from Post Processing and render audio separately and match them, or delete carefully placed references. If I was able to simply put current scene above references in sequencer and render it with audio life would be prettier.
It would also allow me to put reference and 3D scene side-by-side, instead of having to do that after the render.
I can talk about more scenarios for days. In total it would be much better change. Import-export & append of Sequence blocks alone is a priceless feature. It would also make Scene data-block more useful and widely adopted, instead of being something only pre-2.8 people are aware of (mostly).
@aras_p Question about sequencer as suggestion, why not having two types of sequencer one reside inside scene for some purpose to benefit from its position and its result can be added as new type of " Strip " ( or an existing type or rendered video or proxy ?.. ) inside the Sequencer that reside outside the Scene … this need more deep thinking and naming …
As a freelance motion graphics artist, I usually contain my entire project within a single Blender file, relying heavily on the scenes system and Video Sequence Editor (VSE) for the final compilation. The VSE plays a crucial role in this workflow.
In the Motion Design field, smooth transitions between scenes are often essential.
It’s all about iteratively refining the timing, pacing, and previewing of transitions to assess their quality.
Without the VSE, I would be forced to re-render my animations multiple times.
I tend to use Solid View mode, which works well with relatively fast processing. One challenge, however, is that Solid View doesn’t provide an alpha channel. But, I have found a workaround for that.
While yes, it is possible, I believe that the primary goal for the VSE shouldn’t be to become a tool that fully competes or replaces professional NLEs in the sense of “pure video editing”.
I make an emphasis on primary because, even though its primary goal shouldn’t be that one, the task of assembling a final cut of a project (a film/short film, a series episode, a demo reel, a social media video, etc.) should still be one of the goals of the VSE.
Blender has the tools to do the whole pipeline of animated productions (except for sound production), it makes complete sense being able to assemble the final product within it too.
New Top Level Object seems clearly less confusing than a Scene Override.
Able to perpetuate workflows that people are used to, and at same time, bringing this ability to tweak 3D Scenes, in UI automatically modified by context of use of this object.
UI will have to make it clear for user what is manipulated.
In BCON22talk, indication is set in VSE header ; but there is also tweaking in sidebar of 3D View, while Topbar is referring to a scene name, we should not care about.
If editors are adapted to use of this top level object ; user will feel a lot more confident about changes made.
well, it doesnt really need to compete, even though it already kinda does (blender is almost always in the list of open source video editing apps) hahaha.
but if the video editing capabilities inside vse becomes decent enough, it certainly would attract a lot of video editors. i mean, the video editing community is huge. a slice of that would be a big win for blender. u know, more users more chances of devfund survival
grease pencil is doing a decent job attracting 2d artists/illustrators to blender. vse has the potential to do a similar job.
the nle side of vse should be just as important as everything else planned here imho.
while its nice to make the vse be a powerful extension of the 3d workflow, it wont bring new users.
so for longevity’s sake of the blender project as a whole, i hope they keep those things in mind.
we gotta play all the angles man
I think Blender should also add sound mixing, to attract producers and let them contribute to dev fund. The ground has already been laid with that GSoC project.
Text editor can be turned into a screenwriting tool also, to benefit from that WGA union money, and speaking of VSE it should be able to assemble presentations, there is a strong large community of students out there.
But most of all its a shame that the Spreadsheet editor is that weak and abandoned. Hell, can you imagine the money flow in dev fund if middle managers start contributing?
Not going to lie, I wouldn’t mind if the Text editor supported Markdown rendering (even as an addon), lol.
Slightly back on topic, I think it’d be great if every place where there is a Text-type object (text object in 3d view, String Node in GN, Text strip in VSE) it would support bringing in a text datablock from the text editor. It makes a lot of sense to keep blocks of reusable text around in a project.
A concern I have with moving VSE data out of scenes is how well coordinated the effort would be with other efforts to set up some kind of higher-level, project-scope, settings and tools. It’s a solid idea, I would absolutely love to see more decoupling on this front, but it’s also something large enough to merit its own set of workshops. I’d hate to see a situation where a stopgap solution turns into an obstacle for later iterations.
On OTIO, the python option is not very stable, whereas the option in C seems much more solid and it has paved the way for implementations in ex. DaVinci Resolve and Premiere, and might be a relatively simple thing to implement in Blender.
Thanks everyone for the comments! Good suggestions. But just to keep expectations in check: while it would be nice to do everything that is needed, realistically right now amount of people working on VSE sums up to… about one (i.e. pretty much two people, both part time).
VSE vs compositor, nodes, etc. – yes, as mentioned, that is generally wanted. No concrete design in how exactly that would look just yet.
VSE data outside of the scene, or some other ways to edit sequence timeline while being able to tweak some other scene at the same time – yes, that is part of the whole “larger picture” section.
Hardware acceleration (NVENC etc) – yup, under GPU section.
VSE “primary goal” – sure, while primary goal is “tools for making stories”, there is nothing that prevents you from just stiching several movie clips in VSE. No one wants to take that away or make it complicated.
This is so great! Awesome big scope breakdown and insights!
I particularly love the removing of caches and complexity which is of current dubious usefulness.
Congratulations on the merge of ConnectedStrips
As a geeky note, it’s very exciting to see a good grip on the pixel processing pipeline too.
