Proposal: Using compositor nodes on VSE strips

Thanks for proposal.
I would like to implement such flexible system, but this definitely needs to evaluated bit more in detail. Here is my as short as possible overview and my opinions.

Personally I like your thinking of node groups as interchangeable blocks similar to way how GN handles this.

Thing I don’t quite like is handling of input and output. Having nodes that processes channel without any relation to particular strip is quite weak design, that would hinder interchangeability. GN can handle this well by exposing node tree inputs in modifier panel, I don’t quite see why not use this approach in VSE to provide particular inputs that can be then moved around without affecting final image. Using custom resolution idea does seem to be unnecessary, images should be processed as they are. To operate in custom framerate seems very odd. Node tree should do just image processing not think about time.

I think important point is, that while nodes from compositor looks like they could be used for image processing needs of VSE, I wouldn’t mix compositor node groups and VSE specific one. These editors serve quite different purpose and they should be kept focused on this purpose. So VSE should have own node editor/view, its node trees should not be usable for compositing and there could be VSE specific nodes and 3D specific compositing nodes should be excluded.

What’s not mentioned here is handling of inputs based in time. For example currently effects are meant to use strips that do overlap, in 2.79 or perhaps before they were marked as invalid if this wasn’t the case. While you can have time remapping/offset nodes to access strip content outside of “host strip” boundary, this is not practical way to normally do things, because otherwise, why would you even need strips? You could then use image sequence nodes and apply time offsets until you get them lined up in time. But again this is impractical and VSE would suit this situation better. I don’t quite have satisfying solution for this problem, but I think this is most important one to address to quite good satisfaction, otherwise there is good chance that implementation will fall apart and good amount of time will be wasted.

Assuming only me working on this half time, I think this could be done in relatively reasonable time if all things go well (never happens), but it would be probably best to think about such change in 4.0 release. Not only because of scope, but I am not sure if current files could be ported over to new system, so this would be breaking change.

I think we can discuss this here, but feel free to poke me on chat too, but these discussions should be done openly.

4 Likes

Thanks for the review.

Yes I agree. The idea was to be able to have eg. a (asset browser) preset text animation, which would run at a constant predictable framerate. This is not an issue that’s specific just to the compositor though, and adding it would only add unnecessary complexity. I talked about it with ZedDB and he mentioned that other animation assets will also need to be usable at a constant frame rate. It could be achieved eg. by retiming the keyframe durations when applying on a scene/strip/rig with a different frame rate. (afaik how it will be implemented is still in the open)

I’m not sure about excluding 3D nodes from the sequencer. Just like you can use a scene render as a sequencer strip, one might want to use scene renders in the compositing nodes of strips. Maybe you want to add effects to the render, or composite multiple scenes or render layers together. There is a performance penalty for it, but I think there are also many use cases that people would take the performance penalty for.

On the sequencer side there are some obvious nodes that can’t be used for 3D scenes like the time remapping nodes you mentioned. I’m not sure whether it’d be worth splitting the editors entirely because of those though. Reusable node groups like vignettes, color grading setups, etc. would have to be remade for the sequencer and the 3D compositor individually if it was split. An error could be shown if invalid nodes are used, like how the compositor shows an error when the composite node is missing.

Maybe there could be an option to mark a node group as either a sequencer node group, 3D scene node group or both? Invalid node types would be highlighted as red and greyed out in the add menu, and they could only be applied if it’s marked for the use in that context (sequencer or 3D scene). That could work as a compromise between them. I’m not sure about it though.

Accessing strips outside the “host strip”?

I asked @iss about this in the chat, and the general idea he has is being able to use strips outside the strip the node group is applied on. Somewhat like this:
image

I intentionally left it out, since I think the compositor and the sequencer should be separated. Eg. you shouldn’t be able to bring in strips from the sequencer to the compositor, other than the strip the node group is applied on.

The strips in the sequencer are layered on top of each other in a clear order. If you were able to bring in strips to the node tree, this layer system would be broken, which would make it confusing to use. Maybe a strip is used by multiple strips, maybe it’s used multiple times on different “layers” in the same strip. With nodes it’s easy to see at a glance how each input is used and where, but this wouldn’t be possible to visualize in the sequencer.

The only thing that the sequencer could be used be used for in this case is timing the strips. Since it would break the layering and make it confusing to use, my suggestion is moving that responsibility to an animation/keyframe editor. This is something that I briefly mentioned at the end of the proposal. The videos that were outside the host strip would instead be brought into the compositor as an image sequence node. The start and end times of those clips would be adjusted in the node keyframe editor.

In addition to providing a solution to that issue, it would also help displaying node tree keyframes in general. Node tree keyframes are displayed in the dope sheet using the name of the value being keyframed. The node name isn’t visible, which makes it hard to find what you’re trying to keyframe.
Which Fac is which?:


The new editor would show a clear hierarchy of what nodes the values are under, which would make working with it easier.

I haven’t had time yet to make a full proposal for it, but it would work very roughly like this:

The VSE and strips are useful for combining different shots / video clips together, adding transitions between those clips, and timing audio. That’s something you can’t do with nodes. Nodes are useful for adding effects onto those clips. The timing of the different effects applied on the clips is relative to that clip, and exposing them outside the node tree would make it difficult to work with the timing of clips in the sequencer.

Selecting video clips that aren’t in the timeline as inputs could be useful though, if you for example have some stylized node group preset that tiles different clips together.

Input/output handling

Yeah, I think there should be some further consideration on how the input and output would be handled. My earlier reasoning for using channels for everything was that you could reuse them in different situations, but looking back on it I don’t see why one would use eg. a node group for a transition on a normal strip. I think a better design should be considered for this.

As for exposing everything in the node group input, I mentioned in the proposal that it would be hard to see what is passed and where. Eg. if you have a transition applied on a node group with four inputs, where are the two clips used for the transition passed?
@iss’s solution for it in the chat was selecting each of the inputs manually. Something like this:


I think combined with the idea of including strips outside the “host strip” it makes sense, but it also has downsides. Mainly that you have to manually choose what goes where every time you add a transition. This would slow down the workflow a lot, since generally you only apply the transition in a certain way: from the in strip to the out strip with the transition factor being based on the transition length. That’s why I think it should be defined in the node group itself.

My revised idea is having separate input nodes for normal strips, transitions, and the special meta strips. I think those three would be enough for all cases, since transitions are the only effects you usually apply in an NLE. More complex effects can be done inside the node group, as it is a better tool for the task.
Transition example:
image
When used inside another node tree the inputs and outputs (From, To, Factor) would be shown like in the previous proposal.

If I understood correctly the thing that would be different from my proposal is having a separate node editor as a work around? I think that sounds like somewhat of a hacky way to solve the issue.

I guess, that’s what ISS is writing about here:

1 Like

Okay. If it makes sense from the UX perspective, I think we should do it that way. If it’s there only to work around old code, I think that’s a bad way to solve it. I’m currently not quite convinced by the idea though because of the reasons I mentioned in my reply to iss.

Moving the VSE content out of scene data structure would allow for much better integration into the various editors including the compositor since it would make it possible to access the current scene in scene strips(which was possible up to 2.79) and thereby avoid the problems of recursion(both in and out of the sequencer and compositor). So, for me, this is a superior solution. Suggestion: Make the Sequencer contents into data-block, which can override the switched to Scene-Sequencer contents And solving the slow speeds of scene strips should be prioritized over any node work, imo, since it is already possible to get compositor output into the sequencer that way: The poorly exposed Scene Strip properties, which results in bad UX:

Currently, there are already UIs for effect strips and UIs for modifiers, adding a UI for basically the same functions in a node editor(but add similarity to Resolve workflow), would just result in more UI clutter(though nodes are nicer than strips(but adds Afx like workflow) and modifiers(but adds similarity to 3d workflow)).

An alternative way to get strips in and out of the compositor could be as movie-clip-data-blocks. If the VSE had the option to convert imported a/v material into movie clip/sounds/image data-blocks(add on for this: https://gist.github.com/tin2tin/0da859896432245fd15cfe960fdd9ee9) Using the movie clip node ex. the movie clip data-block(source file) can already be imported into the compositor, so what’s missing is storing in and out points of the movie/image data-blocks from the strips and reading this into the node, and then a data-block output node would be needed. This way the source strip could be round-tripped in the compositor and returned into the movie strip(with an option to select the compositor-data-block output instead of the source-a/v material).

Converting a/v material to data-blocks will also have the advantages of the Asset Browser and the Outliner will be able to show these files and users can tag, organize and drag & drop them into the sequencer:

1 Like

Afaik effect strips are currently being phased out. The transform strip has already been removed, and the general idea seems to be that once there’s a better system they should be removed altogether:

So this would replace both the effect strips and modifiers. That would remove clutter from the current system.

I think an issue with that is that the node setup is linked to a specific video clip. The same setup couldn’t be easily reused on different clips.

Moving the VSE content out of scene data structure […] is a superior solution.

I think these are separate issues.

No they’re not, if this happened, the VSE could pretty much use the Compositor as it is. The reason for the VSE being unable the use material from the Compositor(or 3d view) of the same scene as the sequence-edit is in, is because the Sequencer data is stored within the scene. The Movie Clip editor, the Image editor the Text editor do not do this. So this is not uncommon practise to store editor data outside the scene data.

It’s not about performance, it’s about workflow. My point is, that sequencer does not have access to some data like render layers, so it you want to composite 3D render with image sequence when doing motion tracking for example, it is best to use compositor as you have much finer control over layers.

In compositor you make a shot, in VSE you edit shots. Technically this line could be blured, but this is another design talk.

For reusing node setups this could be made easy by other means, like copy paste logic and attach appropriate inputs. or even whole tree could be copied and endpoints can be automatically substituted to fit VSE usecase. In any case there are many ways to resolve this, I don’t think it’s super important to go into details, but it has to be recognized, that there are nodes that make no sense in one or other context/workflow.

Regarding to accessing other strips - you say that these could be represented as image sequence and timing could be adjusted: In this case you don’t really need strips at all then, which is not necessarily bad. My point is, that strips do actually represent footage timing and offsets. If you use strips as inputs, these should be respected various effect factors could be keyframed of course or plugged into something like “Time Info node”. Otherwise if you decide to change transition time in timeline, you will then have to adjust node tree which will be very annoying.
Not sure if I understood this point well enough though.

Not quite sure if I understand. Do you propose, that there would be special “empty” strip type to do transitions?

Realistically speaking, you will have node trees with 1 or 2 inputs - plain effect and transition, so this could be possible to enforce by design and would solve a lot of problems, but also limit flexibility.

It can be done as 2 main types and still allowing for more image inputs if needed, but relationship between these won’t be managed in any way.

I wouldn’t say they are being phased out, just that I am not sure how they will work with channel headers. Nodes can definitely help, but this still needs a design that will work well so in meantime it’s most likely I will hack channels with effects as well as I can.

@tintwotin This proposal is about using compositor like nodes for individual strips not whole sequencer output. How strips are stored doesn’t change the fact, that it has to be designed if and how node system can work for VSE. Hacks could be done even now to use copmpositor in almost any imaginable way.

1 Like

As Blender is now it is possible to roundtrip a strip in the compositor by opening the movie path as a Movie clip it becomes a data block that can be added with a movie-clip node in the compositor in an external scene. This scene can be returned to the sequencer. It is a tedious process, but it could be automated with an add-on, however, the performance is pretty bad and there is no caching of scene strips. The gain is that users actually can use this process to comp 3d elements with VSE strips, which isn’t possible in the solution ISS suggests, as I understand it.

With the upcoming realtime viewport compositor: https://www.youtube.com/watch?v=GdN3eLrdyXA how do the VSE get any benefits from this? Currently, the performance of anything going through Scene strips is pretty poor, and having to reference external scenes is also tedious, which is why I keep returning to moving sequencer data out of scenes as the solution to a better integration of the Sequencer into current and future editors.

If that’s a technical limitation that can be overcome, it would be useful for 3D transitions, or other similar 3D “templates” where you want to bring a strip from the sequencer into a 3D scene.


The 3D features are probably something that might be better to discuss about in a separate proposal later though, since they’d require planning on how they should be implemented.

Okay. As long as there’s a way to reuse nodes in some way I think it makes sense to split it into another editor.

Is there a thread/design for the channel headers somewhere?

VSE strips in the node editor

Yes, kind of. I think I didn’t explain the idea very well earlier. The image sequence nodes would work the same way they do as of now. They load the video or image sequence from the hard drive. They wouldn’t have any relation to any strip in the sequencer timeline.

The Media In node would bring in only the media of the strip the node group is applied on, the Transition In node would bring in the media of the two strips being transitioned, and the Meta Strip node would bring in the media of each channel of the meta strip as an input. (The Meta Strip thing is mostly a convenience feature that could be implemented later if there’s demand for it) These would be the only nodes that bring stuff in from the sequencer.

I think it’s best explained with images:

Media In, and Image Sequence:

Transition:


The transition factor would be 0 at the start of the transition strip, and 1 at the end of it. That way the transition would work with no need for modifications when adjusting the length of it.

Yes, if I understood correctly:

Meta strip with separate channels:


This meta strip feature lets you bring in each channel as a separate input. It’s convenient if you have multiple strips after each other that you want to layer and add effects to in the same way, since with nodes it’s inconvenient to add strips after each other. The different colors in the example represent different clips (this made up example doesn’t make much sense as you could do the same thing in the VSE, but it’s there just to show the basic idea).
This isn’t a must-have feature, but something that could be implemented later if there’s demand for it.

Yea, for adding effects on strips they wouldn’t be needed (except the strip it’s applied on and transition strips). Timing audio and making a sequence of the different shots would be done in the sequencer.

Yes, that’s my idea. For more advanced effects requiring more inputs, the image sequence nodes can be used.

In my proposal more media inputs can be brought in with the Group Input node. You can’t choose any strips from the sequencer for those inputs though, only image/video files.
They would be exposed like shown here (except without the channel input field):

Transitions can already be done in the compositor by converting strips to Image data blocks(but if sequencer data was moved out of the scene, you wouldn’t have to do the scene switching(unless you already had an output node in it)):

On the topic of separating strip and scene nodes again: I noticed that there already are cases where a single editor is used for multiple contexts with differing nodes.

In the shader editor, there’s a dropdown for object, world, and line style:
image
Each dropdown option has their own available nodes. Some are shared, some aren’t. For example for object materials the Material output node is available, which Line Style doesn’t have, but Line Style has the Line Style Output node.

Maybe the sequence/scene node separation could be done the same way, like this:
image

1 Like

Just being able to save and switch between different node trees in the compositor just like material and geometry nodes would be awesome.

For me the best solution is:
.to have a NEW video editor (called PRINCIPAL VSE) which is a copy of VSE but independent of scene. The only editor that could add a “SCENE STRIP” because it is not link to any scene. So like that we could change the scene without changing the Principal sequencer content.
. The old VSE (can be called SCENE VSE) must be disallowed from adding a " SCENE STRIP". Nothing will be changed here. it will continue to serve for render setting or for storyboard and animation video editing. We could also select some videos in PRINCIPAL VSE and send them to SCENE VSE for regrouping or pre-composing.

At the end we will have two video editors as we have many node editors:
PRINCIPAL VSE & SCENE VSE.

SOME ADVANTAGES OF THAT NEW P-VSE:
. From That new editor PRINCIPAL VSE we could send our selected videos in NODE COMPOSITOR( for pre-composition, FX and transition needs ) , in CLIP EDITOR( for 2d/3d tracking, masking, stabilisation creation) , 3D SCENE ( as background image or 3d effects), or in Scene vse.
. STORYBOARDING FACILITY: Changing scene while the sequencer is constant.
. Could serve as RENDER QUEUE.
. It must have it own timeline or use the timeline of clip editor.

Note that with this implementation all our request will be possible. @iss please can you try this even as a proof of concept.

@Timothy_m , @ok-what , @tintwotin what do you guys think about my proposition.

I already created an add-on that do most of all those stuff. The add-on is call BACHADAM VFX ADDON.
I will link it video here soon.

Bachadam vfx add-on first version

2 Likes

Second version of that add-on .

If you need to test that add-on just let me know.
The last version of that is not yet available it is for blender 3.2

That’s an incredible addon. Thanks for sharing it.

I posted a reply to your ideas here, as that thread is more relevant for it: Suggestion: Make the Sequencer contents into data-block, which can override the switched to Scene-Sequencer contents - #6 by ok-what

I think its a great add-on and I’d love to see something like this implemented in Blender.

I want to share that addon of mine here for free so you can try and see it potential and limitation. HOW TO SHARE A .PY FILE ON THIS PAGE