GSoC 2025 VSE: Retiming improvements / compositing modifier

Contributing to Blender Summer of Code
1

Hi,

This post is for discussing if either of the following ideas I have would be suitable for a GSoC project. There are many great proposals already from other people, but I’m giving it a shot. :slight_smile: If there’s interest, I can flesh this out into a proper application.

VSE Retiming improvements

The retiming data would be stored as an f-curve, where the x-axis indicates the original frame, and the y-axis indicates the remapped frame for the strip.

The following improvements would be made to retiming:

Custom transitions

As of now, the VSE only has one soft speed transition as part of the retiming tool. There is a way to achieve custom transitions with the speed effect strips, but these are very cumbersome to use.

As the change would make it so that the data is stored as an f-curve, the transition of the retiming tool would be fully customisable in the graph editor. As this wouldn’t change any existing retiming tool workflows, this would have the upsides of both the old retiming effect strips, and the new retiming tool.

Reversing video and audio

As of now:

  • The only way to reverse a video is with the “Reverse Frames” toggle
  • It is not possible to reverse a video, while preserving the existing adjustments made to the handles of the strip
  • The “Reverse Frames” toggle doesn’t work as expected with the retiming functionality
  • There is no way to reverse audio

To fix these issues, reversing would be included as part of the retiming functionality. This comes naturally with the f-curve changes. As this would replace the old “Reverse Frames” toggle, old code would be removed making the VSE more maintainable.
A “Retiming → Reverse” operator would be added. The operator would preserve any already existing changes in the strip, as expected.

These changes would not change any existing workflows

VSE compositing modifier

A “compositor node tree” modifier would be added. This is a highly requested feature.

A concern that has been raised about this before is if it would end up being unusably slow.

I have experimented with code that adds this functionality before (video):

(Uploading a video here didn’t work. Hopefully this link does)

If there’s interest in this, I can describe it in more detail. The retiming functionality is my primary focus, though.

Contributions to Blender

I’ve done bug fixes and small improvements to the VSE. Here are the retiming related ones, to show I have some familiarity with the relevant code:

4 Likes
2

Thanks for the proposal. This is bit of a mixed bag, which is not a problem in itself, but some topics are quite complicated to tackle properly.

This could be done, but you would need to clamp Y value of the curve, as well as deal with strip changing length as you edit the curve in graph editor. That is not a great workflow. So IMO first you need to address workflow itself, which could be drawn out process with lots of design discussions.

Me and Francesco have talked about VSE having a feature, where you could edit f-curves in the VSE timeline itself. There will be some limitations probably, but it would address some aspect of workflow issue mentioned above. I think, that the scope of this project is enough to be own “large” GSOC project.

This is also something we want, but question is how? At least on user level.
Few things to discuss here:

  • Compositor does not have multiple node trees right now. That wound need to be addressed, and it’s quite big step. This is mentioned in #134214 - Compositor: simplify compositing in Blender - blender - Blender Projects, perhaps elsewhere. I can’t really mentor this, since I am not super familiar with compositor and node system.
  • You need to set up data so compositor can crunch it (involves new input node)
  • You need to configure compositor engine and call it from VSE.
  • Finally, you need to download image from the compositor

Alternative to last 3 steps is to create custom node tree, substitute the global one temporarily and call compositor from usual pipeline.

Also, I think that it would be important to edit the node trees from VSE context, and compositor would be just engine, that is hidden to the user.

There are probably other ways to do this, but these details should be discussed. I see this as another “large” GSOC project.

Since this is your own idea, please make sure to specify what timescale/scope you are looking for: 90-hour (small), 175-hour (medium) or 350-hour (large).

Also as I mentioned for others if you have made PRs in Blender, please list some in the proposal. Or if you made some modifications to the code, you should share these.

1 Like
3

Thanks.

Retiming improvements

My initial post was quite vague, so I’ll clarify on the design.

On channel 1 is the original video strip, that is only moved to start at frame 10. No changes to the handles or retiming.
On channel 2 is the same strip, also moved to start at frame 10. On this strip I will show what each kind of change does to the strip, and retiming data.
In the graph editor is the retiming curve for the channel 2 strip.

First is an example with no changes to the speed/timing of the strip:

image
image839×819 83 KB

Zooming in on the timeline to further illustrate the next examples:
image
image802×786 80.3 KB

As we move the left handle (but not the retiming key!), the f-curve stays the same:
image
image848×841 87.9 KB

(On a side note, it would be nice to have an indication that there are non-visible retiming keys to the left/right of the strip. Maybe that could be part of a stretch goal. This doesn’t have anything to do with this specific change however.)
If we were to move the retiming key instead (in the sequencer timeline), both the left handle (“Strip Offset Start”) and the keyframe would be moved. Here we see, that compared to the previous image, the keyframe has been moved 10 frames to the right:
image
image802×786 87.6 KB

Contrasting it with the case where the keyframe is moved in the graph editor. Now, until frame 20 the y-value stays at 0, so there’s effectively a freeze frame until frame 20. Graph editor changes have no effect on “Strip Offset Start” or any other property of the strip, so the length of the strip is not changed, and there is no need for any special code to update it:
image
image816×796 82.8 KB

Essentially the following changes would be made:

  • The rendering side would only need to read the y-value of the graph. The y-value would be clamped between 0 and the video length. This clamped value would be used as an index for the video frame to display.
  • When drawing the timeline, the keyframes of the retiming f-curve would be drawn on the strip.
  • “Add Retiming Key” would work essentially the same as the “Insert Keyframes” operator
  • Sliding a keyframe would change the x-value of the keyframe, while keeping the y-value the same. If the retiming key is at the start of the strip, the start and end offsets of the strip are adjusted accordingly.
  • “Add Freeze Frame and Slide” would add two keyframes, both with the same y-value. When “sliding”, the x-value of the second keyframe would just change, like described above.

Since this involves versioning code and quite fundamental changes to the retiming code, I expect this would be a “large” project.

Other

I’m short on time until next Wednesday, so as I see the chance of the compositing modifier being accepted as lower, I’m focusing on the retiming proposal for now. I have ideas to address the points you brought up, but the discussion about them may end up going over the GSoC submission deadline.

I’ve updated the original post with some of the relevant PRs I’ve made.

1 Like