GSoC 2024: Sprucing up the Video Sequencer (Feedback)

Feature & Design Feedback
1

This is the dedicated feedback thread for my GSoC 2024 Project aimed at improving the overall video sequencer UX.

You can view the first draft of features at my original proposal.

You can also check out progress over the summer at the weekly reports thread.

Project Details

Snap strips to markers (suggested by Nebulon1212)

I have implemented this suggestion and the patch has since been merged into main.

Link and Unlink Strips

I propose an additional option to “link” and “unlink” strips in the sequencer context menu. If a user links together multiple strips, selecting one strip will automatically select all other linked strips, and operations on one strip will also be applied to all other strips. Any newly imported video files will have linked strips by default.

430×684 34.3 KB

If a user selects any strip which is already linked, there will be no option to link it with other strips and cause double-linking. Linking could have a keybind similar to joining in object mode (ctrl+J), but P is reserved for setting a preview range, so if there is an unlinking keybind it’ll have to be something else.

The visual design of linked strips is subject to feedback, but one solution could be to append a strip “linking number” to the name/source/duration of the clip identifying its link partners (which could also be color-coded). For example, if a user links an audio and video strip, they could both be associated with link 1 / L1. Then further linking will increment the link number.

Active Channels

I propose adding a new “active” property to channels which will give the user the option to explicitly select “destination” channels for operations that create new strips (such as adding or pasting). A mockup of the altered channel look may be seen below:

active-channel-mockup

The mockup is subject to feedback, but the intent is to let the user click on a channel’s identifier in order to select it, similar to how modifier selection works.

The active channel property would work as follows:

  • Channel 1 will be active by default
  • All pasted or added strip(s) will attempt to place themselves in the active channel at the current playhead position
    • Overlap mode will decide how the strips behave if there is already a strip present in a given channel at the current frame
    • If strips from multiple channels were copied, or if imported media consists of multiple strips (e.g. audio/video of the same file), the pasted strips will not overlap each other, and will remain in separate channels.
      • In this case, the active channel will be the destination of the lowest strip in the stack (e.g. audio strip for a file with audio + video channels), such that higher strips will attempt to place them in the next highest channels.
  • Potentially: it might be useful to let the user decide which direction pasted strips will go based on the position of the mouse when pasting, similar to how the location of the mouse when cutting determines which side of a strip will be selected after the cut.
    • That is, if a user has their mouse positioned anywhere above the current channel, then the pasted strip will attempt to place itself in the next free channel above. If they have their mouse positioned below the current channel, then the pasted strip will attempt to place itself in the next free channel below.

Preview Snapping Support

Currently, users can grab (G), rotate (R), and scale (S) strip bounds in the preview window. I intend to add a snapping toggle for the preview window which may function similarly to sequencer snapping but with differently defined snap points.

703×362 107 KB

The below image shows the areas of the preview in which I intend to implement snapping support. Bounding boxes of a strip may be snapped to the dotted red lines corresponding to the top, bottom, left, and right render borders. The pivot point of a strip may be snapped to the dotted blue lines corresponding to the horizontal and vertical centers of the render output.

Note that the dotted lines are only meant to clearly illustrate the snap points. For an actual implementation, the drawSnapping() function in transform_snap.cc could be used to draw lines when a valid snap is found.

Although text strips’ bounding boxes are larger than they should be to benefit from these snapping changes, there are options to choose text alignment with respect to the render output, so they already benefit from snapping in a sense. I will investigate dynamically sized bounding boxes for text only if I have extra time.

Additional Tasks

If all deliverables are completed and thoroughly tested and there is still time remaining within the project, I will take on additional tasks:

  • Snap Strips to Start/End Frame, Preview Range option
  • Snap Strip bounds to hold offset (so that extending a clip only extends to the end of its video data and doesnt create a hold)
    • Potentially rename this option to snap strip to “Content Start/End”
  • Snap bounding boxes to bounding boxes in preview
  • Move channels feature
  • “Add channel below” feature
  • Give Sequencer & Preview View Type all functionality afforded to the separate view types
  • Improve pitch management of audio strips (retiming)

Changelog:

  • 5/8/24 – Change active channel selection mockup to mechanism similar to modifier selection (instead of having separate buttons like mute and lock). Also clarify linking/unlinking visual design.
  • 5/16/24 – Add additional goals to tackle, clarify active channels proposal further
14 Likes
2

Nice! Some random thoughts:

The visual design of linked strips is subject to feedback, but one solution could be to append a strip “linking number”

Would probably be good to discuss this with @pablovazquez and/or @fsiddi. I wonder whether some sort of “visual” linking indication would work better than trying to see and compare the numbers. I don’t know how exactly it would look like though. Maybe arrows/lines (similar to how nodes are connected) that would “connect” the strips together, drawn under everything else? Or maybe some sort of “blurred colored blob” under the strips that show it.

Preview Snapping Support

There was a discussion about additional visual indicators in the preview area, including snapping to them, that might be useful. Quoting from the chat that I found:

  • “but some more helper lines, like center and thirds and golden ratio would be cool to have for the sequencer preview, probably in the n panel”
  • “It’d be great to have the same helper lines the camera objects have”
  • “maybe the Gsoc snapping in the Preview could have snapping grids to ex. thirds as an option?”

I will investigate dynamically sized bounding boxes for text only if I have extra time.

That might (or may not ;)) happen independently. Trying to make text boxes smaller than whole screen is on my todo list. This might also help with performance (especially for blurred shadows / outline text), but also would make text strips much easier to pick/select in the preview area.

6 Likes
3

Suggestions for a “linking” feature:

  • display a link icon in the text labels area
  • selecting a clip with links should be the same as:
    • select all related clips
    • make the first clip that was selected the active one
  • treat all strip operations (transform, deletion, etc) as if we are dealing with multiple selection

Besides the link icon, this feature could be implemented very transparently.

3 Likes
4

Would it be possible to also add snapping to frame range start & end? Most constant headache I have in VSE

4 Likes
5

I initially thought that would not be enough (just the “link” icon does not indicate which other strips it is linked with), but now that I think of it, I guess majority of usages would be where video and audio strips are right above each other, or similar situations. So indeed maybe worth starting with just that, and see how it feels. And then if it feels like that is not enough, someone would need to think of a more elaborate approach.

6

In the VSE, there is already highlighting used for indicating a “link” between an effect strip and the channel or strip it is connected to:
link

This however, is properly too vague indication for an action which potentially could change the linked-to-strip(s).

So, looking at the “industry standard”, when you select the parent, the child is also selected, same thing with handles, drag operation etc. In PP and DR all linked strips are highlighted with the same kind of outline, and in DR there is the icon, as mentioned too. Which would mean that all strips and handles linked to the active strip(the most recent selected strip), should get white outlines(in order to be in line with IS). Question is if that will be a source of confusion since the white outline is also an indication of what strip properties are displayed in the sidebar. So, the user would have to remember which one of the strips with a white outline was clicked last. In DR multiple strips are highlighted with the same color outline, and only one of them has it’s properties displayed. So, I guess that would work fine, but it could be tested.

The main thing is that all linked to strips are manipulated, and therefore all the indicators for what is being manipulated should be activated, so it is clear that ex. dragging a left handle will mean that the linked strip also will have the left handle dragged.

Another thing is if the link should be a parent/child relationship like in the 3d space? So, if the parent is manipulated the child will also be manipulated, but if the child is manipulated the parent isn’t manipulated. This would mean that if the parent is a video strip and the child is an audio strip, the audio strip will follow the video strip, but if you ex. want to make the audio start earlier for a J-cut or end later for a L-cut, you can just change the audio without unlinking to the parent first.

7

On the snapping, maybe I’m not understanding it well, but do you plan to support all of these options for strip object’s pivot point?

image

And what snapping-to points do you plan? Other strip objects? If so, what part of the strip objects? And how to select between them? Composition grids?

It can become very complicated, and a lot of options to implement. And properly too many, so, what are the essentials?

(For me, the most essential thing would be to use snapping to ex. align texts or images in columns or rows, with equal spacing (without using X and Y type-in values))

8

On the Active Channel, I think you may be complicating things a bit. Isn’t the only purpose for this to let the user control what channel to insert Menu > Add > File Browser material? The “industry standard” is having a column of radio buttons in the channel headers which both indicate where V and A goes. As Blender does not have strip-type dedicated channels, we can’t follow those lines.

Or they should have 3 state radio buttons(Video, Sound or Off), but that wouldn’t work, because you would unintentionally switch off the video insert channel, when toggling from off to video, to sound.

However, if audio is always inserted below video(The selected active channel is always for video/image, and the channel below it, is always for audio), it would simplify things. This is an industry standard, too, that video is inserted above and audio is below. In Blender this was the ambition too, but it was never implemented correctly: #118898 - VSE: Sound vill be inserted above video, if there is a strip in channel 1 - blender - Blender Projects If this is fixed, then there would be a consistent way to insert movie+sound strips.

So, if the channel 5 is selected, video will be inserted(overwrite, insert or shuffle) in channel 5 and sound in channel 4(overwrite, insert or shuffle). In Channel 1, it would be understandable that movie would go into channel 2 and sound into channel 1.

Btw. the insertion methods have a lot of room for improvement, ex. the Extend mode, should really be an Insert (industry standard) mode where a cut is made, and the new material is inserted at that frame, while all following strips are pushed to the right, Instead it inserts the new material at the nearest existing cut, and is actually shuffling instead of inserting. In more detail: Design: Missing functionality in the edit modes in the VSE

One more btw. in the File Browser there are some values indicating that the user can control into what channel to insert the imported material, however, afaik these have never worked on this side of Blender 2.49+, and properly could/should be removed:

image
image286×505 15 KB

9

Thanks all for the comments & ideas.

This is a great idea, and shouldn’t be too difficult, so I’ll look into implementing it.

To avoid cluttering the snap menu too much, would it be fine to combine these two into an option for snapping to “Frame Range”? I can’t think of a situation where you’d only want snapping to the frame end but not the frame start. Or is more fine-grained snapping needed?
Also, would snapping to preview range be useful? In that case, maybe all these options could be combined into an option to snap to “Ranges” (which would include frame + preview range)

I think it would be more understandable if a link relationship meant that changing either strip would affect both. If it’s desired, I could add a setting somewhere to make strips unlinked by default (reverting to old design) so that if users want to quickly perform J/L-cuts on newly imported media they still can without having to unlink first.

Yes, that is the plan. Regardless of what your pivot point is set to, it should be able to snap to the blue dotted lines in the mockup.

I don’t see a good reason for snapping pivot points to bounding boxes of the render, (red lines), so you would only be able to snap the bounding box of your video to the render edges.

Snapping to other strips’ bounding boxes in the preview is something I didn’t mention in the original post, but something I could add if I have extra time – it would ideally allow for video edges to be snapped to each other.

This is what I was going for-- if you add new media then the “active” channel will be the destination of the lowest channel of the imported media. In the case of two-channel A/V this would be the audio channel, then the next higher channel for video.
Will update original post to clarify this.

For pasted strips, it would try to retain the channel offset of each strip to each other after pasting, but with overlap mode taken into consideration.

Thanks for bringing this to my attention, never noticed it. Will probably add some sort of button to choose whether or not you want it dropped into active channel (default) or a manual channel.

1 Like
10

If you aim to complete the strip snapping options, snapping to hold offset is missing, and due to the handle design it is very difficult to drag to the last unhold/unfreezed frame without the snapping: #90280 - VSE: Snap to the 'Hold Offset' of strips being transformed - blender - Blender Projects

11

I saw this discussion when trying to figure out what “Snap to Hold Offset” even did – as far as I can tell, it might have worked at some point during the original snap patch but now you can’t even snap strips to other strips’ hold offsets, let alone themselves. Definitely something I would like to take a look at, but maybe @iss has worked on this in a local patch?

12

If you import a movie strip, grab a handle drag away from the strip before/after, the strip will get a darker color, these are freeze frames, and in most situations something you want to avoid, which is why a snapping point at these points would be very useful, but only when dragging handles of that strip. Strips should not snap to other strip’s hold points.

13

It did work, maybe it was broken by allowing negative strip offsets (seq->start/endofs). For hold offsets (seq->anim_*ofs) the math may be bit outdated, you can use SEQ_time_start_frame_get() and SEQ_time_content_end_frame_get() as snapping points. The option would be perhaps better named as snap to content start/end.

14

Absolutely love the suggested additions, especially active channels and linked strips. When I edit, I usually do two edits of the same video, vertical and horizontal, and having this, means my editing workflow will be faster by a few minutes every time! Thank you!! The snapping and pasting and active strips will help a lot too! Thank you!

2 Likes
15

@linen I noticed the remark on channel header selection was still missing a design. As mentioned, a good and safe way to figure out a design, is to see if there is something already doing this in Blender (channel selection), and then implement something in consistency with that. The NLA has channel selection, where you just click the header area, and the full header area goes greenish, and so does the full channel(track). And the channel(track) name goes yellow.

select header_nla

You could simply do the same, and to indicate where the audio will be going you could have the channel below the active channel(could be blueish like movie strips), become a slightly darker greenish (like sound strips) color, and it’s text should become a slightly darker yellow. Or the texts could follow the strip colors for selection: pure white when active and slightly grayish when unselected. And the audio should always go into the channel below the video.

2 Likes
16

I think it would be interesting to expose the waveform preview power of the sequencer in API to create waveforms, either as image and/or data.

An example to give context:
I’m the author of the Sound waveform display addon: GitHub - Pullusb/sound_waveform_display: blender addon - Display audio waveform in dopesheets, graph, timeline
It’s quite popular amongst animator. But currently it downloads an FFmpeg binary to generate waveform, and the generation process is super-slow.
Having an internal waveform generator would allow fast waveform creation without any external binary download (and thus I’ll be able to submit it to blender extension platform :grin:)

||| Side-note related to ffmpeg : In the past, I asked if blender’s integrated ffmpeg could be exposed (to avoid using third party binary for addons). But it’s not a trivial task, and the idea was not considered at the time. I still think it’ll be a great thing to expose it for many other video related purposes. In API or as background command for external call. |||

First, I thought it would be nice being able to create an image datablock within the blend, using an API call or an operator on sound datablock. Then we could do whatever we want with it, either in UI or API (use it, save it on disk, …).
But maybe it’s too much work on the dev side, I don’t know.

In the end, it would be probably easier -and more useful- to get data instead of an image.
In short, an API call on sound datablock to create and return lists of points coordinates.
That way, user will be able to draw with openGL gpu module or even create 3D geometry.

As I see it, it would be a list containing one sub-list per audio channel. Each sub-list would be points coordinates:
X : Time (in seconds)
Y : Sample value (normalized, same as it’s done for sequencer strip already)
Audio 2D coordinates would look like this:

[Vector(0.0, 0.2), Vector(0.03, 0.6), Vector(0.06, 0.7), ...]

The function would need an argument to define the sampling.

coords = bpy.data.sounds['my_sound.wav'].extract_waveform(sample=0.1)

Probably best to have resample as function argument, otherwise, we would need a separate function to resample the raw lists, as it’s done in pyav : Audio — PyAV 9.0.3.dev0 documentation
Also, not sure if this should be a value in second as I’m not a sound data expert, but for example 0.1 here we would get 10 samples per second.

Or, it could be a dedicated Waveform class, that would have a resample method:
coords = bpy.data.sounds['my_sound.wav'].waveform.resample(sample=0.1)

What do you all think ?

2 Likes
17

@Pullup I would imagine this feature to be useful for various use cases, but you are commenting on feedback topic for GSOC project. It is very off-topic and feature request which is against forum rules. Feature requests should be posted to https://rightclickselect.com/

18

I’ll take the blame for this – I didn’t know feature requests were disallowed. @Pullup dmed me this suggestion on blender chat and I told him to mention it in the devtalk thread so we could have a public discussion about it. Apologies…

19

I am not mad and don’t want to be discussion police either. If you want to discuss this as potential goal of GSOC, you certainly could.
Personally, I am not sure if this API would exist and if it will, I am less sure this would come from sequencer or sound datablock in current shape.
In Blender this would be either implemented as input node for geometry nodes, or sound volume metering. But in either of those use cases, I don’t think, that there will be array with downsampled original sound.

If somebody would like to provide access for waveform data in current state, I would say go ahead - perhaps better than having nothing. But it may not be super easy - I have never tried to spawn threads from there, but maybe it’s OK.

For particular addon in question - I would just launch blender in background, render scene to wav, read it from python. Should be quick…

1 Like
20

A header related problem, introduced when it became possible when there is close to unlimited number of channels, is it can become very difficult to exactly see what channel the strips you ex. want to mute are on. Maybe some kind of hover over header highlighting of the channel could be introduced or something else to make it more visible what channel a clip is on(when the channels are very small).

1 Like