GSOC2024: Draft: Sequencer Improvements

Name

Ruoyu Lin

Contact

Email: [email protected] [email protected]

Blender.chat nickname: alxdsptr

Synopsis

The project focuses on the strips in video sequencer and the goal is to improve users experience when editing video and audio in Blender Video Sequencer. The main goal is to add a function to link strips, which allow two strips to have strong bound between each other, and when applying tools like transform, retime, cut, to one strip, another strip will change at the same time. Another goal is to improve positioning of strip on adding and pasting. I plan to add a function to select channels just like in Premiere, when adding or pasting strips, strips will appear first on the selected channel. Then if time permits, I will try to add a function to allow retiming the audio strips without changing its pitch.

Benefits

Adding the link strips function will allow users to edit videos and sounds more quickly and easily, improving the Sequencer workflow overall. Currently the selection of a channel when adding new strip or pasting them from a clipboard feels random on user-level, and the improvement of positioning will make this behavior clear and predictable. Pitch correction allows users to do something that in the past they can’t do in blender. Ultimately, the project would improve users experience when editing video and audio in Blender Video Sequencer.

Deliverables

  • a function to link strips

  • better positioning when adding and pasting strips

  • pitch correction when retiming audio strips

Project Details

For the link strips function, I would add a button to right click menu that says “link strips” or things like this. If the user selected one strip and click this button, then if he clicks another strip, these two strips will be linked together. If the user selected two strips and click the button, these two strips will be directly linked together. (Since I don’t see points for linking two strips with different length, and it’s hard to implement and maintain consistency if the two strips have different length and need to retiming or cut them at the same time, so I would restrict this function to strips with same length). And when a video with sounds is dragged into sequencer, it will be automatically linked with its sound strip. After two strips are linked, when you select one strip, the other strip will be automatically selected, and when you drag the strip, they will move together, when you add retiming keys, dragging the retiming keys and cut the strip, the two strips will have the exactly same changes.

For channel positioning improvement, I will add a select channel function, when clicking areas around the channel name, the background of this channel will turn blue, meaning that it’s selected. When adding or pasting strips, the strips will appears on selected channels, channels near the bottom will have higher priority, and if after pasting (or adding) the strip will overlap with another strip, blender will act correspond to current selected “overlap mode”. If no channels are selected, blender will act just like now.

And for the pitch correction function, I plan to add a option “pitch correction” in “Time” (like in the figure), if it’s selected, when retiming the audio strip, its pitch will remain unchanged. After doing some search, I think implementing this from scratch and get a good result is quite hard, I would consider using soundTouch to implement this.
image

Project Schedule

My last final exam is on June 19th and I can start to work on June 20th. I expect the project lasts 12 weeks and ends on September 12th.

Timeline:

June20-30 (1.5weeks) get familiar with the blender codebase and discuss about the details with mentor

July1-21(3weeks) Work on and complete Link Strips deliverable

July22-Aug11(3weeks) Work on and complete Channel Positioning deliverable

Aug12-Sep1(3weeks) Work on and complete Pitch Correction deliverable

Sep2-Sep12(1.5weeks) clean up time and add documentation

Bio

I’m currently an undergraduate student in Peking University. I’m interested in computer graphics and I’m currently doing research about character animation in my school’s Visual Computing and Learning Lab. I learned to use Blender several months ago, and as for video editing, I’ve used premiere pro for a long time and edited lots of video (including a micro film of my class). I’m familiar with C/C++ (since I use this to solve algorithm problems, and I’ve also write a few little projects using c/c++), and I’m getting familiar with python (I’ve use python to write web crawlers and deep learning, but haven’t systematically learn its syntax). I’ve trying to get familiarized with blender now by looking at past projects submission.
Here is my pull requests for a bug fix: #120184 - Fix the alignment issue caused by fix for #120077 - blender - Blender Projects

8 Likes

Hi, Thanks for proposal, overall it seems good to me, but have few notes:

Most importantly, this year I am unable to mentor, so please message @aras_p in this matter.

It would be good to know what project timescale are you shooting for. The strips linking is kinda sorta rabbithole IMO, so I could imagine this to be quite large project. There is infrastructure to do this already, but it needs thought out design, that would likely be iterated on few times at least partially. For design of pretty much any feature, I would recommend to consult proposals with @fsiddi.

The “pitch correction” feature I have already started to work on. I have chosen a simple phase vocoder algorithm, which is not that complicated, but I am not good with math, so instead of looking at theory and making implementation, I just reverse engineered some python implementation and used human readable variables and function names (as it IMO should be). And that is where I ended for now. Unfortunately I did not have time to develop this further, but if you end up working on this I can send you some resources on this topic. What may be a bit complicated, is implementation in our sound library - Audaspace. But since this works with pitch, I would say, that it shouldn’t be that hard. Perhaps hacky. It would be hacky anyway, since this would have to be done as part of library data setup…

Finally just FIY, it is almost requirement to make at least one pull request on our tracker, to fix some issue in whatever area of blender, mainly to see, that, well basically, that you can do that :slight_smile:

Please avoid references to other commercial software due to concerns of intellectual property rights. Eh, this is the world we live in :confused:

In any case, good luck!

Great to see a proposal for the VSE. A quick note on this:

Not being able to do J & L cuts with linked strips is a serious limitation to standard video editing, where you often add audio fades, or let the audio start sooner and/or finish later than the video strip.

Blender has a system for parent/child relationship, maybe you can take some inspiration in that on what to include and what to leave out, and how to do it? Should you be able to link more movie strips together, how should that affect image rotate, scale, move? How many strips should be possible to link to? Doing ex. surround the user would have several panned audio strips to move or not? How to illustrate the linked relationship? Etc.

3 Likes

Thanks a lot for your feedbacks! I will try to make a pull request and fix some issues. And I won’t mention other commercial software in the future (lol)

Thanks for your reply! I will consider your suggestions.
Actually, I’ve considered some of these, for example, about image rotate, scale and move, but since video strips and audio strips can be linked together and audio strips obviously cannot be rotated or scaled just like video strips, so if linked video strips will be rotated together, then the link function loses its consistency. But yes, I do think sometimes having multiple video strips linked together and can be rotated or scaled together is useful.
And for J&L cuts, dragging and cutting feels natural, but if someone wants to retime an L cuts from the right side, then things will got really tricky.

1 Like

Power to you!!! Really looking forward to any progress you achieve.

Generally my thoughts are similar to what @iss said. w.r.t. linked strips, it would obviously need some sort of “design” work (from Pablo or Francesco I guess?), like for example, how are “linked” strips indicated visually in the timeline? and so on.