GSoC 2025 Proposal - Hardware accelerated video encoding/decoding for Blender's VSE using Ffmpeg

Hardware accelerated video encoding/decoding for VSE

Name

José Vicente Barrachina Bosch

Contact

Synopsis

This project aims to incorporate into Blender’s VSE the necessary functionality to take advantage
of the hardware-accelerated video encoding/decoding mechanisms currently provided by the Ffmpeg libraries.

Benefits

The main and most notable benefits of using hardware acceleration for video encoding and/or decoding are the increased speed and performance. Additionally, using dedicated hardware frees up the CPU for processing other tasks to a large extent and allows for lower power consumption, improving performance in laptops.

These advantages can make Blender’s VSE more attractive to users who need to perform video encoding, decoding and transcoding, thus allowing them to incorporate it into their workflow.

Deliverables

  1. The code required to implement hardware-accelerated video encoding and decoding in VSE depending on the user’s platform and configuration, including changes to the UI necessary to customize the process:

    • Updated Ffmpeg integration in Blender to support hardware acceleration, taking into account licensing compatibility.
    • Implementation of available hardware detection mechanisms.
    • Functions required to manage efficient data transfer between GPU and CPU.
    • Updated Blender’s user interface and added new options to enable or disable hardware acceleration. Added options to select appropriated decodification/codification settings and display performance or status information.
  2. Updates to the user documentation required to explain the usage of the new functionality.

  3. Appropriate tests and benchmarks.

Project Details

The VSE (Video Sequence Editor) is one of Blender’s video edition tools, allowing the user to integrate this workflow with others such as 3D modeling. Thus, VSE permits users to avoid the need for external tools.

Currently, VSE uses the Ffmpeg libraries to encode and decode video through the CPU. However, Ffmpeg provides also some cross-platform support to take advantage of specialized hardware for these tasks.

If you look at the Ffmpeg documentation, ongoing platform support can be summarized as follows:

Linux Windows Apple
AMD Intel Nvidia AMD Intel Nvidia macOS iOS
NVENC/NVDEC/CUVID F F
Direct3D 11 F F F
Direct3D 9 F F F
Media Foundation F F F
OpenCL F F F F F F F
VAAPI P F P
VDPAU P F
VideoToolbox F F
Vulkan F F F F F F

where P indicates partial support and F means full support.

Ultimately, the main objective of this project is to give Blender users the option to take advantage of the hardware-accelerated video decoding/encoding options offered by Ffmpeg, depending on the user’s platform and availability, by implementing the required architecture.

Due to the large number of options and time constraints, the scope of the project should be limited to achieving minimum support for as many platforms as possible.

In addition, there is access to two machines running Linux and Windows 11, one with a dual Intel/Nvidia graphics card an one with AMD, so the project should, for the time being, be limited to this configuration.

The hardware decoding process does not affect the final result, generating an output equivalent to that produced by a software decoder, beyond reducing the number of features supported in more complex codecs. Therefore, it should be used by default if the user’s platform allows it.

However, hardware encoders, despite the obvious advantages in terms of processing speed, tend to produce slightly inferior results to their software counterparts. Therefore, whether or not to use them should be the user’s choice at the time of use, so it will be necessary to modify the VSE UI to allow its activation and configuration according to the available options.

Project Schedule

  • May, 8 - June, 1 (3 weeks): contact with mentor(s) and documentation period.
  • June, 2 - Aug, 10 (10 weeks): coding, test and benchmarching.
  • Aug, 11 - Aug, 24 (2 weeks): project documentation, last adjustments and bugfixing.
  • Aug, 25 - Aug, 31 (1 week): final adjustments, if needed.

Bio

I am what you might call an atypical Spanish computer engineering student. I am currently 49 years old and, although I have worked all my life in other sectors not related to computer science, the year before the COVID pandemic I decided to start studying a subject I have been passionate about since I was a child.

The confinement, and a certain weariness with my job at that time, led me to make the decision to dedicate my time exclusively to finishing my degree, in order to change profession. On last year I finished last degree’s course and I am about to hand in the final project.

Unlike what I have observed in other contributors, I am not a regular user of Blender, but I know of its existence for a long time since I am passionate about computer graphics and video game development. Learning how to use it is yet another pending task in a long list.

I have been an advocate of free knowledge and free and open source software for as long as I have known of its existence and, in this regard, Blender seems to me a clear example of how this kind of projects can be viable and compete with proprietary solutions.

To this day I have collaborated with Blender less than I would like to, largely because of the respect it gives me as a beginner. Last year, in view of participating in the GSoC and to acquire some knowledge of the code, I sent my first pull request #119117. I have collaborated with other free and open source projects but only contributing translations or minor corrections.

I don’t have any experience using the Ffmpeg libraries either, so I have started a small personal project to learn how it works.

As far as languages are concerned, I am quite fluent with C/C++ and I can handle Python and Lua.

25 Likes

The proposal is still a draft subject to change. I would like, above all, to be more specific in some sections.

Any input or opinion on this matter is welcome.

Thank you.

3 Likes

Thanks for the proposal!

I am not listed mentor, but I would encourage you to go bit more into details (not too much probably)

As I am not super familiar with whole HW decoding, I would have questions like: Ok you freed up the CPU, but is it just waiting now? Or will there be some scheduling system for HW decoding and stuff like that. Also VSE is not the only editor that uses ffmpeg, so would for example compositor be able to use HW decoding?

That is probably all feedback I have for you

… How can people live like this …

1 Like

Thanks for your feedback.
I’m trying to figure out how the two, ffmpeg and vse, relate to each other to see how much it might affect other parts of Blender. It’s difficult to provide answers without a clear understanding of how it’s currently integrated.

Overall looks good! Movie decoding and encoding within Blender is not specific to VSE per se, but rather it is kinda isolated into “movie” folder inside the code. If you’d implement say hardware decoding, then VSE would use it, but also compositor, movie tracking and anything else that goes through that code path. Same for encoding. That is a good thing! i.e. do not try to specifically limit it to just VSE in some way.

Which OS/GPU combinations you have access to, by the way? As you figured out from the hardware acceleration matrix, there is quite some possible combinations out there.

2 Likes

Thank you very much for the comment.
So, I gather that changes to the UI will not be confined to the VSE only.
I have access to Linux/Windows 11, a laptop with Intel/nvidia and a PC with AMD.
With my current knowledge it is difficult to estimate how many combinations I could cover. Perhaps it would be better to ensure the minimum cross-platform support in the GSoC lifetime and try to add the rest of the combinations afterwards.

1 Like

Thanks for the proposal. I wouldn’t worry about all possible permutations and combinations, and indeed starting small with the most common configurations sounds like a good plan.

The thing which is not super clear to me is whether you plan to implement all the APIs from the table. Would be nice if we don’t add all of those APIs to our permutations but instead stick to one, and make sure it is rock-solid. And the UI ideally also should super clear for artists: maybe we can have a single “Use GPU” setting in codec panel (similarly to render denoiser, i.e.).

1 Like

Thank you for the feedback.
Yeah, you’re right. As @aras_p has commented to another participant with the same proposal especially in the case on encoding, where there is an impact on quality, activating acceleration should be an option available to the user at the time of use, conditioning the rest of options if activated.
Regarding the number of options, I agree with you too. Better one functional option per platform, if possible, and add the rest over time.