Hi. I’m interested in joining GSoC at Blender this year. Here’s my proposal on Google Docs. Please feel free to add comments on the Google Docs to let me know where and what I can improve in the proposal.
Thank you!
3 Likes
Thanks for the proposal! It is very detailed, and while I am not assigned as mentor for this one, One thing caught my attention - you have mentioned, that memory management needs to be optimized, I assume this is to prevent GPU->CPU texture copy if unnecessary?
VSE has CPU only cache, and I don’t think, that it would make sense to introduce GPU cache during this project. On the other hand, if you don’t need to copy to CPU, there is good chance, that you don’t need cache at all.
But that said, there are also other areas, that use ffmpeg, which already work with GPU textures, so it would make sense to provide interface to just get GPU textures.
I don’t think this should be discussed in the proposal, but not sure if you thought about things like this.
1 Like
Also forgot to mention, if you have contributed code to Blender, make sure you share this in the proposal, as it is almost a requirement, since we would like to have means to judge your skills, especially when it comes to navigating in large codebases.
1 Like
Overall looks good; I think you are likely to run into some unforeseen challenges while doing this (but it is hard to know/guess what these challenges will be). One possible complication is that ffmpeg hardware acceleration bits are differrent depending on both the operating system (Windows/Mac/Linux) and the GPU. It might help to spell out in the proposal doc which OS/GPU you will be developing on, or have access to.
For decoding with hardware acceleration, it is likely that “just using it” when available makes sense. Perhaps with some heuristic, e.g. do not try to use hardware acceleration on videos that are very small in resolution (since the cost of GPU->CPU transfer likely will be higher than just decoding on the CPU). But having an option in preferences, at least for debugging or troubleshooting, indeed makes sense.
For encoding with hardware, I think it has to be an option under ffmpeg output settings. Primarily because hardware encoders do not produce the same quality result as CPU encoders do. So it has to be user choice; whether they need best possible quality (in which case they would use CPU), or whether they prefer potentially slightly worse result but much faster encode (GPU).
And, both for decoding and encoding, whether you can use hardware depends on GPU, driver version, ffmpeg codec type, used pixel format (with or without alpha; 8 vs 10 vs 12 bits per pixel) etc. ffmpeg APIs should be able to return a result of “whether this codec/pixel format can be hardware accelerated”, and yes if it cannot then everything should fall back to existing CPU based code path.
1 Like
Thanks for the feedback, I appreciate you looking into the details. Regarding memory management, you’re right, the goal is to minimize unnecessary GPU-to-CPU transfers where possible. I understand the VSE currently uses a CPU-only cache, and introducing a full GPU cache is likely outside the scope of this initial project. My primary focus will be integrating the hardware decoding and encoding first, which, within the current VSE pipeline, will often necessitate GPU-to-CPU transfers for processing effects or when CPU cache is used, but I will aim to make these transfers efficient. I did consider that other Blender areas work directly with GPU textures, and providing an interface to pass GPU textures directly from FFmpeg where applicable would indeed be a valuable enhancement, perhaps for future integration or if time permits after the core goals are met. For now, the priority is establishing the core hardware acceleration functionality within the VSE’s existing structure.
Okay, thank you for pointing that out; I appreciate you highlighting the importance of prior contributions. I understand that contributing code is crucial for demonstrating the ability to navigate a large codebase like Blender’s. While I haven’t submitted any code contributions yet, I have already successfully built the project, set up my development environment, and am actively spending time familiarizing myself with the source code structure. I am currently looking for a good first issue to tackle to get started. Since my project focuses on the VSE, do you happen to know of any suitable beginner-friendly issues related to the VSE that I could potentially work on to help demonstrate my skills?
Thank you very much for the insightful feedback! I understand the complexity with varying OS/GPU/codec support and the likelihood of challenges.
To ensure broad testing, I have access to a laptop with an Intel GPU and a PC with an AMD GPU (both dual-booting Windows and Linux) . I can also borrow a MacBook to test the macOS/VideoToolbox implementation . While I don’t currently own NVIDIA hardware, I plan to acquire or borrow an NVIDIA system to test the NVDEC/NVENC implementation as well, covering the major hardware acceleration APIs across platforms.
For decoding, I agree: enable by default when supported, investigate the low-res heuristic, and keep the preference toggle for debugging.
Crucially, for encoding, you’re right. it must be an explicit user choice in the Render Output settings due to the quality/speed trade-off, not a global preference. I’ll adjust the UI plan accordingly.
A robust fallback to CPU processing when hardware acceleration isn’t available or fails is a core part of the plan, using FFmpeg’s capability checks.
Thank you again for this guidance. It definitely helps refine the project’s approach and manage expectations realistically.
For VSE these are usually fixed right away, but feel free to check issues with “good first issue” label: Issues - blender - Blender Projects
1 Like