Final Report
As the Google Summer of Code is coming to an end after a 12-week period, it might be a good time to look back on my journey and reflect on what I’ve accomplished so far. It is one of the most memorable summers over years, and I still can’t believe that I’m now well-equipped to contribute to Blender, my favorite open-source software I’ve been using since high school, with which I’ve made some astonishingly vivid renderings for my high school graduation ceremony.
A Bit of Context
The main objective of this project is to enhance the distributed rendering and task execution capabilities within Flamenco through several key improvements.
Firstly, we introduce the ability to pause jobs and submit them in a paused state, providing users with increased control over their rendering workflow and resource allocation. This feature will be particularly advantageous during peak usage periods or when prioritizing specific tasks.
Furthermore, we address the challenge of distributed rendering of single images by adopting a tile-based rendering approach. This method ensures more efficient utilization of computational resources across nodes, minimizing memory usage and avoiding artifacts caused by boundary dependencies.
The full original proposal could be found here.
Current State of Deliverables
Support Pausing Jobs (PR)
I have implemented the ability for users to pause jobs in Flamenco, introducing a new paused state with relevant status transition logic. The frontend has been updated to allow users to pause a job, and comprehensive unit tests have been added to ensure the new functionality works correctly with existing systems. This feature has been fully implemented, reviewed, and the pull request has been merged into the main codebase.
Allow Jobs to be Submitted in paused Status (PR 1 and PR 2)
I’ve made necessary changes to the backend, including modifications to OpenAPI and the job compiler. On the frontend, I’ve added a “Submit As Paused” checkbox to the add-on interface. A new initial_status field has been introduced in the job description to support this functionality. Like the first deliverable, this feature has been fully implemented and integrated into the main codebase.
Distributed Rendering of Single Images (PR)
I’ve developed a working pipeline for rendering an image in multiple pieces and merging them together. This involves a custom JOB_TYPE that generates border rendering tasks based on the specified render tile size. The solution supports user-defined tile size on each axis and adaptive sampling by doing a 16-pixel overscan for each individual tile. I’ve also implemented a complex node structure to merge rendered pieces while preserving the user’s compositing setup.
Here is a demo showing the difference between the distributed rendered image and the image rendered as a whole.
Dreams and Future Plans
While the core functionality is in place, I’m still addressing some challenges. These include adding support for denoising across merged layers and optimizing the integration of Python scripts within the job type. Currently my solution does not support denoising after individual tiles have been merged, and the solution is passing in a long Python script as a command line parameter, which will ultimately hit a length limit. The ideal solution for these two challenges would involve a much greater scope. Changes will be made in Cycles to expose API endpoints for outputting the raw data for denoising, including denoising albedo, denoising normal, and denoising depth. The job compiler in Flamenco would also involve some refactoring to allow custom job types to be able to use the fs JavaScript library and generate Python scripts based on a config file. I also need to fix the limitation that tile sizes cannot lead to non-integer translations. I’ll learn about how the compositor does rounding optimization and support that in this job type.
A few minor things are also worth paying attention to. As more tests are conducted for the single image job type, I realized that a lot of Blender features require manual support in order for this job type to be fully put into production, which needs some community effort. I’ll be around for a long time at least to fix relevant issues and add features of high demands.
Final Words
There are more takeaways than what I could describe with words, and here are some of them. First, I’d like to express my greatest gratitude towards my mentor Dr. Sybren. We’ve been meeting weekly despite of the time difference, and Sybren is always supportive to unblock me from challenges and help me stick to the right track.
This project has also made open-source contributions into one of my habits. The experience demystifies open-source development. It is the hundreds of millions of developers and user communities contributing tirelessly with passion that makes up of the vibrant open-source culture and countless production-level software. I’m so honored to start this journey with Blender, a software that has a huge impact on myself, and I’m aspired to continue this habit beyond Flamenco, beyond Blender.