When the 2.8 beta status was declared I decided to try to port some add-ons I maintain to 2.80. In the beta phase, the add-on API has mostly stabilized, but there’s still occasional changes that can make add-ons misbehave. So to make sure add-ons work, I often like to test them against the most recent version of Blender’s code.
There are two options to get the latest and greatest updates to Blender: downloading the source code and compiling it yourself, or downloading one of the daily Blender buildbot releases. While compiling Blender from source is fairly straightforward, I’ve found downloading daily builds to be easier.
After downloading daily builds, sometimes I compare them against previous releases with a diff tool to see exactly what changed. During these comparisons I’ve noticed a large number of files inside two daily releases will be byte-to-byte identical. Seeing all the duplicate files, I was curious how much bandwidth could be saved if just downloading the files that changed was an option instead of downloading an entire build.
To investigate this I compared these 2 daily builds I had on my hard drive against one another:
blender-2.80.0-git-dc3b5024be1a-windows64 (2019-01-23) Directory info: 4,824 files, 498 directories Total build size: Compressed: 119 MiB (125,167,090 bytes) Uncompressed: 329 MiB (345,067,650 bytes) blender-2.80.0-git.a1ae04d15a9f-windows64 (2019-01-30) Directory info: 4,820 Files, 498 directories Total build size: Compressed: 119 MiB (125,176,390 bytes) Uncompressed: 329 MiB (345,072,704 bytes)
To make sure there was a fair number of changes between the builds I used daily builds released roughly a week apart.
With daily builds, it’s usually just the Blender executable and scripts directory that will be updated. This was the case with the above two builds. Here is how the executable and scripts directory from the two builds compare:
blender-2.80.0-git-dc3b5024be1a-windows64 Blender executable size: Compressed: 30 MiB (31,514,665 bytes) Uncompressed: 81 MiB (85,029,376 bytes) Scripts directory size: Compressed: 30.7 MiB (32,224,547 bytes) Uncompressed: 86.8 MiB (91,072,440 bytes) blender-2.80.0-git.a1ae04d15a9f-windows64 Blender executable size: Uncompressed: 81.1 MiB (85,066,752 bytes) Compressed: 30.0 MiB 31,529,457 bytes) Scripts directory size: Uncompressed: 86.8 MiB (91,040,118 bytes) Compressed: 30.7 MiB (32,220,885 bytes)
The Blender executable and scripts directory usually take up around 160 MiB uncompressed, but they can be squeezed them down to around 60 MiB by zipping them. 60 MiB a substantial reduction from the usual 120 MiB file size for the windows daily releases, but there are better ways to reduce download size.
To really save on file size and bandwidth, file patching is the way to go. File patching is basically updating software to a newer version by applying a patch containing only the bytes in a file that changed instead of downloading a whole file. This practice is very common with operating system updates, video games, web browsers, and other software, but not so much with Blender releases. So I wanted to see how much benefit file patching could provide.
As this was a just for testing, I used the first open source option I could find with a directory patching option. In this case that option was “xdelta3” with the aid of the “xdelta3-dir-patcher” python script. With this setup creating and applying patches has to be done from the command line. To create a patch (or “diff”) from the Blender builds, this was the command I used:
python C:\patch_test\xdelta3-dir-patcher.py diff C:\patch_test\blender-2.80.0-git-dc3b5024be1a-windows64 C:\patch_test\blender-2.80.0-git-a1ae04d15a9f-windows64 patch_dc3b5024be1a_to_a1ae04d15a9f.tar.gz
The resulting patch file…
patch_dc3b5024be1a_to_a1ae04d15a9f.tar.gz 13.0 MiB (13,677,202 bytes)
Yes, the patch is an impressive 13 MiB in size. That means if this patch was available online, it would be possible to update from the “dc3b5024be1a” build to the “a1ae04d15a9f” build just by downloading the 13 MiB file and applying the patch instead of having to download the full 119 MiB daily build (close to a 90% reduction in file size). This is obviously a huge bandwidth saver.
To verify the patch works, I applied the newly generated patch back to the older build directory:
python C:\patch_test\xdelta3-dir-patcher.py apply C:\patch_test\blender-2.80.0-git-dc3b5024be1a-windows64 patch_dc3b5024be1a_to_a1ae04d15a9f.tar.gz C:\patch_test\new-a1ae04d15a9f-dir --ignore-euid
The newly created new-a1ae04d15a9f-dir was byte-to-byte identical to the contents of the “blender-2.80.0-git-a1ae04d15a9f-windows64” daily build.
So this raises the question, could this setup be used against the current Blender daily builds? It’s possible, but there are several caveats that are worth mentioning:
- This could likely be done directly with buildbot, but generating patches for daily releases after building them would create some additional overhead
- Xdelta3 stores some username information in file patches which may not always be desirable
- the xdelta3-dir-patcher script does not currently support patching over date and time info on Windows (which is why the
--ignore-euidoption was used above). This means Windows users would have all the files in the “patched” directory have their creation time set to the time of the time the patch was applied instead of when the updated build was released.
- There are more robust (but more CPU intensive) patching options available like Google’s Courgette (these weren’t tested as I was unable to find an “off-the-shelf” directory patching option)
- Significantly reducing the size of updates will certainly lower bandwidth consumption for people downloading Blender, but it may actually increase bandwidth consumption on the other end because of something called Jevons Paradox
- As a counterpoint to the above, patching could also lead to code updates getting more end-user testing if it led to end-users updating daily builds more often