I hope this message finds you well. I’ve spent a considerable amount of time with Blender and have always been impressed by its capabilities and the value it brings to the 3D creation community. I’d like to suggest the integration of fuzz-tests to enhance Blender’s security profile. I’m fully aware that the blender team incredibly busy and largely supported by volunteers, so I don’t want to add any unnecessary maintenance or review burden. I’m ready and willing to take on the development work required for integrating google/OSS-Fuzz (continuous fuzz testing suite) with Blender, as well as championing the addition of new fuzz testing harnesses.
OpenSSF and Best Practices for Fuzzing
The Open Source Security Foundation (OpenSSF) emphasizes the importance of fuzz-testing in fortifying open-source projects. Their best practices can offer guidance on enhancing both the security and trustworthiness of software.
Quick Intro to Fuzz-Testing
Fuzz-testing, or simply fuzzing, is a technique that exposes the system to a myriad of random and unpredictable data inputs. Its primary goal is to identify potential vulnerabilities, acting as a rigorous stress-test for code.
Advantages of Google/OSS-Fuzz for Continuous Fuzzing
Here’s a brief overview of how OSS-Fuzz could be beneficial:
Uncovering Hidden Vulnerabilities: OSS-Fuzz is adept at detecting potential threats that might elude standard tests.
Consistent Automated Fuzzing: With its distributed architecture, OSS-Fuzz performs in-depth fuzz-testing regularly.
Cost-Efficient Security Boost: OSS-Fuzz, supported by Google, provides its services without any financial commitment (i.e it’s free for open source).
Comprehensive Reporting: OSS-Fuzz offers detailed reports, aiding in swift issue resolution if discrepancies arise.
Incorporating OSS-Fuzz could be a valuable step in Blender’s continuous evolution. I’m going to follow up with a minimal pull request in the next week so that you can get a basic idea of what this would look like.
I’d love your feedback, and would like to get the go ahead from the Blender team before I sink too much time into it
Random combinatorial testing of geometry nodes could indeed be very useful! A fairly large part of crashes/asserts issues are caused by zero values or simply unexpected nodes.
Great! I’ll add that to my list of potential targets for fuzzing. At the moment I’m working on a fuzzer for the blenloader api as it seems like the low hanging fruit. i.e. it’s easy to fuzz and it is one of the first external inputs that blender will use.
Projects like this require careful coordination and often large amounts of dedicated resources from the project itself so it’s probably best to at least get in touch with @ThomasDinges first to set expectations and to discuss what types of issues would be considered in the first place. He’ll be able to point you to the most up to date resources around the security/reporting policy as well.
No worries, I’ve got a bit of experience doing integrations with google/oss-fuzz. So if there are particular friction-points or questions I’m happy to answer/help. We can keep chatting here or if it’s easier I’m happy to chat realtime
Personally I think that if the goal is to harden .blend file loading, there are more efficient things that can be done. .blend files are a memory dump and the code was never designed with security in mind.
Fuzz testing is handy because it allows you to get started without having to understand the code itself much, since you can just treat it as a black box. But I think it’s mainly going to show random issues while really making this secure requires looking at the code and addressing certain insecure patterns throughout (checking allocation failures, integer overflows, values outside of valid bounds, …).
Hmm I would say the goal isn’t specifically to harden just the blenloader API, moreover the goal is to harden blender in general. I think the blenloader API is just a nice place to create a super simple/stupid fuzz harness as a proof of concept before more involved hardening.
With respect to your concerns about it showing random problems. I think this is true with fuzzing in general. But combined with the dashboard provided by google/oss-fuzz you can filter out crashes that don’t pose a significant security threat and triaging severities is significantly easier.
I also acknowledge that you are right, that actually making the blenloader API more secure is going to require actually looking at the API and identifying problematic design choices. But at least for me the I like the convenience of just letting the fuzzer identify the problematic design choices and then fixing them. I’ll also add that by automatically fuzzing you can prevent new bugs from being introduced into upcoming releases.
There’s also another method of fuzzing called property testing. Which is used to verify the soundness of a given algorithm rather than the security. E.g. you might automatically generate two overlapping cubes, apply a Boolean modifier and assert that the volume before is greater than after.
I dont know much about testing, but I think it’s great to start with nodes and operators. They accept a few explicit input slots, so it may be easier to generate random input for them. I’m looking forward to see what kinds of bugs fuzz testing can find.
Ultimately the purpose here is to make Blender more secure and stable. We are aware of various problematic design choices that require some significant work to fix. So I think the primary thing is to have a developer who is willing to work on that, and for them a fuzzer will be a very useful tool. I’m happy to discuss steps to improve the design of .blend file loading if someone is interested to work on this.
But the other way around probably leads to identifying a random small subset of the problems and fixing them, which will cost time while not making Blender measurably more secure or stable for users.
Integrating this as a proof of concept can be useful, and maybe it motivates someone to start tackling the underlying issues. Just by itself I don’t think it makes sense to integrate into the development process.
Integrating this as a proof of concept can be useful, and maybe it motivates someone to start tackling the underlying issues. Just by itself I don’t think it makes sense to integrate into the development process.
Well I’ll start with that. What is your intuition on what “significant work” looks like in the specific case of the blenloader API?
But the other way around probably leads to identifying a random small subset of the problems and fixing them, which will cost time while not making Blender measurably more secure or stable for users.
So in my mind to demonstrate the value of fuzzing for the blender project, I’d like to;
Write a fuzzer, to identify potential issues.
Integrate this with oss-fuzz so that continuous fuzzing is enabled.
Fix (myself) any issues that come up inside the oss-fuzz disclosure window of 90days.
In the specific case of blenloader API It seems you believe this to be an overly ambitious goal. I’m inclined to go with your intuition here. If you have other suggestions as to different API’s that I should start with that perhaps have a smaller footprint I’d love your input.
I’d settle for just the first point of your plan for your poof of concept (and maybe the 3rd if you find any issues), integration into oss-fuzz may be more than you can chew off as an experiment, as the run-time environment is rather restricted and they aren’t keen on shared libs, with the blender code being somewhat of a circular dependency hellscape, you’ll have a hard-time separating out just blenloader into your fuzzer, like it or not, you’ll end up linking all of blender into it (the same problem applies to our unit tests) and there’s very little that can be done about that.
Also word of caution historically the blender admins haven’t been super keen on using 3rd party services and prefer to run things on premise. Not sure if @brecht has realized oss-fuzz is actually infra google runs on their cloud as this is usually a hard no for them.
so i’d let the oss-fuzz part be for now and just focus on the fuzzer tooling and making a proof of concept that outputs something actionable.
Good to know, I’ve done one integration with oss-fuzz that used dynamic libs. It was a little painful but not as bad as you might imagine. It is worth noting that oss-fuzz itself is a google operated instance of google/clusterfuzz, which is entirely open-source. I’ve never tried to run my own instance as it looks quite involved, but it is possible. It seems like the blender team has been burnt by third party service providers before, so I won’t push the point. However it seems a shame to pass up on something that is entirely free, and open source.
How would things like mesh data be handled? I’m sure that if a file contained a bunch of garbled random mesh indices, and that mesh was used in some procedural setup, Blender would crash somewhere. But as far as I know this has been a purposeful choice, since validating every mesh when reading it would be very expensive.
In other words, how far does this fuzzing go? And is the goal to be able to read and elegantly reject arbitrarily randomized data?
To make .blend file loading itself secure I think we need to at least:
Validate all integer struct members to be in a valid range. Thinking of array sizes and indices being >= 0 and matching the allocated array memory, vertex indices on edges and faces being within range, etc.
Validate meshes topology to avoid things like duplicate edges and face. This can indeed be slow, at least the existing mesh validation function, maybe it can be made faster or doesn’t need to check as much?
Validate that pointers point to the right data type, that certain pointers are not null, that linked list are valid (e.g. not circular).
Ensure strings are zero terminated.
Go over the DNA code and think about which kinds of invalid DNA specifications could be problem.
Have some mechanism to gracefully handle these types of errors in .blend file loading, either by using exceptions and making the code exception safe, or by checking error return values throughout the code.
Check for any cases where integer overflow can happen, often multiplying two integers for allocating memory that can be replaced by specialized array allocation functions that check for this.
Handle failed memory allocations more gracefully throughout Blender, or alternatively abort the application before anything bad can happen.
And when we address that and probably some other things I have forgotten, then I think fuzzing becomes more useful to identify what we have missed.
When we get to that stage, I’m not sure exactly what our policy would be regarding doing this on 3rd party services, I would need to discuss it with others. If it’s critical to our infrastructure we want to run it on premise, but something like this doesn’t really break anything for us when it goes offline or has privacy implications, so maybe it’s ok.
I just thought I’d mention the approach I typically take while writing new software or changing old software. I typically will follow test driven development (TDD), with BOTH fuzz harnesses and unit tests. I’m not going to explain unit-testing here but here are the things that I’d typically try and capture in a test-first fuzzing strategy;
Certain invariant properties specific to the algorithm are maintained for all inputs. e.g. as a toy example let’s say my algorithm was simply to take the average of a vector, I might test some invariant properties e.g. assert(average(&my_vec) >= min(&my_vec)), where my_vec is constructed from fuzzed data. While this sounds like perhaps a silly thing to assert, it might catch edge cases like a vector containing an Inf or a NaN.
That it doesn’t crash with undefined behaviour or address sanitizers (to your point on integer overflow).
Note on the specific blenloader API
It doesn’t solve every problem in that list, but the first thing that comes to mind immediately is to use something like a stream of flatbuffers. It might even be possible if carefully designed to maintain the current structure of the .blend file, while switching over portions of the format to use flatbuffers. That would solve the majority of the validation problems that you mentioned (as that validation is baked in). But I don’t want to get too off topic just yet on the specific implementation details.
Moving forward
In your opinion do you think that the blenloader is a good place to start. Or are there simpler and easier places to get started?
I don’t think that’s really practical. The DNA system is so tightly coupled to many parts of Blender that you’d need to rewrite a lot of code, and then still you’d have to keep both file reading implementations for backwards compatibility.
It depends what the goal is. I think .blend file loading is the most important thing to address regarding security.
But a fuzzer that constructs random scenes, node setups and runs operators using the Python API could help identify cases where a user might run into Blender crashes. For example if you create random geometry node setups and discover real bugs in them, then fixes for those are likely easier to get merged. Because we do already try to make sure that those are handled gracefully, it’s not likely to raise bigger design questions.
