Proposal: Google Benchmark Library

Blender Development
#1

Often I create test cases that are used to measure performance of a certain area. The goal is to isolate a part of code that are important for performance. Using the test time you can get an idea of the performance.

This is far from ideal. as it isn’t clear what these test cases are for by other developers. No comparisons with previous runs. and for each variation you need to add a new test case.

Google benchmark is a library that can provide this. Using this could keep track of performance important parts of the code base in a repeatable and structured way.

My proposal is to add google benchmark to be included in the source tree similar how google test is currently used. See https://github.com/google/benchmark for details about the reporting and ways how to use it.

We currently have benchmark performance test cases for lattice and threading. https://developer.blender.org/D11255 would provide test cases for GPU Batch extraction.

1 Like
#2

I did an experiment on google benchmark to get data out of it.
See https://developer.blender.org/D11401 for the patch.

#3

My personal opinion is that in the big picture, unit test style performance tests are not worth writing and maintaining. And that instead we should put our limited time in performance testing actual operators and .blend files, and the infrastructure required for that.

The library seems fine for what it does, just personally I’d rather not deal with any of this. Other developers may well disagree though, I know my opinion on unit tests is different than some. It just seems to me that with this approach we end up testing a tiny subset of functionality without a realistic path to significant coverage.

1 Like
#4

Usually when i’m doing perf work, i go ‘oh a benchmark framework would be nice!’ but then i remember pil_time.h exists and it covers 99% of my direct needs (ie how long does this thing take, I rarely have the need to track the metric over time/releases)

I do think performance regressions are important enough to have tests for/keep track of, but unless we back that up with some infrastructure (a dashboard / alert system of some kind) it will probably suffer the same fate as our current unit tests, (ie they can fail for days an no-one notices/cares) picking a library (something pil_time based or google benchmark or just sticking the ctest results into something ) seems to be the least difficult/time consuming part here.

I’d (and i’ll be honest by “I” I totally mean someone else) rather spend time on some exploratory work on how improve our CI reporting workflow ,rather than picking a rather low level component at this point in time and figuring out what to do with the data it produces later.

1 Like