Blender Benchmark Launcher CLI and Documentation?

Hi there, Happy New Year! I’ve downloaded the new blender benchmark, which per the website also has a CLI version. When I run it, the process seems interactive. Is there any way to batch the process? Where does it save the data if I don’t want to upload it?

There’s an option to dump it out to the console in json see the output of

benchmark-launcher-cli --help for general help

The Blender Open Data Benchmark launcher command line interface

  benchmark-launcher-cli [flags]
  benchmark-launcher-cli [command]

Available Commands:
  authenticate Request a new authentication token
  benchmark    Benchmark scenes
  blender      Commands for managing Blender versions
  clear_cache  Removes all downloaded assets
  devices      List the devices which can be benchmarked
  help         Help about any command
  interactive  Run the launcher in interactive mode (the default command)
  scenes       Commands for managing scenes

      --browser           open the verification URL in the browser
  -h, --help              help for benchmark-launcher-cli
  -v, --verbosity uint8   control verbosity (0 = no logs, 1 = error logs (default), 2 = debug logs, 3 = dump blender output) (default 1)
      --version           version for benchmark-launcher-cli

Use "benchmark-launcher-cli [command] --help" for more information about a command.

the json output is in the benchmark command

Benchmark scenes

  benchmark-launcher-cli benchmark [scenes, ...] [flags]

  -b, --blender-version string   the Blender version to benchmark
      --device-name string       the name of the device to benchmark
      --device-type string       the type of the device to benchmark
  -h, --help                     help for benchmark
      --json                     dump json output to stdout
      --submit                   submit the benchmark to the Blender Open Data website

Global Flags:
  -v, --verbosity uint8   control verbosity (0 = no logs, 1 = error logs (default), 2 = debug logs, 3 = dump blender output) (default 1)

Hi there, when I do this:

benchmark-launcher-cli benchmark bmw27 --blender-version 2.81a --device-type CPU --json --verbosity 3

I get some errors:

2020/01/10 01:41:41 metadata.go:13: Fetching Metadata:
2020/01/10 01:41:42 metadata.go:48: Successfully fetched metadata
2020/01/10 01:41:42 metadata.go:70: Getting current launcher.
2020/01/10 01:41:42 checksum.go:13: Calculating checksum of current launcher.
2020/01/10 01:41:42 checksum.go:32: Launcher checksum: 88e5b9f3da9abce857347217f8999b93a9ae2bec72f9c5c7f74d2476054ecf9b.
ERROR: Blender version 2.81a seems to be broken. Consider running the 'clear_cache' command.
ERROR: Blender version 2.81a with checksum 87355b0a81d48ea336948294b9da8670eaae73667fae028e9a64cbb4104ceea1 is broken: missing valid flag file

Is there something else I need to do?

that command seems to run flawlessly for me

I only unzipped the CLI version, should I have anything else installed on the machine?

EDIT: Okay, so it looks like I need to run the tool once and answer some prompts first, and then the command invocation above works. Is there any way to get this to work without doing the interactive CLI step?

Just to bump this. A CLI version needs to be true CLI.
The benchmark needs to be offered as a standalone, not a run-and-download.
It’d be nice if we could choose where to install it - but a true standalone fixes that need.

Edit: Also, we need the authentication requirement to be removed. I’m running the benchmark across multiple systems not connected to the internet, with no intention of uploading results (given that I have access to NDA systems). The GUI version of the benchmark doesn’t seem to have the same authentication requirement, so I’m scratching my head as to why the Windows CLI version does.

I’ve downloaded the source and will work on building/debugging this issue. No timeline yet. How are issues logged and assigned when it comes to Blender?

Task on Manifest, and then claim it using the comment box, then a task on Differential.

Logged as:

1 Like

I just did a sync and rebuild of the launcher, and I’m seeing something odd. When I try to run it, I get:

ERROR: fetching metadata failed: current launcher not supported

Any ideas why?

Just a follow up on this issue. It should be addressed with the latest version. Someone beat me to it. There’s an accompanying readme that explains how to use the command to get the version of blender, download the data you need, and run the benchmarks. Note that there’s an issue with running it on a machine with no internet access, so that’s something that needs looking into. I’ll start a new thread on that one.

EDIT: Topic logged here: Blender Benchmark CLI won't run if no access to Internet?

Is there a way to write the total render time to a specific file?

You could redirect the output to a file and see it at the bottom, or run the render via Python and parse the output as it gets … outputted.

So Im currently running the blender benchmark cli through a subprocess.Popen() in my python script.

Im using the communicate() option to save the output. But for some reason, it can never grab the timing on the very last line of the output. Ive tried printing the output to the console to check it, and for some reason, it never captures all the output lines.

Ive also tried debugging by just running the command manually, and the render takes about 20 mins to complete. However, I noticed around the 18min mark, it just stops running. Please help.

This is what I do:

                command = [BLENDER_LOCATION,
            "--engine", "CYCLES",

            command.extend(["--render-format", "PNG", "--render-output", "//{}_render_".format(currFile), "--render-frame", "1"])

            process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)

            # Keep reading status while Blender is alive.
            while True:
                line = process.stdout.readline()
                if line == b"" and process.poll() is not None:
                line = line.decode().strip()
                if line == "":

Mind you I time the whole thing, but this does grab the last line, at least for me. The source for the blender benchmark is also a decent resource (it does this too if I remember correctly). Does that help?