Blender Benchmark Launcher CLI and Documentation?

Hi there, Happy New Year! I’ve downloaded the new blender benchmark, which per the website also has a CLI version. When I run it, the process seems interactive. Is there any way to batch the process? Where does it save the data if I don’t want to upload it?

There’s an option to dump it out to the console in json see the output of

benchmark-launcher-cli --help for general help


The Blender Open Data Benchmark launcher command line interface

Usage:
  benchmark-launcher-cli [flags]
  benchmark-launcher-cli [command]

Available Commands:
  authenticate Request a new authentication token
  benchmark    Benchmark scenes
  blender      Commands for managing Blender versions
  clear_cache  Removes all downloaded assets
  devices      List the devices which can be benchmarked
  help         Help about any command
  interactive  Run the launcher in interactive mode (the default command)
  scenes       Commands for managing scenes

Flags:
      --browser           open the verification URL in the browser
  -h, --help              help for benchmark-launcher-cli
  -v, --verbosity uint8   control verbosity (0 = no logs, 1 = error logs (default), 2 = debug logs, 3 = dump blender output) (default 1)
      --version           version for benchmark-launcher-cli

Use "benchmark-launcher-cli [command] --help" for more information about a command.

the json output is in the benchmark command

Benchmark scenes

Usage:
  benchmark-launcher-cli benchmark [scenes, ...] [flags]

Flags:
  -b, --blender-version string   the Blender version to benchmark
      --device-name string       the name of the device to benchmark
      --device-type string       the type of the device to benchmark
  -h, --help                     help for benchmark
      --json                     dump json output to stdout
      --submit                   submit the benchmark to the Blender Open Data website

Global Flags:
  -v, --verbosity uint8   control verbosity (0 = no logs, 1 = error logs (default), 2 = debug logs, 3 = dump blender output) (default 1)

Hi there, when I do this:

benchmark-launcher-cli benchmark bmw27 --blender-version 2.81a --device-type CPU --json --verbosity 3

I get some errors:

2020/01/10 01:41:41 metadata.go:13: Fetching Metadata: https://opendata.blender.org/benchmarks/metadata/
2020/01/10 01:41:42 metadata.go:48: Successfully fetched metadata
2020/01/10 01:41:42 metadata.go:70: Getting current launcher.
2020/01/10 01:41:42 checksum.go:13: Calculating checksum of current launcher.
2020/01/10 01:41:42 checksum.go:32: Launcher checksum: 88e5b9f3da9abce857347217f8999b93a9ae2bec72f9c5c7f74d2476054ecf9b.
ERROR: Blender version 2.81a seems to be broken. Consider running the 'clear_cache' command.
ERROR: Blender version 2.81a with checksum 87355b0a81d48ea336948294b9da8670eaae73667fae028e9a64cbb4104ceea1 is broken: missing valid flag file

Is there something else I need to do?

that command seems to run flawlessly for me

I only unzipped the CLI version, should I have anything else installed on the machine?

EDIT: Okay, so it looks like I need to run the tool once and answer some prompts first, and then the command invocation above works. Is there any way to get this to work without doing the interactive CLI step?

Just to bump this. A CLI version needs to be true CLI.
The benchmark needs to be offered as a standalone, not a run-and-download.
It’d be nice if we could choose where to install it - but a true standalone fixes that need.

Edit: Also, we need the authentication requirement to be removed. I’m running the benchmark across multiple systems not connected to the internet, with no intention of uploading results (given that I have access to NDA systems). The GUI version of the benchmark doesn’t seem to have the same authentication requirement, so I’m scratching my head as to why the Windows CLI version does.

I’ve downloaded the source and will work on building/debugging this issue. No timeline yet. How are issues logged and assigned when it comes to Blender?

Task on Manifest, and then claim it using the comment box, then a task on Differential.
https://wiki.blender.org/wiki/Process/Contributing_Code

Logged as: https://developer.blender.org/T73421

1 Like

I just did a sync and rebuild of the launcher, and I’m seeing something odd. When I try to run it, I get:

ERROR: fetching metadata failed: current launcher not supported

Any ideas why?

Just a follow up on this issue. It should be addressed with the latest version. Someone beat me to it. There’s an accompanying readme that explains how to use the command to get the version of blender, download the data you need, and run the benchmarks. Note that there’s an issue with running it on a machine with no internet access, so that’s something that needs looking into. I’ll start a new thread on that one.

EDIT: Topic logged here: Blender Benchmark CLI won't run if no access to Internet?