But I’m trying to test across multiple systems that don’t have online access in an automated fashion.
Why does the CLI version need an authentication token, but the GUI version does not?
It makes no sense.
Please remove the authentication requirement from the CLI version - I have no intention of uploading the data (as I have access to NDA systems that can’t submit anyway).
Edit:
So I tried moving all the relevant files and scripting up a benchmark.
The CLI version still wants to connect to get an auth token. Even though I haven’t used the upload flag.
Then the benchmark won’t run, even using the files copied from a different system, because…
ERROR: Blender version 2.81a seems to be broken. Consider running the ‘clear_cache’ command.
ERROR: Blender version 2.81a with checksum 87355b0a81d48ea336948294b9da8670eaae73667fae028e9a64cbb4104ceea1 is broken: missing valid flag file
This all requires to be online, obviously. I can’t script what appears to be a manual process.
So that’s a negative on being able to use the benchmark on AnandTech
Hello, thank you for the Blender Benchmark and Open Data project!
My feedback about the website:
I suggest to separate/indicate the “Device Type” GPU in the Fastest GPU Graph.
I suppose that it is a median of the all GPU “device type” with probably a different “device type” distribution/mix for each GPU ( in this way slower GPU could appear faster in the Graph).
I read a lot of confusion recently in the social media about it
I would extend the Fastest CPU and GPU Graphs to the top 50/100 devices ( with dedicated pages or a Popup ?)
I would add the “Victor” Benchmark in the homepage “Top 50 Devices” ( i can see “Victor” only in the filtered “Search data” ) and separate/indicate the GPU “Device Type” here too.
In the filtered “Search data” i would sort the lowest Median Render time as default in the search result.
In the “Operating system distribution” and “Device type distribution” i would add the % beside numbers.
I would filter out the “impossible spike data” like an old mobile GPU faster than any device ( example Victor Median Render time)
I’m not sure if this is the right place to post this. If there is a better place too, please let me know and I will go post there.
I am trying to process the benchmark data however I can’t seem to figure out how the “mean render time” for a device is calculated.
Take the Radeon RX 5500 XT for example,
According to the site, there are 8 benchmarks with a median render time of 457.895s
Going through the individual scenes,
2 benchmarks for the scene Barbershop Interior, with an average time of 629.234s = avg(585.373,673.095)
1 benchmark for the scene BMW27, with a time of 160.604s
1 benchmark for the scene Classroom, with a time of 330.417s
1 benchmark for the scene Fishy Cat, with a time of 240.533s
1 benchmark for the scene Koro, with a time of 166.770s
1 benchmark for the scene Pavilion Barcelona, with a time of 895.54s
1 benchmark for the scene Victor, with a time of 1758.630s
There are 8 benchmarks however I can’t figure out how to calculate the 457.895s figure.
Averaging them doesn’t work —> 597.390
Using a weighted average considering the first scene has two benchmarks, doesn’t work —> 601.370
Using the middle 5 benchmarks doesn’t work-or any combination of benchmarks —> 358.773
Averaging the average render time for each of the two benchmark sessions doesn’t work—>632.109
I wrote a program to test all linear combinations of the render times using fractional constants from 0, 1/2, 1/3, … to 1/10
The cleanest result was (160.604+240.533+166.77+895.54)/5 + (330.417)/2 = 457.898
Still inaccurate and convoluted.
I went through the source code and found the file I believe is responsible for calculating the median render time. The relevant part starts at line 51. The sql is a bit too complex for me to understand. I see it is using the continuous percentile function a lot however I can’t see how it all fits together. Could anyone help me figure out the formula used?
i was looking for some viewport benchmarking and found absolutely nothing for blender 2.8. is it planned at some point to include it in this benchmarking package? rendering performance is nice and all but it would be nice to have some "working condition: benchmarks to get an idea how certain hardware performs.
Good Day. I was trying to run the latest download of the Blender Benchmark and it immediately crashes upon startup. I.e i get a flash of a window that quickly disappears.
It is an older system, but still does run blender and i even rendered a couple of the scenes using the full blender application.
i have tried running as administrator and also deactivating the Antivirus software.
Hi.
I was wondering about barbershop_interior scene. By default scene (CUDA/OpenCL) uses Branched Path Tracing. OptiX does not support BPT. So is the benchmark using a modified scene using Path Tracing to make more reliable results when comparing OptiX vs other device types?
For reliable benchmark results, one would like the scenes to be configured with similar supported features by all device types. Otherwise, side-by-side comparisons are meaningless.
EDIT:
In short, I think barbershop_interior scene should be modified to use PT and delete old render times results, or remove OptiX results for that scene. This makes the benchmark less reliable and can give some confusion:
i made a proof of concept for a viewport benchmark, since we usually do not spend time on our systems while rendering and its great to see how long, or in this case short our breaks/waiting time will be. it would also be great to have a benchmark to tell the users how well his system is actually performing while using it.
Therefore i propose to add a benchmark system that can measure different performance for different work done in blender, like sculpting resolutions, painting performance, maybe animations and so on. the possibilities are endless^^
Blender benchmark 2.90. Scene victor crashes stating something like missing JSON data. I cleared the cache and tried two more times with same error message. Scene victor working with 2.83.
I think the site for each benchmark (ex. https://opendata.blender.org/benchmarks/4796cd6c-6dca-4955-bd2f-510be48e29ee/) should list more info because right now it only lists device name. It should list at least the date created, which blender version, which CPU (if different from device), which OS, because right now the page gives very little info.
I know this info exists because I can search for all of it with the raw json data ex. jq 'select(.id=="4796cd6c-6dca-4955-bd2f-510be48e29ee")' opendata-2021-05-13-064037+0000.jsonl
I’m guessing whatever cat /proc/cpuinfo says. The actual clock speed most likely varies over time too so it would be hard to get a single number. Unless the benchmark is actually recording clock speed then it doesn’t have that info.
The overall trend is nearly impossible to see because of some extremely slow systems that took 94 hours to render. Have a hard cutoff at 3 hours or use a log scale (which I think is not as useful)
I cannot complete benchmark Victor on my Windows machine with Blender 2.92 and an old GTX 650 Ti. I had the same issue last year with Blender 2.82 for both my GPU and CPU. Seems like it got fixed for CPU.
Error ID: 438abbd1fc4744dba7b45bfc4d607dc5
same for scene victor on my Quadro M1200 when trying to run OPTIX (Ubuntu 20.04, blender 2.92)
I froze display when Victor started so tried again after forced power down and reboot without Victor and submitted the results.
I went back and ran Task Manager while only Victor was selected and watched it struggle but it did complete to get a token.
When I went to verify token the system was still acting frozen but it did copy a link to clipboard.
I was able to get into browser to verify but it got less responsive again so did another power cycle.
Now I have a token but without submitted results.
Is there a way to recover the Victor results to submit for existing token without the frustration of restarting ?
Hello, I just have a quick question. I was wondering where the benchmarks (all the 6 files: bmw27, classroom, etc.) were downloaded to.
I tried searching online to find out and also looked around on my computer but could not locate the ~600MB worth of files downloaded for the Benchmark test.
I did my benchmark test on a computer running macOS Big Sur (11.4). Thank you for any help in locating these files.
I apologize I was not quite clear in my post, what I meant is the local location of the files on a computer running macOS. I have run the benchmark test once and in doing so I have downloaded onto my computer all 6 files. Now I would just like to know where those files reside locally on my computer. I cannot locate them. Thank you for your help.
