Here I’ll post weekly reports on my project.
Further links:
I had to do quite some preparations before the actual coding period started:
NV_DX_interop) to share resources between OpenGL and DirectX that seems to be available on common recent hardware. I didn’t manage to do any tests with it, so I don’t know how stable it is nowadays.To use OpenXR, mechanisms to dynamically link to the OpenXR runtime need to be set up. The best way to do this is via an OpenXR loader. Focus for this week was to get OpenXR up and running via the loader provided by the OpenXR SDK.
Also, spending a few great days at the Libre Graphics Meeting. Bit of an eye opening experience: They all seem to look up to Blender. Very motivating 
Heads up: I’m on a camping trip over the weekend. Brought my laptop with me but after mentioned OS upgrade my network config is messed up. So I didn’t manage to add a hotspot connection like I planned. I have some changes to be pushed which are part of last week’s work. Once I’m back on Monday I’ll push them and write a proper report. I also chatted with Dalai (as my mentor) on the progress and issues I’m facing. Nothing worrisome, I expected such problems.
Sorry about the delay of course, these are unusual circumstances.
Focus for the week was working on VR session management, just like planned. This should work as expected now, although testing was difficult for a number of reasons:
No major issues, but annoyances that made testing hard and took time away from coding… Well, it’s bleeding edge software and I expected such hiccups.
I still got enough done to be totally in schedule:
So VR sessions are there now, time to jump into rendering!
Next week is going to be interesting since I’m going to spend time on DirectX compatibility. As outlined in the schedule, this is an important feature but it’s quite a bit of an unknown.
Goal is being able to draw the OpenGL viewport into a DirectX window with minimal overhead.
Worked on DirectX compatibility as per schedule. In short, reason for this is to support the rather important Windows Mixed Reality OpenXR runtime.
This may very well be the first Blender window drawing with DirectX 
:
Finally, it’s time to plug in the HMD and start pushing pixels to it! The next two weeks should be mostly about getting the viewport rendered in the HMD. First, setting up OpenXR swapchains to get framebuffers displayed on the HMD. Then digging into the draw manager to make it render based on HMD input. There’s more to it, e.g. managing spaces and frame syncing, but those are the main points.
Of course, I’m going to work on the mentioned final step for DirectX interoperability too.
I’ve also decided to port the OpenXR access layer to GHOST. That means basically porting the contents of windowmanager/xr/ to C++ GHOST. Couple of reasons for this:
XrGraphicsBindingOpenGLXlibKHR
std::vector for dynamic extension array building rather than messy raw-array violence; state management can be improved through objects; RAII for aborting sessions safely in case of errors; etc.We may still want to keep a small higher-level API wrapped around GHOST_Xr calls.
I will work on this first as it will make some further work much easier.
Exam phase started and hit me hard this week, so very stressful days here. I didn’t get as much done as I hoped to, but again enough to stay in schedule.
Main achievement: Finishing the DirectX compatibility layer (637b803b97, 71c7d61808). This is a Blender DirectX Window rendering to an offscreen OpenGL context, and then sharing the result with DirectX for presenting:
The implementation uses shared resources and GPU side framebuffer blitting. That should be as fast as it can get. I expect virtually no performance overhead. It’s made possible with a WGL extension (WGL_NV_DX_interop) which, although rarely used, should be available on most modern-ish hardware.
Move OpenXR access API from WM to GHOST/C++ (f30fcd6c85, 4eeb7523e9, d15d07c4f6). Reasons outlined in previous report.
Pass low level graphics library data (GLX, WGL, D3D) to OpenXR graphics bindings (9d29373b06, b18f5a6a81, 2ee9c60c20)
Use lazy initialization/creation of OpenXR instances (a8519dbac2). Avoids costs of OS wide XR configuration reading and dynamic linking on startup. Also, don’t pay for what you don’t use!
Continue working on rendering: Set up OpenXR swapchains, query and use spaces from OpenXR, frame timing, etc. Hopefully within one or two weeks I’ll see a viewport render in my HMD.
One question I didn’t think of earlier: When we do the entire VR view rendering offscreen, who owns the viewport region? Not tying this to the usual wmWindowManager->wmWindow->WorkSpace->bScreen chain would open a can of worms. Maybe introduce a virtual window type? Interesting to look into, but not a hard target for the week.
Exams/deadlines will keep me busy for two more weeks. I’m still trying/intending to get the expected 30h per week done though.
Here we go, Blender rendering a viewport to a (in this case simulated) HMD screen!:
. The screenshot also shows a nice benefit from using official runtimes: They can extend the experience as useful. E.g. here Windows inserts a notification into the VR scene.
To my delight, performance seems quite fine for reasonably complex scenes. Further speedups are certainly possible, but it’s a nice start. This basically means the minimum viable product is reached, as defined in my project proposal.
Once again, this has been a fight. Patch needs some cleanup and isn’t committed yet. Note that this uses the DirectX compatibility layer. OpenGL only drawing will need a few more tweaks to work.
Changes done to get there:
wmSurface type and API, as a container to manage non-window drawable surfaces (cf4799e299). Manages OpenGL and GPU context and allows drawing an offscreen viewport outside of the normal window drawing. Means we don’t need to set up the wmWindow → WorkSpace → bScreen → ScrArea → ARegion chain just to draw a viewport.wmSurface type (109be29e42)Also:
As per schedule: Add various debugging/validation tools. This is really needed to make further work easier. I’d also like to work on good error handling (e.g. fail with good user message when no compatible OpenXR runtime is found).
Besides that, rendering needs some polish and OpenGL only drawing needs to be finished.
Due to a bigger assignment deadline on Friday, I didn’t manage to do much over the week. Tried to catch up a bit over the weekend, but outcome is small because of yet some more annoying mysteries I had to solve. On the positive side, the worst part of my exam phase is over now. So starting this week I should have more time and brainpower to spend on this project. I’ll also try to publish all further weekly reports on Fridays.
So there are only few things I’ve done:
I’m still planning to do a nicer demo of the current state of things, for now, you can check the following.
Both videos use headset simulation. Didn’t find a nicer way to screen capture this yet. Lighitng is still too dark.
Continue work from this week (stability, debugging/validation tools, rendering polish). This was planned to take two weeks, so I’m still in schedule.
Worth mentioning that the reason to work on this stuff now is to prepare the project for inital user testing. Users would have a hard time testing if Blender crashed every few clicks. Also, I need ways to debug issues they find.
Worked on stability, debugging and validation tools, as well as some rendering refinements. More concretly:
Errors in OpenXR or GHOST_Xr code will now trigger specific error message popups and cause the VR session to be canceled cleanly (330f062099, 3a2e459f58).
For example this is the error shown if we successfully connected to a OpenXR runtime, but no HMD is found:
Refactored GHOST_XrContext code for consistency and good error handling (c1e9cf00ac).
Check if GPU driver meets requirements of the OpenXR runtime (67d6399da6, 09872df1c7). If not, fail with specific error message.
Allow enabling the OpenXR API validation layer (b98896ddd9).
Requires some set up steps to run. Needs more work to be usable out of the box.
Introduce --debug-xr command line option for verbose debug prints (b216690505, 6e3158cb00).
Enable and use the OpenXR debug extension for --debug-xr (84d609f67b).
It allows creating a messenger through which OpenXR can submit error messages to us.
Although Windows Mixed Reality lists the extension as available, it’s apparently incomplete and causes OpenXR instance creation to fail. Disabled it on Windows for now (99560d9657).
The branch should now be mostly ready for some initial user testing. I’m going to write a page with instructions. However, I’m afraid only the Windows Mixed Reality runtime can be used for testing. Monado is complicated to get running and appears rather unstable.
So with this user feedback, I’m going to continue ironing out bugs and make the branch rock stable.
I still need to address some further things:
Seems like I’ll have to open another report with excuses… I made the huge mistake of thinking I could take a break from exam preparations last week, so this week things struck back: Finishing up an assignment took much more time than expected, leaving me with very little time to prepare me for an exam today. Needless to say, I had to focus on that and neglect GSoC.
I still am in schedule, and I can probably be satisfied with the overall project progress. But I still thought by now I could focus more on the project, less on studies.
Here are the bits I got done regardeless:
Note that you’ll also see me doing some commits to the filebrowser_redesign branch the next days. I’m actually doing this as part of another university assignment. My teacher was very interested in my Blender work, so I managed to squeeze some in
Same as last week, as per schedule.
Okay here we go… I’m done with this semester and can fully commit myself to GSoC - finally! I waited with publishing this report until I crushed some issues with threaded drawing, which I started working on on Friday, but didn’t want to work at all (it does now).
Focus was a) solving performance issues and b) fixing most important issues that block user testing. I’ve also added perfomance debugging utilities.
In this week alone, we went from ~20 FPS to ~43 FPS in my benchmarks with the Classroom scene That’s a rather fine performance for moderately complex scenes. @LazyDodo reported even bigger improvements IIRC.
temp-vr-draw-thread. Performance tests above don’t include this.GPU_offscreen and GPU_viewport on every redraw, greatly improving performance and fixing a troublesome memory leak (this relates to previous point in that it would add more context switches - 55362436d9).--debug-xr-time command line option to print VR session frame render times and FPS info to console (d58eb8d29d, 57b77cd193).filebrowser_redesign branch, together with a university colleague and William Reynish. See T62971. It’s close to being ready for a 2.81 merge.Area.ui_type during area change (D5325).OpenXR 1.0 got released yesterday! The Oculus support is also announced for this week.
Hence, I’ll update our OpenXR loader to version 1.0 and update my How to Test Page.
At this point I’m going to digress from my schedule. The final weeks were kinda fillers anyway, there’s more important stuff to do now.
temp-vr-draw-thread).Short week, I did what I planned to do:
temp-vr-draw-thread branch):
GPU_matrix stacks per GPUContext (D5405). This is needed for thread save GPU_matrix usage.soc-2019-openxr)Need to check with Dalai (who’s at Siggraph currently), if I should prepare the branch for a merge yet. I’d rather spend more time on threaded drawing. This may take a few more days to complete. Besides that, I’ve listed possible tasks for the remaining GSoC time here: GSoC 2019: Core Support of Virtual Reality Headsets through OpenXR.
GPU_matrix stacks per GPUContext (D5405). This is also for thread safety. Since the question of thread local storage performance was raised, I spent some time on carefullly testing this. Reported results in the patch.With my mentor I agreed on getting the branch ready for merge after GSoC. That way, we have an outcome from the project ready to be built on, even if I don’t manage to continue work on it. Hence, I’ll continue work on polish as well as a refactor of the CMake changes I did for the OpenXR SDK dependency.
I may also do some first changes to get the barebones for input & haptics with OpenXR implemented. That way we can soon experiment with VR event input (e.g. controllers and gestures) to get a more informed knowledge of what needs to be done for interactive VR experiences.
Mainly worked on getting the build system side of the branch ready for merging into master, as well as preparing further XR development for after GSoC.
We’re trying to get the file-browser redesign ready for a merge ASAP. So I worked on the remaining bits, and did many small tweaks and fixes. It’s sooo close to being ready!
The final week! I’d like to get the branch into review in this week. The build system related stuff should be ready, I’ll have to go over the changes to see how much polish is needed.
For a merge into master, I don’t think it’s a good idea to expose the VR session toggle by default (which is the only VR related thing exposed in the regular UI right now). VR support is a big topic and much more work is needed. We’d only disappoint users if we showed a VR session toggle in this early state.
We could disable WITH_OPENXR by default, or allow enabling it via a Blender command line argument (e.g. --enable-vr). However, my prefered solution would be to do this via an Add-on that makes the early state of things clear and that is disabled by default.
There’s not much user documentation to be written, given that only a single button was added to the UI and that the VR session doesn’t allow any interaction yet. I might still put down some general information, but I’ll do that in the shape of Add-on documentation.
AFAIK I have to write a final project report though.
If there’s time left, I’d still like to play a bit with input and haptic support.
So that was it, the final week! And with this week passed, I’d consider my project as complete. Which in this case means: All targeted deliverables as per proposal are there, the branch seems to work with all available OpenXR runtimes, things are stable and polished for a merge. Further, the main branch (soc-2019-openxr) is submitted for review, I’ve written documentation on how to test/use the outcome and I’ve done quite some work to kickoff further and continuous development for XR experiences (see T68998).
All that is left to do for GSoC should be writing my final report and submitting my final evaluation.
What I’ve done this week:
soc-2019-openxr branch for review! (D5537)Awesome, I hope XR support with continue