Currently, roboticists and AI researchers are still struggling with lots of basic challenges to do simple manipulation tasks and enable a robot/manipulator to grab objects properly. However, they are also be looking for tools that provide realistic renderings and physics simulations and enable the robot in simulation to transfer its capabilities to the physical world in a seamless manner.
Current popular frameworks such as MuJoCo that many roboticists use for implementing their AI models for manipulation and planning tasks lack the notion of ‘material’ the way Blender has. That’s one of the main reasons why these frameworks are unable to provide realistic renderings in addition to having a good rendering pipeline (EEVEE in Blender 2.8+). Even if those frameworks did have a good system for making BPR materials, they would probably be still unable to provide realistic renderings in REAL TIME.
A framework like MuJoCo is able to give real-time physics simulation which is unrealistic as the parameters in the physics engine have been highly optimized for these purposes; it provides very simple notion of materials and also unrealistic renderings in addition to a good interactive mode.
I personally think Blender 2.8 could potentially be an ideal framework for roboticists to model their robots and enable them to interact with the world in a more realistic way that is not possible through other frameworks. Currently, Blender 2.8 lacks an interactive mode, its high-quality physics simulations are pretty slow but it provides real-time, realistic renderings. Although Blender is not designed for these purposes but with this post, I just wanted to bring this opportunity to Blender developers’ attention, and hopefully to Ton as well, that incorporating the capabilities that I mentioned above into Blender will potentially make it an ideal place for roboticists to use Blender as their main framework for their research.
I know the new part-time hire will start working on the interactive mode for Blender 2.8 soon and that’s why I decided to write this before development of the interactive mode begins. In addition, I have a strong feeling that Blender 2.8 could potentially become the ideal framework for roboticists as it already has a real-time rendering system and will soon have an interactive mode. So I hope the Blender Institute will invest more in developing the interactive mode to make it usable for roboticists as well. Of course, the ideal interactive mode for roboticists should not have the limitations of a framework like MuJoCo which I think Blender 2.8 has the potential to fill in the current gaps; mainly it should have more realistic and real-time physics simulations. I am also aware that real-time and realistic physics simulation requires a lot of computation but I hope there will be new solutions for that as well (or at least physical simulations can be improved in some way compared to MuJoCo).
In my opinion, in the long-run, this investment can also open doors to many new opportunities for Blender organization. It wouldn’t be hard to imagine that once Blender has a good, usable interactive mode for roboticists many companies will be willing to invest in developing tools based on Blender and many new developers will start working on Blender’s code and improve it’s capabilities and make it more suitable for their robotics or AI research.