@ThinkingPolygons I feel the same and it’s a big pain that keeps me away from doing any serious stuff in Blender, since I often work with very detailed architectural models.
I am not sure if this would be part of this GSoC project, but may be it is somehow related. A few years ago I tried to import large generated scenes into Blender for further processing and animation. I even wrote my own Python importer for scenes consisting of primitives (like boxes, cylinders etc.) and triangular meshes (stored in obj files) with PBR materials which were quite new back then. However, I lost two months just to realize that Blender kept loading my large scene with a single thread running for more than a day. Just to compare - loading the scene in Mitsuba CLI took around a minute. I couldn’t believe how Blender can be so slow. Someone suggested me to merge the objects by materials. Well easy said if you don’t have hundreds of unique materials and if you don’t want to animate the objects. So merging was not an option for me. Then, I got feedback from the developers saying that Blender has deep in its core an update mechanism that updates the whole scene after each object addition. This update iterates over all existing objects, resulting in quadratic complexity! I started feeling it with around 2000 objects, it became a pain at around 5000 and with more than 10K it was already impossible. I needed to reach 50K at least. I tried to mitigate by hierarchical merging of scene parts into .blend files and merging them together, without success… I was searching for a scenario or setup that would allow to load a batch of objects without any scene updates in between, but I failed. There was no way to postpone the internal scene updates until all objects have been loaded through the Python API. Since then, I tried loading larger scenes now and then a few times, most recently two weeks ago and it is still a pain (loading 300Mb of data took 2.5 hours). The inevitable scene update call is still somewhere there being called.
Blender improved a lot over the years and there is awesome new experimental stuff like Spectral Cycles going on, but import of highly complex scenes remains one of the last barriers for using it on a regular basis, at least for me.
Sorry for the long story. @ankitm You may crash into the same issue as me, just to let you know. I hope you will be able to create a glorious workaround. I wish you all the best with the GSoC project! If you succeed, Blender will finally become my tool of choice for any project.
