Hello, I’m Hugo, and I’ll be working on this proposal. Essentially, this project will make the OBJ, STL and PLY importers and exporters faster, keeping the same functionality. Most of the code will be written in C++, make use of memory mapped files and try to be generic, while still clear and performant.
I’m not entirely sure if this is the correct place to ask this, but I’m having some questions about materials. @brecht or someone else’s help would be rather appreciated. It seems that the current OBJ exporter essentially uses the old BI properties, which from my understanding correspond to the preview shading in newer versions. However, there are only three of these now: diffuse color, metallic and roughness. I think it’d be more useful to instead export the values of the principled BSDF that’s closest to the output node, as not much else can be done about complex materials, from my understanding. Not sure if anyone even uses OBJ materials.
I think OBJ uses the Principled BSDF now? There is a Python helper module for add-ons to use:
https://wiki.blender.org/wiki/Reference/Release_Notes/2.80/Python_API/Modules#Shader_Node_Import_.2F_Export
Potentially .mtl I/O could remain in Python since it’s not that performance sensitive.
will u also manage to make alembic import all attributes from houdini not just one?
As mentioned on IRC, I think you should also work closely with the glTF maintainers to ensure your generic support also works for them. It’s a popular format that is also slow when transferring geometry data between itself and Blender.
You don’t have to work on their code yourself, but you can leverage them instead. It would be cool to say at the end of the summer that external partners are already leveraging your system in their own addons etc.
Perhaps @JulienDuroure or @donmccurdy could chime in to see if they’re interested in being the first such users for your project.
2 Likes
This is a nice news.
It would be great, in addition, to have an easy way to import (…and export, by object or collection) multiple file at once!
Thank you,
Riccardo
3 Likes
Hi Hugo,
I’m one of the maintainer of glTF addon included in Blender. As mentioned, I could be interested by your development. Don’t hesitate to contact me if you have any questions about glTF import/export process, and how your development can help.
Julien
6 Likes
The link to the proposal seems broken for me, does someone have a corrected one?
I’m not sure how much development time I will have over the summer, but would love to stay in touch with you and Julien on this, and see what we can do. My email address is listed on my GitHub, if needed.
Best,
Don
2 Likes
What a great idea for a GSOC project.
Just the other day a coworker asked me why he couldn’t just drag and drop multiple OBJ files from the Windows Explorer into Blender, and I couldn’t give him a satisfying answer.
Will something like that also be part of this project?
It would be great if you can provide a link to your proposal, I’d love to give it a read!
I apologize for the missing link, but it should be fixed now 
Here it is again, for convenience. I’ll try to create a wiki page, but I wasn’t able to so far…
1 Like
Hello Hugo,
First of all I want to say thank you for dedicating some time of your summer to this project.
I would like you to consider in this project to make the importer module “auto-detect” the file type. What I mean by this is, we would not have a list of different file types to import, but instead would have only one “import” option that could accept all supported file types. Ideally this would also be supported by just dragging and dropping a file into the blender window just like it happens with .blend files. This would make the workflow of bringing models into blender much faster.
Thanks,
5 Likes
I like that idea, actually. However, I don’t think it’s up to me to decide that. @jacqueslucke, @brecht What do you guys think of this idea?
It’s something we’ve discussed before and would like to have. But no one has got around to making a concrete design for it yet.
1 Like
I think everyone likes that idea. I even worked on it a little bit in the past.
It’s just a bit out of scope for this GSOC project.
A few questions regarding the proposal:
-
As mentioned, there were several attempts to replace the Python scripts by C/C++ implementations. If I remember correctly, one reason why not much effort was put into native importers/exporters was because of the desire to keep them hackable (=Python). It is safe to assume that this attitude changed?
-
The proposal talks a lot about speedups by moving to native code. It does not say anything about memory efficiency however. Python scripts running single-threaded is just half of the problem, the other half is the excessive memory utilization especially with the split option enabled in the OBJ importer. Is reducing the memory consumption also a goal of the rewrite?
-
Will the common framework hold (a copy of) all data and provide methods to transform the data in-memory, or how will data be dealt with? Or will it be limited to utility functions and the specific implementations for a format handle the data?
-
The scope of the project is re-implement import/export functionality of simple formats. What about complex formats however? The assumption appears to be that other scripts will also be written from scratch, using the common framework. The COLLADA and FBX formats are known to be difficult to implement. Were existing libraries such as assimp considered but found to be problematic to integrate? (also see the 6-year old blender-assimp)
-
Are there preliminary benchmark results or will there be benchmarks of the C++ and the Python implementations, taking performance as well as memory usage into account? Could the test suite also act as benchmark perhaps?
Regarding the single import option with auto-detection of the file type + drag’n’drop support:
-
If you drop a .blend file in 2.80, a menu opens to ask if you want to open, link or append the file. If you choose append for instance, then the file dialog with the import settings opens.
-
Dropping files like .obj should work similarly. However, there is only a single action per file type: import it using the appropriate importer. Based on the file type, the expected file dialog should show up. There are two problems however: there could be multiple importers for a single file format, and the file extension does not necessarily indicate the file format.
-
Maybe there are two different implementations of an OBJ importer, both with the benefits and drawbacks. There is probably no easy way to automatically determine which one is better in a certain situation, therefore the user should decide. A possible solution would be to roughly follow the concept of intents in the Android OS: “import OBJ” would be the intent, and every importer can announce whether it can fulfill this user intent or not. If there is only one importer which implements OBJ import, then it will be used without asking the user. If there are multiple choices, then a popup menu should list the available importers and the user can pick one. For convenience, the user could be given an option to make one importer the default for that intent (import OBJ) to skip the menu - but also be able to undo this decision in the settings to be asked again or to make another importer the new default.
-
While I don’t know of any file extension ambiguities among common interchange formats, there could be situation in which the extension is not enough to determine the file format and thus the correct importer. My above suggestion using intents should solve this (ask user if there are multiple choices). A file might also have a wrong or no file extension - but that is probably rare and I think we can ignore this special case (to be fixed by user).
-
The drag’n’drop case is actually simpler than a single Import option in the File menu: the importer is chosen before the file dialog is shown. With a single Import option however, the file dialog would show when it’s unclear what file type will be imported. It could show supported file types only (based on the intents system), but can not make other assumptions until the user selects a particular file to import. The UI with the format specific import options would need to update unless all import options would be unified (unlikely), but in case of multiple importers for a single format how to let the user select the one he wants? Also, what if multiple files are selected with different file extensions?
4 Likes
Hey Hugo,
Please consider adding support for vertex colours on .obj format. I know this is not in the standard specification but quite some tools support this by now (zbrush, substance suite for example). These colours are often used as simple masks or some sort of base texture. I think this is mostly used on game content creation workflows.
Guess it’s kinda trivial to include this once the standard behaviour is implemented.
Just drop me a line if you need an example file.
Curious to see where this project will go - we need faaaaast imports/exports 
9 Likes
I’ll consider it, but it won’t be a priority. An example file and/or a page describing the format would be useful, so do please send it
3 Likes
While I don’t think the attitude has really changed, I think the need for a faster solution outweighs the “hackability” argument.
The speedups are the main concern, I think, RAM is there to be used. However, there will be inherent reductions in the memory cost, since it’s possible to avoid copies that are hard to avoid in Python. I will make it as memory efficient as is practical (deduplication could be done with no extra memory, at the cost of time, but that would kind of defeat the point).
I’ll avoid making copies of the data. I’ve noticed Alembic takes a different approach where it, for instance, makes a vector which holds a copy of all the vertices. I intend to solve this by instead providing a function template that takes a function object and iterates over the vertices; each exporter would then be free to do what it needs with each vertex, but generally a copy wouldn’t be necessary. (this is already written, as well as a variant which deduplicates this info, but if someone suggests a better approach, I’m open to suggestions. I expect some people would think doing it this way would incur at least a higher function call overhead, but from my experience it’s not a problem at all, since the optimizer is able to inline it.)
FBX is notoriously difficult, yes. The plan so far is really only to implement OBJ, PLY and STL, due to the time constraint. However, I would like to take a crack at implementing more complex formats after I’m done with these (even after the summer). I did see assimp, but I’m not sure if we want to have another dependency. It’s an option, though, but I didn’t look at the code.
I posted those somewhere, I think in my original diff. Essentially I saw something like a 40% speedup in a debug build, without using memory mapped files, which should make it significantly faster. (I tested a release build, but I don’t have the measurements around. Currently building)
Interesting idea, I think it would be a neat way to do it. But, again, for later.
Could be a menu with an option to pick any importer. But i don’t think it’s a real issue.
The same way, just a pop up menu.
Go through them in sequence.
1 Like
At a certain size, with certain files, RAM is much more of a concern than speed.
I’ve had files in the past where I don’t care if it takes 2-3 hours or more to import, but I need them imported. In such cases, where the python importer used up 32+gb or RAM, we ran out, and the only course of proceeding was to “hack” the importer to drop data we didn’t need (in this instance, normals, smooth groups, uv data, materials)