GSoC 2020: Faster IO for OBJ, STL & PLY. Feedback

Contributing to Blender Summer of Code
10

@isolin That sounds painful! I’m currently(in first 4 weeks) not working on importer, so cannot comment on what the new one would look like, but definitely not this bad. (see next comment for the hopeful preliminary results.)

@testure Yes, I had seen the thread when I was going over the project list, some months back. I’m taking care not to make those mistakes .

Thank you @_name for telling me about it!
Reminds me to be careful about memory allocations & function overhead.

2 Likes
11

Week 1 Report : June 1 to June 6

  • Received a lot of input from the mentors regarding coding style, comments, documentation, and file structures. Spent quite some time on cleaning the hastily committed rB485cc433. See current version at Diffusion
  • Currently the exporter exports vertices, face normals, and face indices; in world coordinates.
  • It’s lacking in (1) multiple objects, (2) texture coordinates (3) Axes transformation. If you compare obj txt files written with python & C++ , you’ll see differences in vertex order, and some coordinates with opposite sign. But the shape is correct.
  • Did a comparison between the old & new exporters. Release configuration, lite build + opensubdiv, default cube with 6+3 subsurf modifiers. See images: F8586987 F8586984
    • wm_handler_fileselect_do takes 82 seconds (python) vs 9 seconds (c++) (which is expected to be less than 14-15 s with texture coordinates).
    • File size is 186 MB.

Week 2 Todo:

  • Textures, multiple objects in the same file.
  • Modifiers, Curves, Multiple frames , Axes transform, Progress logging

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_1

13 Likes
12

A new thread for reports, separate from this one (feedback) would be better, no?

2 Likes
13

The only possible upside, I see, is a single place to read all weekly reports. That I’m already doing at the wiki.

And more-than-needed links = bad UX (:

14

Previous years we had 2 threads, one for reporting so it’s easy to see what was going on, and one with lively back and forth between users and student where a weekly report easily would have gotten lost in the noise.

4 Likes
15

An update on the OBJ exporter: vertex coordinates, texture coordinates, and face normals.

image
image786×418 39.1 KB

The last case (icosphere + 8 subsurf) memory usage: ~6 GB in python. 1.67 GB in C++.

The table on weekly reports is 3 days old. In case you don’t use texture coordinates, check the smaller file size & less time taken at
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_1:_June_1-6

A build to test should be available (thanks to Lazydodo!) after these changes are merged in soc-2020-io-performance.
The code upto rB86845ea907f8ae24b3bed4ab591f8e3687ac5f22 crashes. So please don’t try that.

19 Likes
16

Wow, those export times looks amazing!
FINALLY! Quite a few times I was embarrassed to first show people how great Blender can be and then having to show them how to import assets into Blender just to sit there in awkward silence for 5 min.

10 Likes
17

Please test the build!
Thank you @lazydodo
f01a7d508d0d is the latest hash as of now.

The exporter is in the expected place, File > export > OBJ new. It only exports the single selected object as of now. And that should be mesh only. Not curves etc.
Also, if you run blender by console, you can see the export timings too.

Let me know about the bugs! Best is to share the steps to make a problematic mesh. If sharing the file is a must, please minimise its size.

3 Likes
18

It is currently crashing for me

19

@EitanSomething Thanks!
Fixed. rB3cbd4516c68f89942742c175ba7cdc2ce951b2d7 is the latest hash.
rBc2fbfb421ba4 fixed the issue.

Updated build should be available about 12 hours from now.

1 Like
20

I kicked it manually, will probably be 8-10 minutes done!

7 Likes
21

Week 2 Report: June 8 to 13

  • The OBJ exporter now exports Texture Coordinates. rBf01a7d50. Not the MTL file yet. It’s a goal of Week 3.
    Also see the table of comparison in #15
  • Since fstream is not adding any benefits and is slower too, it was removed.
  • It also exports multiple objects in the same file. But so far only OB_MESH object types are supported. rB40736795.
  • Multiple frames can also be exported now, in individual files. rB6d088bdd.
  • Progress is now shown on per-file basis in Terminal and timing in the end. It should be improved to act on per-object basis when there is a single frame.
  • The windows build to test is up: https://blender.community/c/graphicall/Pmbbbc/

Week 3 Todo

  • Curves, Axes transform, Scale transform.
  • Object Grouping, Indices (absolute is already there, will discuss about relative.)
  • Materials, Triangulation.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_2

12 Likes
22

Edited wrongly. Removed.
See #21

1 Like
23

This is awesome! Thanks for your work on this Ankit, it’s really appreciated. Obj export got 10 times faster. Still using OBJs because of zbrush and it’s not gonna change any time soon, so this speedup is going to be a huge time saver.

4 Likes
24

Hey all! More features and less memory usage this week!

Week 3 Report: June 15 to 19

  • Animation: Allow all frames to be exported, instead of 0-1000. Also filenames are also edited with frame number only if there are multiple frames. rB45461012
  • Worked on exposing more settings in the UI:
  • Add support for curves to be exported after converting them to meshes.
  • Refactor the code upto now to object-oriented style. D8089
    This allowed us to have minimal memory overhead with no timing cost. UV vertices, however, have to be allocated on heap and thus show a peak while exporting.

image
image240×560 20 KB

Week 4 Todo

Unfortunately, I couldn’t finish some goals at time: Material library, NURBS and grouping settings. A quick thing to add is export selected objects only.
So the goal for next week is to finish them as soon as possible and start with importer.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_3:_June_15-19
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_3

15 Likes
25

Great progress!

Something that’s mildly annoying is obj’s inheriting that 90 degree rotation as they import (due to Z pointing up in blender I guess) Would it be possible to apply those transforms in the background as importing/exporting? Not sure if it would mess with peoples pipelines, for rigging it can be painful at times.

Also, something that’s probably out of the scope of this project but I personally do constantly in many packages (with the use of addons/plugins mostly) is to optionally export selected meshes as separate objs based on their mesh names, or import multiple objs at the same time with the naming being based on their file names.

I know it’s not on the roadmap and probably falls under a different category as I’ve seen plans for drag and drop and automatically detect the format when importing, but I figured I’d throw it in here. :slight_smile:

26

I know the difference in the axes. The python exporter rotates the object about x-axis on its default settings. The cpp one doesn’t. It exports the object the way it looks in the viewport.
I don’t know why the rotation is there. I will see what other addons do or ask someone why they do it.
It’s a very minor change though.

I will expose the “select objects only” soon. More features on top of the goal of making the cpp one drop-in replacement of the older one should be handled when I’m not at a risk of failing. (:

Drag and drop is bound to the File Browser and the way operators work. (or something to that effect as far as I remember ) I think UI (and data, assets & IO) team should add the feature and the importers will make changes in their own code to make it work.

5 Likes
27

Week 4 Report: June 22 to 29

  • Cleanup and review update in the OOP refactor: rB0f383a3d.
  • Support NURBS being exported in parameter form, optionally. So far, they were exported as meshes. rB31d48017.
    • Support NURBS surfaces to be exported as a regular mesh. rB31d48017.
  • Broke down files to create new ones for OBJMesh and OBJNurbs . rB6c98925d Later on, MTLWriter will also have its own implementation file.
  • Support material library export. We decided against using the existing python writer, since there were no benefits of using it except saving some time now. To extend support, one would have to edit Node Shader Utils file first. Also debugging it would be a trouble. Since nodes are already in C/C++, we have more control there, instead of in the API.
    • Export of single material per object: rB827869a4. Support for multiple materials in one object: rBfaa11ec0
  • New options in the UI:
  • Bug fixes:
    • Duplicate texture coordinates: rBef0eff0b
    • Show red colored warning if overriding an existing file while exporting (it was a wrong flag) rBc39128ca
    • Fix socket vector being read as socket RGBA rB7685d9e4

Remaining tasks and Week 5 todo

  • Smooth groups (and bitflags) & Vertex groups
  • Start with importer. I’m already lagging here User:Ankitm/GSoC_2020_Proposal_IO_Perf#Project_Schedule. But since the refactor I had planned for week 7 (along with profiling) won’t take long, I feel that I have a good buffer to work in.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_4:_June_22-27
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_4

8 Likes
28

Exporter is finished now.

Week 5: June 29 - July 4

Ticked exporter checkbox in the status tracker task. T68936

Week 6 todo

Importer!
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_5:June_29-_July_4
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_5

11 Likes
29

Week6: July 6 - 10

  • Setup clang tidy with Ninja. On macOS, had to build llvm + clang 11 + tidy to get the executable. Also setup a build with clang 11 + leak sanitiser since Xcode’s asan doesn’t provide the functionality.
  • Import a mesh successfully in the viewport. And add its object in a collection. It took a long time, trying to avoid Mesh, BMesh asserts, seg-faults and memory leaks due to un-freed meshes.
    – The mesh has vertex data only, so far. Texture coordinates are to be added.
    – The parser takes a long time. ~70% of the import process. Making a Blender object from the parsed data is the rest ~30%. So need to optimise it.
    1440×900 266 KB

    – The importer is still 3-4 times faster than the python one.

Side note: an interesting diff is under review: Use mmap() IO for reading uncompressed .blends

Week 7 planned tasks

  • Texture coordinates, normals and curves.
  • Vertex Grouping, object grouping.
  • Material library and grouping.
17 Likes