GSoC 2020: Faster IO for OBJ, STL & PLY. Feedback

Contributing to Blender Summer of Code
#30

Nice!
May the force be with you! :muscle:

1 Like
#31

Week 7: July 13 - 17

  • Break code into mesh, objects, and reader files. rB7582bbd5
  • Directly create Mesh without using intermediate BMesh ec04edfe5c. It lowered the peak memory usage by half. Also made mesh creation faster.
  • Use stoi and stof instead of >> for faster parsing. aacb1f4756. Time taken in string to float conversion is less now as compared to >> operator. But the cost is in splitting the string and allocations made for several substrings. Overall this is faster.

OBJ import the same file as last week using stoi . cube subd 8; 23.3MB file.

  • Fix UV crashes and redundant loops in v1//vn1 case. rB92be92be
  • Support meshes with edges with or without polygons. 5a9b983263
  • Importer: Support smooth shading for polygons. rB031c4732
  • Importer: Fix misaligned UV vertices of a mesh rBfe4c5350
  • Exporter: fix vertex normals not being exported when smooth shading is enabled. Calculating smooth groups before exporting normals was the solution. rB501ead4b

Week 8 planned tasks

  • Due to curves, the parser needs some changes in how it stores the vertices. There can be vertices that belong to no object. And then later on an object refers to them with relative indices. So there needs to be another copy of all vertices which can be referred to by curves.
  • Grouping
  • MTL
12 Likes
#32

Week 8 : July 20 - July 25

  • Move vertex (and UV vertex) list outside OBJRawObject so that objects that are defined (o curvename) after the list of vertices is written can access them using relative indices. rB71eadb4b628
  • Support NURBS curve import rB6e21f8c20da. Got help from Howard Trickey for fixing a visibility bug. rBaf278ce58bc
  • Fix build errors: avoid uint with Vector’s size() function. rB7bd38c27761. Style guide was updated to discourage uint usage and {} initialisation doesn’t allow narrowing conversions. There’s still a lot of code in OBJ exporter that uses uint, that’s for a Friday. This commit was only for removing uint with Vector.size().
  • Conform to style guide, add documentation 97aa9d44fa0.
  • Rename exportable_meshes to exportable_as_mesh rB47166605915. (This was long due.)
  • Cleanup: silence clang-tidy warnings; comments. rBffaa1df4397.
  • Move Object creation code in Mesh creation class. rB2bb56c83faa. This helps adding deform groups and materials to the Object in one place and away from the code that adds that Object to a Collection.
  • Support importing vertex deform groups. rB25526821a4b.
  • Use VectorSet instead of vector for def_group. rBb33a4592a38. The commit rB37467ec5e9a used std::vector<std::string> which is slow due to linear searching. Jacques Lucke suggested using VectorSet for faster lookups. Also StringRef is used to avoid copying.
  • Wrote an intial draft of importer’s design document. T68936#982751. Still need to incorporate mentors’ feedback on the document.

Next Week plans

  • Fix design doc.
  • MTL and material grouping.
  • Port a portion of mesh editing tools written in python to C++: bpy_utils.mesh.ngon_tessellate (in release/scripts/modules/bpy_extras/mesh_utils.py

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_8:July_20-_25
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_8

16 Likes
#33

Bro, will you support loading files by drag and drop?

6 Likes
#34

Sadly, not in this project. It is not specific to the three formats mentioned here.

5 Likes
#35

Week 9 July 27 - 31

I could not work for the whole week productively.

  • Off-topic (Master) Enable header auto-complete suggestions in Xcode. rB27d50d6f This turned into a standardised way of dealing with Xcode’s include file paths.
  • Off-topic: Respond to a new developer on a bug fix and the devtalk thread
  • Off-topic: Tested ClangBuildAnalyzer on Blender (lite build).

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_9_July_27_-_31
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_9

Delayed the weekly report by a day since yesterday was a holiday in India.

6 Likes
#36

Replying to this comment again.

I came across this post: https://developer.blender.org/T44670
(Adding generic Drag&Drop to Blender for file formats (obj, dae, bvh, …))

3 Likes
#37

Looks like a dead task tho. :smile:

#38

Week 10 August 4 - 7

  • Accept 3D UV vertex coordinates rB6ee696e5
  • Fix wrong material exporter assert added recently. rB85989931
  • Refactors:
    • Replace enum and its functions with Map for maps rB928736b1
    • Move index offset from parsing to Object creation rB7f289384
    • Use std::string_view for string splitting. This gave 33% speedup. rB9616e2ef


OBJ Import using string_view with the file in Week 7 & 6.

10 Likes
#39

As it might be clear by now, STL and PLY will not be covered in the final week. I’m sorry for that.
We’ll close the project with OBJ only and I’ll write the code for the other two formats later on.

Week 11 August 10 - 15

  • Catch out of range exceptions for stoi/stof. rBbb2eca07.
  • Add vertex offset due to Curves’ vertices also. rBb378f84f.
  • Use const& instead of pointers in Mesh class. rB82eff7a0.
  • Accept 3D UV vertex coordinates rB6ee696e5.
  • Fix loose edge export & import, and offsets. rBc010cf58.
  • Fix vertex offsets for MEdges and MLoops. rBe0c0b8ff.
  • Fix several material import bugs, remove wrong assert. rB606b0d7d.
  • Fix usemtl line not being written. rB69f70829
  • Use function pointer; remove redundant variables. rBba0d376e
  • Export packed image file names as if they’re in current directory. rBd9cdfba2
  • Use const; move constructors to implementation file. rBa725b6f7
  • Fix several material import bugs, remove wrong assert. rB606b0d7d
  • Apply axes transform to exportable Nurbs. rBd882a63e
  • Move MTL writer code near OBJ writer code. rBae1c5f16
  • Skip parsing extra texture map options, move MTLWriter from Objects code. rB99ededd9
  • Clean up: use const, header cleanup. rBa732abd9
  • Move file open errors to constructors. rB491bd8b8
  • Add a public member function in MaterialWrap. rBc815ba0a
  • Cleanup : clang format; cout > cerr rBc5e0e82f

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_11
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_11_August_10_-_15

Week 12 planned tasks

  • Expose necessary settings in the importer UI.
  • Custom normals import.
  • Fix todos I had left in the code.
29 Likes
#40

Great to hear you intend to stick around and continue this work! A true GSoC success – this is one of the things that GSoC hopes to achieve, but doesn’t happen as often as we might wish.

16 Likes
#41

Wow. This is great! I will have to test it out. I usually use FBX, but if OBJ is now 3-5x faster or more , I’ll probably have to switch. As some others said, importing / exporting in blender is often insanely slow for highpoly meshes. Sometimes taking 5,10,15 min vs 30 seconds in other apps or 1-2 min. So with what you’ve done here, maybe blender will be about the same now!? :scream:.

Although another DCC I use , exports OBJ pretty slow or quite noticeably slower than FBX, so I’ll have to compare and see what the time differences are.

6 Likes
#42

Really looking forward to this one. Been used to exporting .obj because of ZBrush and Substance Painter by having to export high poly and low poly for baking and stuff. It was always disheartening to see Blender taking several minutes to export million poly sculpts. If that’s finally fixed, it will make me very happy. :smile:

Can’t wait to see the results in master! :partying_face:

7 Likes
#43

I just did a test on a 16 million Tri Scene, with multiple objects.
It took about 35~ seconds to export with the new obj option and almost 2 minutes with the FBX option.

Although importing an OBJ into 3ds Max is pretty slooow… It also imported all my stuff as 1 mesh and not seperate meshes.

Max imports / exports FBX quite noticeably faster than it does OBJ. Importing this OBJ into max is taking beyond 5 minutes and i ended up just killing the process. :frowning:

The FBX took about 20 seconds to import into max by comparison.

The funny thing is, I often find blender saves files way faster than max can, with collapse/dense geo. At least vs. Max 2016. It’s just importing/exporting out of blender is often slooooooow. :’(

#44

Week 12 August 17 - 23

  • Fix overlapping nodes in the nodetree. rB0c7801fa
  • Vertex normals:
    • Parse vertex normals: vn lines. rBc583afd6
    • Decouple getting smooth groups from export parameters rB0479dc41
    • Export vertex normals if smooth shading is enabled. rB4923087d
  • Refactor:
    • Move file opening errors to constructors. rB491bd8b8
    • Add a public member function in MaterialWrap. rBc815ba0a
    • Move object creation code from constructor. rB4777a6a5
    • Remove unused uv vertex index offset. rBedd5307e
  • For polygons with holes:
    • Commit as it is ngon-tessellate. rBefab0bc7
    • Fix tessellate crash due to wrong Vector size. rB5b0cb5bb
    • Add newly creates faces to imported faces and dissolve edges. rBe76ab124
    • Edit FaceElem list outside polygon creation code. rB10e3f23b
  • Parsing
    • Use remove_prefix for easy transition to StringRef. rBae122533
    • Use blender::StringRef instead of std::string_view . rB73454388
    • Read multiple lines with “\n” line breaks. rBa662ddd7
  • Silence normal being zero warnings in mesh validate’s verbose mode. rB3f8a8d17
  • Silence mesh_validate verbose warnings. rB32fe2869

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_12_August_17_-_21
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_12

10 Likes
#45

Hey everyone, thank you for keeping me motivated, sharing blog posts, and feedback!
Please find the final report here: https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020/Final_report
Differentials under review: D8754 , D8753

21 Likes
#46

Awesome progress, great congrats!!
I can’t wait to try in master:)

1 Like
#47

Wonderful achievements in this GSoC, congratulations! :beers:

3 Likes
#48

:1st_place_medal:

2 Likes
#49

So, what’s the status of this, dead, alive?