GSoC 2020: Faster IO for OBJ, STL & PLY. Feedback

Week 2 Report: June 8 to 13

  • The OBJ exporter now exports Texture Coordinates. rBf01a7d50. Not the MTL file yet. It’s a goal of Week 3.
    Also see the table of comparison in #15
  • Since fstream is not adding any benefits and is slower too, it was removed.
  • It also exports multiple objects in the same file. But so far only OB_MESH object types are supported. rB40736795.
  • Multiple frames can also be exported now, in individual files. rB6d088bdd.
  • Progress is now shown on per-file basis in Terminal and timing in the end. It should be improved to act on per-object basis when there is a single frame.
  • The windows build to test is up: https://blender.community/c/graphicall/Pmbbbc/

Week 3 Todo

  • Curves, Axes transform, Scale transform.
  • Object Grouping, Indices (absolute is already there, will discuss about relative.)
  • Materials, Triangulation.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_2

12 Likes

Edited wrongly. Removed.
See #21

1 Like

This is awesome! Thanks for your work on this Ankit, it’s really appreciated. Obj export got 10 times faster. Still using OBJs because of zbrush and it’s not gonna change any time soon, so this speedup is going to be a huge time saver.

4 Likes

Hey all! More features and less memory usage this week!

Week 3 Report: June 15 to 19

  • Animation: Allow all frames to be exported, instead of 0-1000. Also filenames are also edited with frame number only if there are multiple frames. rB45461012
  • Worked on exposing more settings in the UI:
  • Add support for curves to be exported after converting them to meshes.
  • Refactor the code upto now to object-oriented style. D8089
    This allowed us to have minimal memory overhead with no timing cost. UV vertices, however, have to be allocated on heap and thus show a peak while exporting.

Week 4 Todo

Unfortunately, I couldn’t finish some goals at time: Material library, NURBS and grouping settings. A quick thing to add is export selected objects only.
So the goal for next week is to finish them as soon as possible and start with importer.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_3:_June_15-19
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_3

15 Likes

Great progress!

Something that’s mildly annoying is obj’s inheriting that 90 degree rotation as they import (due to Z pointing up in blender I guess) Would it be possible to apply those transforms in the background as importing/exporting? Not sure if it would mess with peoples pipelines, for rigging it can be painful at times.

Also, something that’s probably out of the scope of this project but I personally do constantly in many packages (with the use of addons/plugins mostly) is to optionally export selected meshes as separate objs based on their mesh names, or import multiple objs at the same time with the naming being based on their file names.

I know it’s not on the roadmap and probably falls under a different category as I’ve seen plans for drag and drop and automatically detect the format when importing, but I figured I’d throw it in here. :slight_smile:

I know the difference in the axes. The python exporter rotates the object about x-axis on its default settings. The cpp one doesn’t. It exports the object the way it looks in the viewport.
I don’t know why the rotation is there. I will see what other addons do or ask someone why they do it.
It’s a very minor change though.

I will expose the “select objects only” soon. More features on top of the goal of making the cpp one drop-in replacement of the older one should be handled when I’m not at a risk of failing. (:

Drag and drop is bound to the File Browser and the way operators work. (or something to that effect as far as I remember ) I think UI (and data, assets & IO) team should add the feature and the importers will make changes in their own code to make it work.

5 Likes

Week 4 Report: June 22 to 29

  • Cleanup and review update in the OOP refactor: rB0f383a3d.
  • Support NURBS being exported in parameter form, optionally. So far, they were exported as meshes. rB31d48017.
    • Support NURBS surfaces to be exported as a regular mesh. rB31d48017.
  • Broke down files to create new ones for OBJMesh and OBJNurbs . rB6c98925d Later on, MTLWriter will also have its own implementation file.
  • Support material library export. We decided against using the existing python writer, since there were no benefits of using it except saving some time now. To extend support, one would have to edit Node Shader Utils file first. Also debugging it would be a trouble. Since nodes are already in C/C++, we have more control there, instead of in the API.
    • Export of single material per object: rB827869a4. Support for multiple materials in one object: rBfaa11ec0
  • New options in the UI:
  • Bug fixes:
    • Duplicate texture coordinates: rBef0eff0b
    • Show red colored warning if overriding an existing file while exporting (it was a wrong flag) rBc39128ca
    • Fix socket vector being read as socket RGBA rB7685d9e4

Remaining tasks and Week 5 todo

  • Smooth groups (and bitflags) & Vertex groups
  • Start with importer. I’m already lagging here User:Ankitm/GSoC_2020_Proposal_IO_Perf#Project_Schedule. But since the refactor I had planned for week 7 (along with profiling) won’t take long, I feel that I have a good buffer to work in.

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_4:_June_22-27
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_4

8 Likes

Exporter is finished now.

Week 5: June 29 - July 4

Ticked exporter checkbox in the status tracker task. T68936

Week 6 todo

Importer!
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_5:June_29-_July_4
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_5

11 Likes

Week6: July 6 - 10

  • Setup clang tidy with Ninja. On macOS, had to build llvm + clang 11 + tidy to get the executable. Also setup a build with clang 11 + leak sanitiser since Xcode’s asan doesn’t provide the functionality.
  • Import a mesh successfully in the viewport. And add its object in a collection. It took a long time, trying to avoid Mesh, BMesh asserts, seg-faults and memory leaks due to un-freed meshes.
    – The mesh has vertex data only, so far. Texture coordinates are to be added.
    – The parser takes a long time. ~70% of the import process. Making a Blender object from the parsed data is the rest ~30%. So need to optimise it.

    – The importer is still 3-4 times faster than the python one.

Side note: an interesting diff is under review: Use mmap() IO for reading uncompressed .blends

Week 7 planned tasks

  • Texture coordinates, normals and curves.
  • Vertex Grouping, object grouping.
  • Material library and grouping.
18 Likes

Nice!
May the force be with you! :muscle:

1 Like

Week 7: July 13 - 17

  • Break code into mesh, objects, and reader files. rB7582bbd5
  • Directly create Mesh without using intermediate BMesh ec04edfe5c. It lowered the peak memory usage by half. Also made mesh creation faster.
  • Use stoi and stof instead of >> for faster parsing. aacb1f4756. Time taken in string to float conversion is less now as compared to >> operator. But the cost is in splitting the string and allocations made for several substrings. Overall this is faster.

OBJ import the same file as last week using stoi . cube subd 8; 23.3MB file.

  • Fix UV crashes and redundant loops in v1//vn1 case. rB92be92be
  • Support meshes with edges with or without polygons. 5a9b983263
  • Importer: Support smooth shading for polygons. rB031c4732
  • Importer: Fix misaligned UV vertices of a mesh rBfe4c5350
  • Exporter: fix vertex normals not being exported when smooth shading is enabled. Calculating smooth groups before exporting normals was the solution. rB501ead4b

Week 8 planned tasks

  • Due to curves, the parser needs some changes in how it stores the vertices. There can be vertices that belong to no object. And then later on an object refers to them with relative indices. So there needs to be another copy of all vertices which can be referred to by curves.
  • Grouping
  • MTL
12 Likes

Week 8 : July 20 - July 25

  • Move vertex (and UV vertex) list outside OBJRawObject so that objects that are defined (o curvename) after the list of vertices is written can access them using relative indices. rB71eadb4b628
  • Support NURBS curve import rB6e21f8c20da. Got help from Howard Trickey for fixing a visibility bug. rBaf278ce58bc
  • Fix build errors: avoid uint with Vector’s size() function. rB7bd38c27761. Style guide was updated to discourage uint usage and {} initialisation doesn’t allow narrowing conversions. There’s still a lot of code in OBJ exporter that uses uint, that’s for a Friday. This commit was only for removing uint with Vector.size().
  • Conform to style guide, add documentation 97aa9d44fa0.
  • Rename exportable_meshes to exportable_as_mesh rB47166605915. (This was long due.)
  • Cleanup: silence clang-tidy warnings; comments. rBffaa1df4397.
  • Move Object creation code in Mesh creation class. rB2bb56c83faa. This helps adding deform groups and materials to the Object in one place and away from the code that adds that Object to a Collection.
  • Support importing vertex deform groups. rB25526821a4b.
  • Use VectorSet instead of vector for def_group. rBb33a4592a38. The commit rB37467ec5e9a used std::vector<std::string> which is slow due to linear searching. Jacques Lucke suggested using VectorSet for faster lookups. Also StringRef is used to avoid copying.
  • Wrote an intial draft of importer’s design document. T68936#982751. Still need to incorporate mentors’ feedback on the document.

Next Week plans

  • Fix design doc.
  • MTL and material grouping.
  • Port a portion of mesh editing tools written in python to C++: bpy_utils.mesh.ngon_tessellate (in release/scripts/modules/bpy_extras/mesh_utils.py

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_8:July_20-_25
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_8

16 Likes

Bro, will you support loading files by drag and drop?

6 Likes

Sadly, not in this project. It is not specific to the three formats mentioned here.

5 Likes

Week 9 July 27 - 31

I could not work for the whole week productively.

  • Off-topic (Master) Enable header auto-complete suggestions in Xcode. rB27d50d6f This turned into a standardised way of dealing with Xcode’s include file paths.
  • Off-topic: Respond to a new developer on a bug fix and the devtalk thread
  • Off-topic: Tested ClangBuildAnalyzer on Blender (lite build).

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_9_July_27_-_31
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_9

Delayed the weekly report by a day since yesterday was a holiday in India.

6 Likes

Replying to this comment again.

I came across this post: https://developer.blender.org/T44670
(Adding generic Drag&Drop to Blender for file formats (obj, dae, bvh, …))

3 Likes

Looks like a dead task tho. :smile:

Week 10 August 4 - 7

  • Accept 3D UV vertex coordinates rB6ee696e5
  • Fix wrong material exporter assert added recently. rB85989931
  • Refactors:
    • Replace enum and its functions with Map for maps rB928736b1
    • Move index offset from parsing to Object creation rB7f289384
    • Use std::string_view for string splitting. This gave 33% speedup. rB9616e2ef


OBJ Import using string_view with the file in Week 7 & 6.

10 Likes

As it might be clear by now, STL and PLY will not be covered in the final week. I’m sorry for that.
We’ll close the project with OBJ only and I’ll write the code for the other two formats later on.

Week 11 August 10 - 15

  • Catch out of range exceptions for stoi/stof. rBbb2eca07.
  • Add vertex offset due to Curves’ vertices also. rBb378f84f.
  • Use const& instead of pointers in Mesh class. rB82eff7a0.
  • Accept 3D UV vertex coordinates rB6ee696e5.
  • Fix loose edge export & import, and offsets. rBc010cf58.
  • Fix vertex offsets for MEdges and MLoops. rBe0c0b8ff.
  • Fix several material import bugs, remove wrong assert. rB606b0d7d.
  • Fix usemtl line not being written. rB69f70829
  • Use function pointer; remove redundant variables. rBba0d376e
  • Export packed image file names as if they’re in current directory. rBd9cdfba2
  • Use const; move constructors to implementation file. rBa725b6f7
  • Fix several material import bugs, remove wrong assert. rB606b0d7d
  • Apply axes transform to exportable Nurbs. rBd882a63e
  • Move MTL writer code near OBJ writer code. rBae1c5f16
  • Skip parsing extra texture map options, move MTLWriter from Objects code. rB99ededd9
  • Clean up: use const, header cleanup. rBa732abd9
  • Move file open errors to constructors. rB491bd8b8
  • Add a public member function in MaterialWrap. rBc815ba0a
  • Cleanup : clang format; cout > cerr rBc5e0e82f

https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Daily_Reports#Week_11
https://wiki.blender.org/wiki/User:Ankitm/GSoC_2020_Weekly_Reports#Week_11_August_10_-_15

Week 12 planned tasks

  • Expose necessary settings in the importer UI.
  • Custom normals import.
  • Fix todos I had left in the code.
29 Likes

Great to hear you intend to stick around and continue this work! A true GSoC success – this is one of the things that GSoC hopes to achieve, but doesn’t happen as often as we might wish.

16 Likes