Doing generative graphs to digest the code-base, what has been done in this area?

[JonasPrintzen]

Hello developers!

I am new in this (blender dev) context, but a senor consultant in SW-Engineering. I often find myself digesting non-trivial code-bases using structure/meta - data extraction with python, graphviz dot and the like. It started a long time ago and that habit has proven quite useful and appreciated.

No, I’m not talking about what Doxygen does. That’s kind of trivial.

I’m talking about what can be done to extract dependency-graph from a convoluted or deps-agnostic build-environment, State-machine graphs from scattered graph-agnostic code-fragments. Similar with interaction diagrams and event-propagation. To the point where you could rearrange an entire code-base automatically …

The value is, of course, the resulting meta-data and visualizations are always up to date, since they are generated from the code-base.

So, while I’ve already started, taking notes while breaking it down, I was wondering…

• Is there anyone else out there doing similar things?
• Are there more of this in there that I have not found?

Please tell me of it so I can align my efforts with what exists in the project …

Oh, and although I initially do this mainly for my own convenience …

As a consultant I’ve been in many different development cultures/environments. Still, a common and very interesting side-effect of presenting this meta and visualizations is that people after a while spontaneously strive to align their work, folder-structure, build-rules, file and function-names, with the rules I use to extract the information. More so the more quality the visualization have. I’ve also noticed a clear positive effect on the common understanding of the shared code-landscape when this happens. It’s an interesting meta-cognitive mechanism that is also one of the few instrumentation’s that clearly help handle continuous improvement in stagnant legacy.

So, If you are, or know of someone who is, interest in this aspect, please inform them/me?!

Cheers!

/jonas

PS:

Note! This is raw. I will soon present a crisp blocked layered arch-view auto-generated from the same meta…

:DS

[YvesBodson]

your work reminds me when I was at UCLA studying computer… We briefly had some classe on graphic code structures and mostly writing graphing tools… At that time Autodesk was starting to work in that field… We’re talking 1989 - 1992
I think this is a great initiative and you may help to bring up to speed new programmers to the blender dev team by helping them to map better the state of the developments current and future…
Personally I would be interested to come back into that field but it is a long time I haven’t been thinking about that approach…
I keep following your work and if I can find a spot where to help I’ll do…

[JonasPrintzen]

Seems we share some common ground and time-frame then.

Thank you for your reply and encouragement

[Josephbburg]

Is that kind of visualization something that could help new developers understand the structure of the project quickly? I’ve tried a few times to write patches for Blender but all I can ever manage is to write a few printf's and get myself confused. I think a graph explaining how all of the pieces are related could be helpful for bringing new developers onboard.

[Aligorith]

I agree that effective visualisations are often the first step to understanding a challenging problem/problem-space. Finding the right way to slice and dice things to make that happen though is the challenge

Have you considered/tried SourceTrail (https://www.sourcetrail.com/) on the codebase yet by any chance? (I’m somewhat curious how well it fares with helping trace the stuff in source/blender/editors/interface

From the look of things, you’re currently generating this graph from the directory-libs that are defined by the CMakeLists hierarchy. IMO, you’ll probably find eventually that this doesn’t exactly yield as much value as you may hope on this codebase as it stands (e.g. things like blenkernel and blenlib end up being outliers as everything ultimately either depend on them either because they provide common/core functionality, or because every module/component is inside those). Then again, maybe that in itself is a valuable insight…

Realistically, most development is concentrated within source/blender (and also release/scripts). You can completely ignore extern (unless you’re trying to fix the build system when updating certain external inlined repos), and intern doesn’t matter unless you’re dealing with certain C++ based standalone nearly-Blender-specific libraries (e.g. Cycles, various sims, IK solver engines)

[JonasPrintzen]

Thank you Aligorith for the reply!

Thanx for the tip on sourcetrail, will take a look, that’s right up my alley

Yes, I am right now scanning the build-system in an attempt to get the anatomy from that perspective.
And, yes, I’m aware it’s an incomplete view. I have at least half a dozen other approaches I often try though. The most interesting possibilities typically show up when combining several views, finding the heuristics to condense into different groupings and filtering the meta-data to understand archetypes and relations of those groupings. I definitely see already the gravity towards source/blender and I’m actually looking forward to see the internal’s if the internals.

I almost always end up with a few graphs showing the de-facto SW-architecture and dependency-patterns and possibly issues (circulars, cohesion, ). In either case I choose to get familiar with the code this way because it’s always worked for me and I find the potential for adding clarity compelling.

As soon as I reach a level of (team-confirmed) clarity I also hope to contribute with some SW-architecture and legacy design pattern documentation. The WiKi would be the best place for that I suppose?

[JonasPrintzen]

Hello Josephbburg!

I totally understand the confusion you are talking about
Even after 40 years of SW-developing and working with 100+ code-bases I get stomped
at start when diving in to something like this. It’s very seldom that clarity prevails over time.

That’s why big old companies are always inefficient compared to new and smaller ones.
Not seldom to the point of self-destruction…
SW quality maintenance is a popular cost-cut among high-strung management people

So, Yes, helping and guidance is my intention. I have learned that not only are visualizations like what I’m attempting important for newbees in a team, like you and me, but it helps the entire team stay structure-aware as well. This is often underestimated by ‘old’ team members.

[Josephbburg]

It looks to me like your work fits neatly into the goal for the new development funds- that is: to improve code quality, consistency, and organization. I think it’s Dalai Felinto and Nathon Letwory (jesterking) that are in charge of this.

[ScottVanKirk]

This is an awesome and daunting approach. I wish you luck and will be watching to see what you come up with

[EAW]

I am a novice to coding in general, just helping out with the bug tracker at this point. I definitely can see this helping with my personal understanding of the code. I appreciate your efforts here

Just a note: I think you are missing an “O” in your first line of your first post. “HELL DEVELOPERS”

[JonasPrintzen]

LOL

U may B right …

Or it’s an ‘i’, as in Heil.

[JonasPrintzen]

Well, it’s really simple to extract some of the information that is constantly not up to date on the WiKi. But then again, when used right, doxygen is king in this area.

So, I think I’l get involved there to. Hopefully I won’t get stuck in documentation (again ) I do want to implement some awesome ideas too… when I feel that I understand enought…

[JonasPrintzen]

Helo @Aligorith !
Hmm, my first impression of sourcetrail is that t’s basically a doxygen-ripoff with a (much) more dynamic UI. I recently discovered that the use of doxygen is in it’s cradle so I’m thinking that’s a better starting-point.

Cheers!

[B_Layer]

I graduated from UCLA with a B.S. in CS&E in '92. Wonder if we knew each other.

Go Bruins!

[YvesBodson]

Cool I got my degree as well in 92 in CS my thesis was DB associated with an AI engine for data pre analysis and structure

In my class they were also Babak Bahesti, Chris Walker… One of our math teacher was Frank D. Little PHD… great guy!

Very cool to connect with you

[YvesBodson]

it helps the entire team stay structure-aware as well. This is often underestimated by ‘old’ team members.

Oh! Yes! (I must fill 20 characters)

[B_Layer]

Ah so you were post-grad. Though mostly around other undergrads I shared some classes with grad students…maybe one of them was you. Regardless, as you say, it’s nice to connect with a once fellow traveler through the engineering school hallways. Cheers!

(Frank Little kind of rings a bell…heard of him…maybe got taught by him…will have to look back at some old materials.)