We are considering breaking changes to the Python API for (shader) nodes, see the details here:
It would be interesting to hear from add-on developers how much of a problem this would be. To figure out if it’s worth to hold back some improvements for users until Blender 4.0 (about 1.5 years from now), or if this would not be a big problem to change in Blender 3.2.
tl;dr: I consumed the API, i’m really bitter about the experience, any change is an improvement, just go for it, the sooner the better.
For a personal project I’ve recently spend a healthy amount of time with our python node api, and honestly, don’t worry about breaking it, the whole thing is in such a state of disarray breaking it further isn’t going to surprise anyone working with it already, they know/expect it is going to break at one point.
Some random issues I ran into:
You cannot tell from a nodes type X (nor any of its base types) if it could go in node graph of type Y, nor is this available in the python documentation , example : ShaderNodeMath can go in both shader and geometry nodegraphs , how can you know? only by trying to insert it and see if it errors out.
the py docs on some nodes are abysmal, example GeometryNodeSwitch has 26 input sockets, none of them are in our docs, this is not limited to geometry node, ShaderNodeMath isn’t any better.
Given you’ll have to know the socket names to program against them (unless you just want to use indexes, which i’ll be honest, doesn’t seem like worse solution at this time), the only way to get them is to insert a node into a graph and query its inputs. Which isn’t great, but once you do, at least the socket names you have to program against make sense right? Nope wrong again! GeometryNodeSwitch’s inputs: Switch, Switch_001, False, True, False_001, True_001, False_002, True_002, False_003, True_003, False_004, True_004, False_005, True_005, False_006, True_006, False_007, True_007, False_008, True_008, False_009, True_009 , it’s outputs: , Output, Output_001, Output_002, Output_003, Output_004, Output_005, Output_006, Output_007, Output_008, Output_009, Output_010, Output_011 very helpful guys! really helps writing nice clean code. (again picking on GeometryNodeSwitch here, since it’s just shooting fish in a barrel there but similar offenders are ShaderNodeMath,FunctionNodeRandomValue, GeometryNodeAttributeTransfer, GeometryNodeCaptureAttribute, GeometryNodeRaycast, etc etc etc)
All the problems from 3 are hidden from the user with some clever UI tricks, GeometryNodeSwitch hides most of it’s inputs based on the input_type field, well that’s at least something… you can set the field and check if a field is visible or not, and generate your own documentation… ha! Nope! the hide field, returns false for all sockets. Not very helpful. you can get through this data by writing a helper function and query the SOCK_UNAVAIL bit in flags but lets be honest… you can’t expect this from people consuming the python API
def is_really_hidden(socket):
return (ctypes.c_short).from_address(socket.as_pointer()+162).value & (1<<3) != 0
Overall I found working with the node api rather unpleasant, i feel like (some of) the nodes have fundamental design issues that are being hidden in the UI layer, I’d like to say the BPY nodes api feels like a mere afterthought, but even that seems a little bit of a stretch, no thought appears to be given at all
so if you want to break a few more nodes by giving things more sensible names, go for it, break things, the only way is up from where we are, you literally can’t make the experience of working with nodes from python any worse (then again, please don’t take this statement as a challenge)
IMO these types of changes are
I dunno these seem pretty minor to work IMO.
@LazyDodo I agree this API should be improved, though what I’m asking about is a different issue I think. Regardless of how the API works, it’s still about the same work to update for socket and node name changes.
I guess if we improve the API alongside these changes that may make them more acceptable for Python developers.
@bsavery I guess you missed a word somewhere, but from what I understand it’s not a big deal to update for these changes for you.
Yes sorry I meant easy enough to work around. (Much easier than changing python version for us)
Please fix the node API in general. These seem like acceptable changes to me, who bases most of their addons around nodes and custom nodes. But for custom nodes, I think there could be done a lot more. I will skip what @LazyDodo has already mentioned but this is additional feedback that would be nice to have:
i guess lastly it would be a blast if as an addon dev you could write shader code for a custom node instead of having to piece together a node group or importing from another file but that is probably a much bigger topic
TLDR: I think its fine to break the node API now but there are a lot of other changes that would make it easier to develop node addons
It all seems a bit ad-hoc and it’s not clear that the name changes will fix more than they break but I have to agree with the previous posters that they’re all changes that will be easy enough to live with.
Honestly, I think that as a user and addon developer who has a few add-ons that would be broken by this, I think that the improvement in UI consistency and ease of use is definitely worth it, even if it breaks some stuff.
As others have said, they’re not hard changes to account for.
@LazyDodo I agree with everything you’re saying, a better node API, would be amazing! Only thing I have to add to it is that you can check if a socket is visible in the UI by using socket.enabled. It can also be set from python, which can make things a bit easier to work with:
Ideally the Blender python API or “interface” for production should remain the same for longevity, engineering, pipeline and general working in studio. Add API, don’t change nor remove what is used - not till it is then made redundant or “legacy” in documentation.
Any change to this API would disrupt pipelines working with the shader API - not very welcome.
If necessary, point out the new API as better alternative in coding practice and documentation, but don’t stop supporting an old API till made redundant.
There needs to be a transition time so no pipeline will be disrupted.
An example where this practice wasn’t good was with the tile.x API being removed from Cycles when Cycles X went to master. It broke every baking addon in the pipeline and it took months for them to catch up. Often the code in the pipeline dependant addons had to be fixed manually, or not fixed at all. Had to burn a few days trying to fix a pipeline baking addon to continue working, or rolling back a few months.
@brecht I’ve spent a lot of time getting a usable design with the current API, so I’m not too happy about having to undo the work I’ve done, so if it breaks something, let it be for a good reason and really for improvement.
@Draise In theory a transition period would be good, but it’s often not practical when there are actual functional changes. The Cycles X render API changes were an example of that, we can’t have an API that pretends rendering works one way but Cycles actually does something different. Blender 3.0 was always going to be an API breaking release also, and there were 2-3 months for add-on developers to adapt to these changes in alpha/beta releases.
In this case also, if we’re unifying nodes or reorganizing sockets, a transition period also implies a worse user experience where they now encounter multiple node version depending on how the nodes are created. For some add-ons, just adding new nodes is a breaking change also. For example if we would have two versions of the Principled BSDF, a renderer or exporter will still need to handle both.
@povmaniac not sure what you mean by having to undo work, we’re not talking about any fundamental changes to the API. The proposed changes are listed in the linked task.
Yes. At first glance, no change seems very profound, but sometimes even the smallest change can break a plugin.
Breaking node api at 3.2 or 3.3 is fine!
I think my addons would pretty much not be impacted at all by these changes. Any change in the direction of unification and clarification are welcome for me.
I think I’ve never written code that references the sockets from name, only because it just completely breaks if the blender instance doesn’t use english as a locale language. And I anticipated some names would be changed at some point like fac or rgb which I saw are on the todo list, so yay !
Don’t get me wrong, referencing sockets by index is a pain with big nodes (like the principled bsdf) and it breaks when a node is updated, like when the emission was added to the principled bsdf.
So what I’d really like to see is a uniquely defined string identifier per socket that doesn’t change based on locale language.
And a pressing matter that would be nice to see fixed before it’s loo late to make breaking changes in my opinion is the Geometry Nodes API, specifically the way the fields are handled in the modifier properties panel.
Cheers
As far as I can tell neither node.inputs['Base Color'] nor node.inputs[0].name is affected by the locale. That would have been very surprising to me, because translation is handled at the user interface level, not the Python API level.
Sockets do have a .identifier, but that is used so that a single node can have multiple sockets with the same name and different type (for example for a Mix node that supports float/vector/color data types). But this is more exposing an implementation detail and not a good API.
Yeah technically there is time to adapt in master… . But in real world cases add-ons don’t develop for master but official releases. So even if master has a window to adapt a new API, the official addon release for an official branch doesn’t. Then we have broken pipelines for a few months at a time or need internal devs fixing other users’ add-ons.
There sometimes is a production speed bump when it comes to add-ons and python Api changes in general - and nearly a third of our core pipeline tools broke the past two years when updating to an official release (yes minor bugs, but at the moment our artists can’t reliably benefit from bug fixes without a pipeline addon UX breaking risk from master - where internal studio builds can only afford to merge only from master to future proof and add-ons devs usually won’t develop for master). Yes we fix add-ons to work for the master, but trend has most add-ons devs do not touch master nor develop for master (reason: unstable API), most only develop for and with the official branches. So this adaptation window is currently non existent in any official release branch cycle with addon development cycles.
Basically, the adaptation window doesn’t happen for any internal builds made from the Blender master with third-party addons because the addon trend develops for official branches for stable API longevity. But to upgrade an addon master to the official branch takes time and this means it’s broken in official releases for sometimes months on end. Pipeline either has to work around bugs with rolled back builds, or broken add-ons.
But yes… I get it. Developing null translators to old legacy API or literal translators would be a chore… but it would be nice to have python API transition translators also built into official branches atleast for one official release cycle. If an addon dev can’t update by two official releases, that would be their problem. But requiring an addon dev or our engineers to always develop future proof pipeline tools in the ever shifting master that doesn’t work with official releases is just not financially sound - even with 3 month warnings for the entire community with many third-party devs out of our responsibility and power to prevent speed bumps with internal builds compatibility with official release branches and master simultaneously.
I am not saying the API changes breaks are big, but they are a speed bumps in internal pipelines.
In general an API needs to be rock solid even in master. And if not solid enough, have enough translator support to pretend to be solid for a while so pipelines can adapt between versions and be educated on what is a legacy API scheduled to be phased out over two official releases.
If anything, what you did with legacy GN was genius and allowed pipeline to use GN legacy operators and API till there was time ready and to use the new methods. This was done beautifully, and it was “optional”. All API changes should have this. The fact it was official also gave non breaking time to third-party devs.
This is what LTS releases are for really. I don’t really see a practical way to add significant new features and improvements in 3.x releases without also breaking the API in some ways, even if we try to avoid doing it unnecessarily.
As far as I know the geometry nodes Python API in 3.0 was not backwards compatible. The old nodes all got renamed, among other changes. The proposed shader node changes are not more difficult than that, in the end it’s also renaming some nodes and sockets in scripts.
We could introduce two versions of every shader or geometry node that changed or had a socket renamed, to provide compatibility for scripts that only import or generate nodes. But that does mean users will encounter these two versions of about 15 nodes and exporter scripts will need additional complexity to handle both cases. That would accumulate for 2 years until 4.0 can break compatibility. It’s not obvious to me this is better, but maybe?
If they are small GUI updates changes great - but why the API? I guess it would be easy to maintain and update any tools without much of a hiccup if anything in GUI updated. If there is a lot of GUI with API changes - you could toggle it with the legacy method in the experimental user prefs and eventually use the new GUI and API without much of a hiccup. If there is a new UX with new API, then yes, you should merge in the new method with the old method for adaptability for a point release cycle, not just master dev cycle. But minor GUI updates wouldn’t need this no?
How much cleaning do you plan to do? Is it an entire refactor? Maybe this type of backwards compatibility would be irrelevant if it’s GUI updates only. Maybe it’s not even invasive at all.
Though, on a side note, the studio here cannot afford to use LTS branches (or any other branch other than the main trunk) due to the required features and bug fixes in master to get the job done - bleeding edge often is the only way (ei, using Apply Attributes to get UV’s fron GN for game assets in the GUI that came out this month), and we cannot fork the LTS branch because we have to merge and use every master update per week to keep code tidy, up to date and relevant - and future-proof our own builds - it is less overhead and easier to merge a single conflict resolve and maintain if from our master with your master this way. If we diverge from master more than a few weeks, the code and ability to merge in any new features into our own deployment is eventually unsustainable (or too big to do in a single chunk). We have to fork or branch from trunk to stay relevant, so far has worked for over 5 years now.
In general though, by planning the API to be future proof with flexibility to work around labels and socket index in the GUI and to allow new features - if additive also - there would be no problem. Legacy code or API can be phased out over time - as long as the trunk API which allows GUI updates is solid meanwhile. On the surface, GN with 3.0 felt like it had the necessary overlap with the “Legacy Nodes” option to work - though some addon still broke.
This is where official gitflow becomes sorely lacking when using the trunk with other studios or internal builds - since the trunk here in the Blender Foundation is the development branch, this makes these API changes much more evident since any studio that forks from the trunk will inherent shifting instability over time as experimental code gets tinkered with. And forking a branch is not future proof specially with the hundreds of commits we sometimes get per week to merge in and make sure it doesn’t step on any toes we develop in our own master, this is an engineers sometimes full time job.
Ideally any API development and breaking changes should be developed in a branch away from trunk to then confirm if it really will/should break anything before pulling into master, then it gets merged close to official release dates with a smooth time to update a pipeline that uses the trunk (and may encourage addon devs to develop in the stable master). But… as it stands, for a stable development trunk, to keep the trunk solid, some extra effort for compatibility with official release branches and their API would be coded in with overlap - and yes, that would be a pain over 2 years. It’s a bit of a chicken or the egg debate… Thanks for reading though!
I see “API change” and get ptsd from when thirdparty addons broke in the pipeline due to this happening elsewhere (not shader nodes).
Breaking the API would be required partially because of poor node API design, which to fix would itself require major API breakage. An also partially because fundamentally this API is about creating/reading user editable data structures, and there’s just some changes to the GUI you can’t pretend didn’t happen in the API, even if they are just additions.
Making these changes experimental makes no difference, either we do the changes or postpone them.
