Blender 2.81 and api compatibility

Archive Python API
1

Cross posting from bf-committers for visibility and discussion:

First of all, Kudos for the 2.80 release, and I think the move to higher frequency smaller releases with 2.81 is great.

One thing that I have not seen discussed with this release cycle change is the python api and addons.

Currently, as things stand, a user would have to reinstall all addons with each new release between 2.80 and 2.81, which is fine with older release cadence, but becomes slightly more burdensome with quarterly-ish releases.

Furthermore, what about api compatibility and python compatibility between releases? Has there been a discussion about that? As an addon developer, I feel the 2.80 release was a bit late breaking some apis (especially for render engines) and would prefer not to see that repeated with each 2.XX release. Us addon developers would prefer to no have to put out a version for each quarterly release if possible (but this might be selfish a bit on our part).

This is all further complicated if an addon uses c extensions that need to be compiled against a certain python version, and that python version changes between Blender versions. Overall, this might be one more vote towards moving to a version scheme like 2.8.1 where api and python version is stable across minor versions.

Anyway like I said I havenā€™t seen this point brought up. If thereā€™s a discussion somewhere I could contribute to, please point me there!

1 Like
2

Personally, I find that there are many ā€˜holesā€™ in the python API right now that could be improved, so if I were told that there would be no improvements to the Python API except once every year or two I would be fairly frustrated by that.

Now, if we were to adopt a non-backward-compatible update policy, where new things could be added but ā€˜currentā€™ API calls wouldnā€™t be deprecated until the next major release I would be all for that. If youā€™re familiar with Unityā€™s C# library, this is basically how they do it- they add new functionality all the time and if they decide that something needs to go it gets flagged as deprecated. Any time you use that functionality youā€™ll get a compiler warning letting you know that it will go away in a future version so you can keep it on your roadmap or fix it right then and there if you want.

The only ā€˜downsideā€™ with this is that point releases arenā€™t usually going to be backward compatible. So, if new python api calls got added for 2.81, those scripts wouldnā€™t work in 2.80 but they would work on 2.82, 2.83, etc. When 2.90 comes out, all of the 2.8x deprecated calls are removed and addon developers who have not kept up will have to refactor. At least this way you can do some planning on when and how you update your addons rather than being forced into it for every point release.

2 Likes
3

Hi @bsavery,

I already replied on bf-committers here. Thanks for your input! Same goes for @testure, appreciated :slight_smile:

As I mentioned on the ML I hadnā€™t really given much thought to it, but I understand the burden the pace of breaking changes is tough to keep up with.

Iā€™d like to see us do a better controlled approach to our API development, taking the add-on eco-system at large into account better.

As @testure Iā€™d say that adding new API should be always possible, even between subsequent ā€˜minorā€™ releases, but a better scheduled (read: longer adjustment period) deprecation scheme along with improved warnings for add-on devs would in my opinion we should try to accomplish.

I hope we can discuss this further also during the upcoming dev meeting.

Weā€™ll need some concrete proposal from the Python team that addresses your points as well as it can. Itā€™d be nice if @ideasman42, @mont29, @BrendonMurphy, @jacqueslucke, @sybren et al to put their heads together and see if they could whip up something in the coming few weeks. Please? :slight_smile:

1 Like
4

First if all, there are different APIā€™s in Blender with different levels of testing, some are stable and itā€™s unlikely we would break them. Others - such as GPU drawing or tool/gizmo APIā€™s, have not been tested enough, so it might not be in our best interest to try stabilize them before we validate they work well.

Given Blender has quite a few APIā€™s, saying ā€œlets not break the APIā€ isnā€™t a useful statement.

Especially since virtually any change to RNA/Operators could be considered a breakage.

We could have support levels, marking areas of the API:

  • Stable, where breakages are considered bugs/errors, we avoid unless itā€™s absolutely necessary (an important bug-fix relies on modified behavior for example).
  • Unstable, where the API hasnā€™t been well tested and changes may be needed in the future.

The remaining parts of the API will be quite stable - in practice, however we canā€™t guarantee they wont change, since new features in Blender often change the underlying RNA/operators.

5

Itā€™s the usual situation of having to break some eggs to make an omelette. You cannot such a well designed API if you donā€™t break it in every release.

The irony here is that we talking Python, the biggest breaking compatibility scandal in the software of coding that I am aware of. Unfortunately no matter how wise one is rewrite code is the only move forward. Itā€™s annoying sure but the example of APIs that refused to rewrite code and keep backward compatibility no matter what and now none wants to touch with a ten foot pole is too many to even count.

Breaking BPY is not even knew thing it has be happening for the last 14 years I have been around.

6

Couple things here: Sorry using common language idioms to relate to software development is one of my pet peeves. But if youā€™re making an omelette like you say, and keep breaking more eggs, in the end youā€™ll be left with a really messy egg soup :wink:

I would actually argue the opposite. If the api was really well designed it should not need to be continually broken!

I wonā€™t claim to know the whole BPY system as well as @ideasman42 but with regards to ones we used for the render engine there was quite a bit of change with 2.80. Unfortunately a lot of these changes came fast and late, which make it hard for addon developers. (Look at how many released non-beta rendering addons are out nowā€¦ very few)

If we move to a quarterly release with similar changes and addon developers are always playing catchup, a robust addon community is going to be tough. Iā€™m not saying things should never change, but some system for depreciation might smooth it out.

Furthermore, if the intention is to get Blender into studio film systems, thatā€™s going to be quite hard with this rate of change. These guys are still using Python2 in most places (!).

And as a last note, @kilon Iā€™m not sure what youā€™re referring to with Pythonā€™s ā€œBreaking compatibilityā€ scandal? Are you talking about Python 2 to 3? Actually Python is pretty good here. Python 3.X.Y scripts are compatible over all ā€œXā€ versions, and as far as I know, any script written for Python 3.2 will work with Python 3.7 (the aforementioned c-extensions not withstanding). If anything Python 2 to 3 suffered from not moving fast enough encouraging people to move to 3.

1 Like
7

@bsavery could you reply in blender-committers mailing list, since this post has more activity there.

8

sorry about late reply but i was on vacations

Yes I am talking about 2 to 3, even Guido (creator of Python) admitted the way they handled the transition was not ideal and the whole thing was a mistake. Python 3 is not by far the dominant force, but took a long time for projects to migrates. Personally I think the mistake was that they broke too many things and really fundamental things that could not be avoided. On the other hand everyone agrees that overall it was a good choice from the perspective of improving the language. If changes were gradual I think people would have followed along instead we got tons of libraries stuck in Python 2 for so long. Fortunately Python 3 nowadays is almost completely embraced by the community.

Of course with Blender is diffirent because Blender is not a general purpose programming language, but a specialist tool which brings very important improvements in every version so keeping up with the latest version is far more important than keeping up with the latest version of Python.

9

Agree with @bsavery here. I want to acknowledge the hard work the blender devs do in making improvements though, before I add my experience to the thread. Its appreciated by all of us making our addons that Blender exists and that its moving forward in leaps and bounds. Weā€™d not be here without it.

BUT, those leaps have brought pain for us addon developers on occasion. Not the changes themselves, but how they happened and were brought to our attention.

I found out about a lot of the API changes that affected us through our users telling us the addon was no longer working. We were watching the wiki, listening to the podcasts, reading the developer meeting notes, but there was nowhere those changes appeared apart from in bug reports our users asked us to fix. That was annoying to say the least.

Actually it was really frustrating. I have actually asked before, ā€œwhere can I subscribe to changes to the python API?ā€, but really got no solid answer to that question.

Today I just had a one and a half hour support call with a really nice guy who found our addon to be failing to start. I spent the time to diagnose the problem and find a solution for him, only to find out that there has been another change to the python API in 2.81.15. The save handlers now have a new argument apparently. This didnā€™t appear in the wiki 2.81 python API - Blender Development Wiki.

I am positive that the changes are for the benefit of those of us using the API, but when the signature of a handler gets changed, or methods require named rather than positional arguments (or any number of breaking changes for that matter), its going to cause our addons to break. Weā€™re going to have support requests to fix these problems, we have to down tools and go help our community and that takes time away from us doing the important innovative work of improving our software.

I really want blender to keep pushing forward, but I ask, please can the development team put some serious thought into how best to alert the addon development community to changes, before they are released into beta or alpha builds. Our users do sit on the bleeding edge, and we do get complaints from them, spurring us into action.

The addon community as a whole will thank you for bringing us into the loop on breaking API changes.
Finally ā€¦

I welcome those changes, I really do. I donā€™t think thereā€™s any need to guarantee things wonā€™t change, just please tell us if they do change!!

Before our users do would be awesome!

Thanks!

10

I agree with your overall sentiment that API changes should be announced and documented, but come on man- you spent a hour and a half doing tech support for a guy who is using a nightly build and expecting full compatibility from your add-on? The customer may always be right in a perfect world but you should consider setting some boundaries. just a random observation from a fellow add-on dev :slight_smile:

11

Weā€™re seriously dedicated to the user mate :wink: also its a one way weā€™ve found to get notified about new API changesā€¦

12

@JamesCrowther, Iā€™ve asked the devs to update the API Changes section. While we were looking at it the change appears to have happened because of https://developer.blender.org/rBa650258158dd7ad8fa9b6cb1b7da749e30ae15c1 - it is not clear whether the change was supposed to affect the save handlers too. Anyway, Iā€™ll let you know when I have learned more from the devs.

Thanks for raising the issue!

13

Hey mate, thanks! Really appreciate that, good to know what the issue was that triggered, it, I was looking today at 2.81.15 after speaking with that user, as far as I can tell the new argument is None all the time, so not sure what if anything our code has to do, other than accept a new argument which weā€™ll likely not use, unless its planned to have this new arg used in some way.

Thanks!

James

14

As you may have gathered from the commit I linked to this was through a commit related to depsgraph handlers. Possibly unintentional change for save handler. Perhaps there is a fix possible to ensure save handler doesnā€™t get the unnecessary argument. I donā€™t know of plans to have arguments to save handlers (but one never knows 100% about the future :wink: ).

/Nathan

15

@JamesCrowther from checking the commit that caused this, it looks like the breakage was unintentional - at least the code attempts to keep existing functions running.

This looks like it might be a bug, although its hard to tell without an example of what fails (I would be guessing at the issues youā€™re running into).

Please submit a bug report about this with an example that used to work but now fails.

16

Yeah, I like the looks of that new change for the depsgraph, Iā€™ll be looking into the depsgraph a bit more.

For now I can easily enough accomodate additional params by using *args since I donā€™t actually use any of the arguments in the save handler. This works well enough, also noticed that it seems a lot of the handlers have had this issue, render_complete, render_cancel, seems like it could be all the handlers have this new argument, apart from the load handlers, they seem to be unaffected. From the look of the code, this is a template of some sort? So all the handlers using it now have a new argument by the looks of things.

17

I donā€™t think this is intended, while I can contrive some examples that fail, Iā€™d like to know the case you ran into, best report a bug.

18

hey man, yeah sure thing, not hard to reprod here.

import bpy
def handler(normal_dummy_arg):
    print(normal_dummy_arg)
bpy.app.handlers.save_pre.append(handler)

That code, in a new script inside blender, then run script, save a file. The terminal/system console will show an error:

TypeError: handler takes one positional argument, but two were given

Iā€™ll write up a bug shortly.

19

Please crosslink between the bug report and this thread too. That will make reading between the two platforms easier :slight_smile: