Proposal: Generate stub files for blender

[grzelins]

Proposal: Generate stub files for blender

I would like to improve blender python auto completion setup. Community created projects to mitigate this issue (see below) but I think this is very basic functionality that should be shipped with blender. I am willing to work on topic in the next months to come. In this post I want to complete design and get feedback.

Brief: generate official blender python stub files (.pyi) as part of release process. Stub files are official way to provide type information for auto completion.

Problem and solutions overview

The problem is that most IDEs can not inspect blender python API, because it is compiled from C. Therefore auto completion sucks.
The problem we are facing in Blender is problem of all c-compiled python modules and there are some solutions to the problems:

Argument clinic (from python developers)

Argument Clinic is a Python internal tool that adds data about function signature to the c-compiled function. This data is stored in __text_signature__ attribute and it allows for the use of inspect.signature.

Pros:

• it works really well, somebody else has done the job

Cons:

• it is tightly integrated with python source code, integration with blender will be hard (code example: _abc.c.h). It is all or nothing situation. In theory we can take only elements from Argument Clinic, but including only part of the pipeline will be very error prone in long run.
• it is python internal tool - the future of the tool is uncertain
• it works only with make

Example:

# Python/clinic/bltinmodule.c.h:533 <- strange name - see the docs
PyDoc_STRVAR(builtin_len__doc__,
"len($module, obj, /)\n"
"--\n"
"\n"
"Return the number of items in a container.");

>>> len.__text_signature__
'($module, obj, /)'
>>> print(inspect.signature(len))
(obj, /)
>>> inspect.signature(bpy.utils.register_class)
ValueError: no signature found for builtin <built-in function register_class>

Stub generators

There are not that many generators. The ones I tried does not give great results in general, partly because they are mean to be hand tweaked and partly because blender has complicated python API.

• mypy (stubgen to be exact) - I compiled blender as module and managed to get it running - the result is not great, stubgen is meant to be hand tweaked
• make-stub-files - does not support c-compiled module
• python-skeletons - todo investigate
• pygenstub - todo investigate

Edit: recently I discovered that pytype, MonkeyType also can generate stub files, but did not have time to investigate.

Blender specific projects

• Blender-PyCharm (and forks) - one file implementation based on sphinx_doc_gen.py (this script is from blender repo, but it is quite hard to read and a bit under-documented). How it works:

1. list blender modules
2. Inspect modules (dir, __dict__, inspect.members)
3. parse each element’s __doc__ (which is formatted in rst sphinx fashion) to get function signature

• fake-bpy-module - successful, but heavily object oriented implementation

1. list blender modules
2. based only on documentation generate python files (this same files are passed to generate sphinx documentation)

Pros:

• not so much work, single file implementation is possible (or as addon)

Cons of the above:

• every mistake in function signature syntax in documentation will cause script to fail, this is a lot of maintenance, see fake-bpy-module/src/patches

edit: @jacqueslucke mentioned also: code autocomplete and blender vscode.

Conclusion and work plan

For the last month I was experimenting with different solutions, the best seem to be either full switch to Argument Clinic or adapting Blender-PyCharm solution. Personally I prefer the later but I would love to discuss that.

How to do it

I have a few options on my mind:

1. ship with blender addon/operator with ability to generate blender-stub package

• pros: it takes minimal amount of space
• cons: you need blender to install stub files

2. when releasing blender, publish package on pypi.org

• pros: standard way of distributing python modules
• cons: another thing for dev to remember about

3. Ship blender with pregenerated stub files:

• pros: we can use standard frameworks (like jedi, rope) for console and text editor completion.
• cons: it will add megabytes to release (but text is highly compress-able)

4. Create language server addon specifically for Blender (based on framework like pyls), thx for tip @rfletchr. This addon would provide things like completion, diagnostics, hoover information, formatting etc. In this solution we do not need to create stub files, as completion can be done inside blender.
   I need to investigate this topic as I do not know how it integrates with IDEs, what happens if you try to use external packackes from pip and autocompletion from Blender…

• Pros: we can autocomplete collection, object, scene names; no need to generate stub files
• Cons: Probably client must be implemented as addon, separately for every IDE, for example for visual studio docs;

5. Create custom script for pasing blender source files and extract information about function signatures. This solution is inspired by Argument Clinic and blender scripts like makerna.

[jacqueslucke]

I’ve also worked on this in the past in two projects: code autocomplete and blender vscode.
However, I’m currently using fake-bpy-module myself, because from the ones I’ve tried, it was the easiest to install and the most comprehensive.

I haven’t thought about it in too much detail right now, so maybe my opinion will change.

It seems the best approach is to ship pregenerated stub files with Blender. Then people can simply add something like path/to/blender/2.92/python_stubs to their include directories and it should just work. At least that is easy in vscode by modifying the python.autoComplete.extraPaths setting.

This also means that we have to build the stub files as part of building Blender. It does not sound too hard to get the basics working. The benefit of building the stubs as part of normal compilation is that we have the most information available. A separate addon would only have access to information available to the Python API, which might not be enough in all cases.

Publishing the stubs to https://pypi.org/ afterwards is possible, but I don’t think it is necessary when they are shipped with Blender. Also, publishing them there might lead to annoying version conflicts later on.

I’ve used jedi in Blender in the Code Autocomplete addon. It works and helps in some cases, but it might not be worth the effort. I think the focus should be on improving the integration with separate editors/IDEs.

[ankitm]

some stub generator links are empty

[grzelins]

thanks for info, fixed (missing https:// prefix)

[rfletchr]

I’m not enitrely sure if jedi sources information from the interpreter its running within or if its strictly gathering information from its own AST. If it is aware of runtime objects you could run pyls within blender and use it as the language server.

[EmileSonneveld]

Hi @grzelins, this would also help to port my addons to a new version Blender.
At the moment we use Blender v2.73, official auto complete would help to check if the ported code calls the Blender API correctly.

[grzelins]

I have some updates. I am focusing on stubbing these internal blender modules:

import _bpy  # not finished
import mathutils
import bpy_path
import bgl
import bgf # not finished
import bl_math
import imbuf
import bmesh
import manta # I do not have it build right now, not tested
import aud # I do not have it build right now, not tested
import _cycles # I do not have it build right now, not tested
import gpu
import idprop

I created initial implementation (tested on master branch, python 3.7):
blender_stub_gen.log.txt (51.5 KB)
blender_stub_gen.py.txt (33.3 KB, ~900 lines)
generated files on my dropbox (less than 1Mb, devtalk does no allow for zips, add this to you PYTHONPATH for autocomplete)

Right now I think the best approach is:

• this framework is based on visitor, so that it can be adapted for generating docs
• I am trying to do a little cleanup in _bpy, do not worry if you see msgbus in different place
• do not parse rst documentation of function, instead add __text_signature__ to docstrings (so that inspect.signature does not fail). I find that parsing docstring is pretty unreliable, example from msgbus

PyDoc_STRVAR(
    bpy_msgbus_subscribe_rna_doc,
    "subscribe_rna(key, owner, args, notify, options=None)"
    "\n--\n\n" // <- this marks the end of __text_signature__
    ".. function:: subscribe_rna(key, owner, args, notify, options=set())\n"
    "\n" BPY_MSGBUS_RNA_MSGKEY_DOC
    "   :arg owner: Handle for this subscription (compared by identity).\n"
    "   :type owner: Any type.\n"
    "   :arg options: Change the behavior of the subscriber.\n"
    "\n"
    "      - ``PERSISTENT`` when set, the subscriber will be kept when remapping ID data.\n"
    "\n"
    "   :type options: set of str.\n");

renders in pycharm as:

Not ideal, but getting there. It is possible for IDEs (tested on pycharm) to automatically read and apply type from :type param:, but proper syntax is needed. The .. function directive is also a special symbol in rst it is not rendered.

Here is example of proper docstring:

And by the way, scripts in blender/release/scripts/modules are pretty terrible to work with. Before I finish stubbing _bpy module I need to dive into rna_info.py. But in meantime I added type hints for rna_info.py/D9356

Do you know what is bgf module?

[KZekai]

However, I’m currently using fake-bpy-module 3 myself, because from the ones I’ve tried, it was the easiest to install and the most comprehensive.

Yes, but also not really. This is how pylance with basic mod enabled looks like:

It is a shame really.

[KZekai]

@grzelins You still working on this? I suggest drop the docs, it just too much work and you always can look them up.
I think it would be better rather to have full proper structure for the module so you can get not just autocomplete but proper statical type checking (for Pyright/Pylance statical analysis). the fake-bpy-module one mentioned earlier avoids this kind of stuff: https://github.com/nutti/fake-bpy-module/issues/84

[jmv]

I am learning the Blender API. This is difficult mainly because there is no help in the code editors.
thanks to fake-bpy-module, blender_vscode and Pylance I finally manage to do something.
fake-bpy-module is not perfect, why are stub files not generated with documentation?

[grzelins]

@KZekai Yes I am still in progress. Right now I have general framework.

Unfortunately getting proper statical type checking with blender is harder than you think.
First many blender structures are not meant to be initialized by user thus running __init__ method raises an error and it is very hard to determine which ones. Similar with __add__ is defined for every python object, but there is no way of knowing which ones are actually subclassed (todo check this).

There are also 2 issues with extracting types from blender api:

• functions argument and return types does not have strictly defined type syntax. There are :type anotations in docstrings but do not follow any strict syntax rules.
• container type hints are very hard to get, see hypotetical example:

class Collection(bpy_struct): ... # true signature

bpy.context.scenes: bpy.types.Collection[bpy.types.Scene] = ...  # invalid syntax

this is the same reason why we have in python list and typing.List. In theory we can introduce special module for blender typing, lets say bpy_typing but it becomes very complicated in practise I am trying different things right now and looking for hints in future versions of python. No easy answers so far.

T = typing.TypeVar('T')
class Collection(typing.Generic[T]): ... # for typing info only. All methods must be (hand?) written to support T type var

bpy.context.scenes: bpy_typing.Collection[bpy.types.Scene] = ...  # valid, but requires special care to be usefull

@jmv I am actually using/modyfing scripts used to generate documentation. My personal feeling is that blender developers are not that convinced about static typing in python (or maybe it is just the matter of priorities). Script that is used for generating docs is hard to work with.

[jmv]

I got the idea to use a modified version of the documentation build to extract json files rather than html. (Sphinx allowing it)
Then code a custom generator in “pyi” format.
Unfortunately my knowledge of English, Python, Blender and Sphinx is not sufficient. I dropped.
In any case, well done that you are thinking about this problem

[KZekai]

this is the same reason why we have in python list and typing.List. In theory we can introduce special module for blender typing, lets say bpy_typing but it becomes very complicated in practise I am trying different things right now and looking for hints in future versions of python. No easy answers so far.

Oh I see now. Yea, this is a nuisance to work that way. When I asked nutti he never specified what is exactly complicated about adding types but now I see.
Yeah, hmm, I’m not sure myself what would be the best approach here.

[grzelins]

I got the idea to use a modified version of the documentation build to extract json files rather than html

Maybe. Even if you have it as json, how do you create usefull python syntax from sth like: bpy.data.user_map

# bpy.data.user_map
def user_map(*args, **kwargs):
    """.. method:: user_map([subset=(id1, id2, ...)], key_types={..}, value_types={..})
    
       Returns a mapping of all ID data-blocks in current ``bpy.data`` to a set of all datablocks using them.
    
       For list of valid set members for key_types & value_types, see: :class:`bpy.types.KeyingSetPath.id_type`.
    
       :arg subset: When passed, only these data-blocks and their users will be included as keys/values in the map.
       :type subset: sequence
       :arg key_types: Filter the keys mapped by ID types.
       :type key_types: set of strings
       :arg value_types: Filter the values in the set by ID types.
       :type value_types: set of strings
       :return: dictionary of :class:`bpy.types.ID` instances, with sets of ID's as their values.
       :rtype: dict
     """
    ...

so right now we can do that, but it would not be that usefull…

[jmv]

Like I told you, I gave up on this idea
(json or xml are examples, although Pickle seems closest to the AST tree internally generated by Sphinx)

[grzelins]

Update: I made first kind of usable implementation. In my opinioin concept works, but more work is needed, TODOs are outlined in task https://developer.blender.org/D9852

Currently generated files weight ~10Mb and can be downloaded here https://www.dropbox.com/sh/xzktk8fbrhldih2/AACGPemH1FyTip_LSu61q-Nca?dl=0

[mysticfall]

I also started a FOSS project called bpystubgen for the same purpose.

Currently, it is used to generate PEP-561 compliant API stubs for both Blender and UPBGE.

It’s also very fast (generates stubs for Blender in about 30 seconds) so it’s viable to integrate it with a CI process and you can easily fork and change it, in case you find anything missing.

Sorry to revive an old topic to make a self promotional post. But I believe my project can be a good addition to the suggested solutions to the problem.

[grzelins]

Ok. Will look into that. I had to take a break from this project because of studies. Maybe I will find some time to revive it in several months. I have also some new ideas. Will see.

But still you take information from documentation, I proposed solution for taking information directly from python interpreter. Not sure which one is better, just saying

[mysticfall]

Both of the approaches have their respective pros and cons but I believe the documentation-based route has one significant advantage, which is the ability to provide type hints.

Of course, using metadata provided by the interpreter can provide accurate information as to what members an module has, for example, which might be missing in the API documentation that is maintained manually.

However, Python has no built-in mechanism to keep or retrieve type information of API members. It means, using what information provided by the interpreter alone, the best you can do is providing autocompletion support for just 1 level of invocation chain.

Consider this scenario: Try importing the bpy module in your IDE, then reference an object (i.e. bpy.types.Object) property like location:

In order to get a result like this, your IDE must know that the type of bpy.data is bpy.types.BlendData which has objects attribute which implements mapping protocol regarding bpy.types.Object type.

With information retrieved by running an interpreter on actual module, however, you only know that bpy has data attribute of an unknown type, and notthing more.

As such, I personally find that generating API stubs based on the documentation is a better approach, even though it may miss a few undocumented APIs which can always be remedied over time.

P.S. : It may not concern everyone. But the interpreter based approach also has another serious limitation for those who use UPBGE like me. UPBGE modules like bge are created on the fly so you cannot reference them without running the game engine.

[grzelins]

Interesting points. I am not sure how you got that "Cube" completion, I will look into your implementation.

In diff D9852, I also noticed the problem about generics, aka use case:

bpy.data.scenes: bpy.types.bpy_prop_collection  # this is possible
bpy.data.scenes: bpy.types.bpy_prop_collection[bpy.types.Scene]  # syntax error, bpy_prop_collection does not support []

and mentioned 2 or 3 possible solutions.

UPBGE modules like bge are created on the fly

Yes but you probably want to manually list the modules that you want to stub anyway. Many of blender modules are also dynamic.

Regarding typing information I cash in on the fact the IDE can get type from docstring, this is perfectly hinted at least in Pycharm. But other IDEs I think does not support it, so I ended up extracting typing information from docstring anyway. And blender does not have any concrete style and this is a pain

def foo(boo):
    """
    :type boo: bool
    :rtype: list[dict[str, Union[str, T]]]
    """