Proposal: Generate stub files for blender

Proposal: Generate stub files for blender

I would like to improve blender python auto completion setup. Community created projects to mitigate this issue (see below) but I think this is very basic functionality that should be shipped with blender. I am willing to work on topic in the next months to come. In this post I want to complete design and get feedback.

Brief: generate official blender python stub files (.pyi) as part of release process. Stub files are official way to provide type information for auto completion.

Problem and solutions overview

The problem is that most IDEs can not inspect blender python API, because it is compiled from C. Therefore auto completion sucks.
The problem we are facing in Blender is problem of all c-compiled python modules and there are some solutions to the problems:

Argument clinic (from python developers)

Argument Clinic is a Python internal tool that adds data about function signature to the c-compiled function. This data is stored in __text_signature__ attribute and it allows for the use of inspect.signature.

Pros:

  • it works really well, somebody else has done the job

Cons:

  • it is tightly integrated with python source code, integration with blender will be hard (code example: _abc.c.h). It is all or nothing situation. In theory we can take only elements from Argument Clinic, but including only part of the pipeline will be very error prone in long run.
  • it is python internal tool - the future of the tool is uncertain
  • it works only with make

Example:

# Python/clinic/bltinmodule.c.h:533 <- strange name - see the docs
PyDoc_STRVAR(builtin_len__doc__,
"len($module, obj, /)\n"
"--\n"
"\n"
"Return the number of items in a container.");
>>> len.__text_signature__
'($module, obj, /)'
>>> print(inspect.signature(len))
(obj, /)
>>> inspect.signature(bpy.utils.register_class)
ValueError: no signature found for builtin <built-in function register_class>

Stub generators

There are not that many generators. The ones I tried does not give great results in general, partly because they are mean to be hand tweaked and partly because blender has complicated python API.

  • mypy (stubgen to be exact) - I compiled blender as module and managed to get it running - the result is not great, stubgen is meant to be hand tweaked
  • make-stub-files - does not support c-compiled module
  • python-skeletons - todo investigate
  • pygenstub - todo investigate

Blender specific projects

  • Blender-PyCharm (and forks) - one file implementation based on sphinx_doc_gen.py (this script is from blender repo, but it is quite hard to read and a bit under-documented). How it works:
  1. list blender modules
  2. Inspect modules (dir, __dict__, inspect.members)
  3. parse each element’s __doc__ (which is formatted in rst sphinx fashion) to get function signature
  1. list blender modules
  2. based only on documentation generate python files (this same files are passed to generate sphinx documentation)

Pros:

  • not so much work, single file implementation is possible (or as addon)

Cons of the above:

  • every mistake in function signature syntax in documentation will cause script to fail, this is a lot of maintenance, see fake-bpy-module/src/patches

edit: @jacqueslucke mentioned also: code autocomplete and blender vscode.

Conclusion and work plan

For the last month I was experimenting with different solutions, the best seem to be either full switch to Argument Clinic or adapting Blender-PyCharm solution. Personally I prefer the later but I would love to discuss that.

How to do it

I have a few options on my mind:

  1. ship with blender addon/operator with ability to generate blender-stub package
  • pros: it takes minimal amount of space
  • cons: you need blender to install stub files
  1. when releasing blender, publish package on pypi.org
  • pros: standard way of distributing python modules
  • cons: another thing for dev to remember about
  1. Ship blender with pregenerated stub files:
  • pros: we can use standard frameworks (like jedi, rope) for console and text editor completion.
  • cons: it will add megabytes to release (but text is highly compress-able)
  1. Create language server addon specifically for Blender (based on framework like pyls), thx for tip @rfletchr. This addon would provide things like completion, diagnostics, hoover information, formatting etc. In this solution we do not need to create stub files, as completion can be done inside blender.
    I need to investigate this topic as I do not know how it integrates with IDEs, what happens if you try to use external packackes from pip and autocompletion from Blender…
  • Pros: we can autocomplete collection, object, scene names; no need to generate stub files
  • Cons: Probably client must be implemented as addon, separately for every IDE, for example for visual studio docs;
  1. Create custom script for pasing blender source files and extract information about function signatures. This solution is inspired by Argument Clinic and blender scripts like makerna.
12 Likes

I’ve also worked on this in the past in two projects: code autocomplete and blender vscode.
However, I’m currently using fake-bpy-module myself, because from the ones I’ve tried, it was the easiest to install and the most comprehensive.

I haven’t thought about it in too much detail right now, so maybe my opinion will change.

It seems the best approach is to ship pregenerated stub files with Blender. Then people can simply add something like path/to/blender/2.92/python_stubs to their include directories and it should just work. At least that is easy in vscode by modifying the python.autoComplete.extraPaths setting.

This also means that we have to build the stub files as part of building Blender. It does not sound too hard to get the basics working. The benefit of building the stubs as part of normal compilation is that we have the most information available. A separate addon would only have access to information available to the Python API, which might not be enough in all cases.

Publishing the stubs to https://pypi.org/ afterwards is possible, but I don’t think it is necessary when they are shipped with Blender. Also, publishing them there might lead to annoying version conflicts later on.

I’ve used jedi in Blender in the Code Autocomplete addon. It works and helps in some cases, but it might not be worth the effort. I think the focus should be on improving the integration with separate editors/IDEs.

3 Likes

some stub generator links are empty

thanks for info, fixed (missing https:// prefix)

I’m not enitrely sure if jedi sources information from the interpreter its running within or if its strictly gathering information from its own AST. If it is aware of runtime objects you could run pyls within blender and use it as the language server.

1 Like

Hi @grzelins, this would also help to port my addons to a new version Blender.
At the moment we use Blender v2.73, official auto complete would help to check if the ported code calls the Blender API correctly.

1 Like

I have some updates. I am focusing on stubbing these internal blender modules:

import _bpy  # not finished
import mathutils
import bpy_path
import bgl
import bgf # not finished
import bl_math
import imbuf
import bmesh
import manta # I do not have it build right now, not tested
import aud # I do not have it build right now, not tested
import _cycles # I do not have it build right now, not tested
import gpu
import idprop

I created initial implementation (tested on master branch, python 3.7):
blender_stub_gen.log.txt (51.5 KB)
blender_stub_gen.py.txt (33.3 KB, ~900 lines)
generated files on my dropbox (less than 1Mb, devtalk does no allow for zips, add this to you PYTHONPATH for autocomplete)

Right now I think the best approach is:

  • this framework is based on visitor, so that it can be adapted for generating docs
  • I am trying to do a little cleanup in _bpy, do not worry if you see msgbus in different place
  • do not parse rst documentation of function, instead add __text_signature__ to docstrings (so that inspect.signature does not fail). I find that parsing docstring is pretty unreliable, example from msgbus
PyDoc_STRVAR(
    bpy_msgbus_subscribe_rna_doc,
    "subscribe_rna(key, owner, args, notify, options=None)\n"
    "--\n\n"
    ".. function:: subscribe_rna(key, owner, args, notify, options=set())\n"
    "\n" BPY_MSGBUS_RNA_MSGKEY_DOC
    "   :arg owner: Handle for this subscription (compared by identity).\n"
    "   :type owner: Any type.\n"
    "   :arg options: Change the behavior of the subscriber.\n"
    "\n"
    "      - ``PERSISTENT`` when set, the subscriber will be kept when remapping ID data.\n"
    "\n"
    "   :type options: set of str.\n");

renders in pycharm as:


Not ideal, but getting there. is is possible for pycharm to automatically read and apply type from :type param:, but proper syntax is needed. The .. function directive is also a special symbol in rst and pycharm does not know what to do with it.

And by the way, scripts in blender/release/scripts/modules are pretty terrible to work with. Before I finish stubbing _bpy module I need to dive into rna_info.py. But in meantime I added type hints for rna_info.py/D9356

Do you know what is bgf module?