2024-01-19 Extensions Project

Meetings
,
1

Extension Project

Attendees:

  • Campbell Barton
  • Dalai Felinto
  • Francesco Siddi

Recent changes

Updates since the previous meeting:

  • Blender 4.1-alpha uses Python 3.11.
  • Better error handling when trying to install packages.
  • All unittests are working again.
  • New schema supported.
  • Semantic versioning implemented server-wide.
  • All fake genreated data on the server is complying to the new schema.
  • Kitsu (valid) example add-on updated.

Road to Beta

This meeting was focused on defining the steps required for the Beta launch of the extensions platform.

The idea is to open the extensions.blender.org website for early beta-testers:

  • Moderation will be close to non-existent for the beta.
  • Full package validation will be implemented during the beta.
  • Packages will be wiped once the platform rolls out of beta.

Main upcoming changes:

  • New user interface.
  • Simplify required steps for testing.
  • Support other extensions types (themes at first).
  • Differentiate local from remote repositories.
  • Remove bl_info requirement.
  • Beta mode changes on extensions.blender.org.
  • User documentation.

Pending topics:

  • Moderation strategy.
  • Package specifications (json, manifest, formats).

For a compreensive list of all the planned tasks for this initial version check the Version 1 issue on the tracker.

No-targets for now:

To allow for an early release, a few topics won’t be tackled. Part of the intended feedback from the beta is to assess their importance and priority.

  • There is no way to roll back to a previous version.
  • There are no tracks for versions of extension (e.g., for LTS updates while a main version goes on).
  • Support for ‘icons’ for the extensions.
  • New fancy UI with rich-text and a dedicated editor.
  • Disable/enable themes.

Design topics

Command-line

These are examples of the initial commands we expect during Beta. The system will still be developed using Python, but internally it will be wrapped via Blender command-line arguments.

./blender.exe --extension validate-manifest <manifest.toml>
./blender.exe --extension create <folder-name-with-manifest/>
./blender.exe --extension install-from-disk <package.txz>
./blender.exe --extension uninstall <name>
./blender.exe --extension add-repository-remote <https://blender.extension.org> <?name>
./blender.exe --extension add-repository-local <~/src/my-extensions-no-json/> <?name>

Local vs remote repositories

  • Local repositories are managed by the user, treated as read-only by the users.
  • Their extensions cannot be updated or removed.
  • They can still be enabled and disabled.
  • Local repositories don’t have an index Json file.
  • Proposal: when clicking on + show separate operators (Add Remote Repository; Add Local Repository).
  • Editing the URL of a repository should be an advanced feature (or not exposed).

Extension schema / package specifications

For this discussion, the extension schema is considered to be a combination of the manifest file (its content and its file name), as well as the expected content of the entire package and the package format itself.

There is still an open topic about what should be the extension schema. At the core of this is whether some choices may limit the possibilities in the future, or at the present. This topic is divisive and will tackled next week. Some of the agreed on points are:

  • Packages should be creatable via command-line and easy-to-use tools.
  • The extension schema is to be treated as an API version. Any change on its specifications should be indicated by a version bump following semantic versioning.
  • The server should be aware of which versions it support.
  • Future: Tools to upgrade (or downgrade) an extension version.

The divisive point is whether the initial version of the extension schema should support hand-crafted extension. A hand-craftable extension implies:

  • Manifest name: manifest.toml.
  • Compression format: .zip.
  • Generated: nothing (manifest.toml inside the .zip).

For reference, the current format is:

  • Manifest name: bl_ext_pkg.toml.
  • Compress format: .txz.
  • Generated: bl_ext_pkg_manifest.json (original toml not included in the txz).

A related discussion is whether future versions would need to be hand-craftable, and the constraints this brings. These discussions can be treated separately depending on the merits of making v1.0.0 future-proof versus treating it as an initial version that could change.

Francesco will collect the opinion of other developers to present next week and get to a final decision.

16 Likes
2

About the extensions format, my suggestion would be to only support zip files, since:

  • It’s most convenient to work with across platforms.
  • Data contained in this will either be small (text) or have its own compression (image and blend files), so there is not much benefit to better compression.
  • Blender add-ons and Python wheels also use zip files.
  • Generating .xz files is quite slow in my experience.

Even an uncompressed .zip file like .usdz would be be fine in my opinion, though I don’t think we have a particular reason to do that here.

2 Likes
3

@fsiddi (and @ideasman42) were you thinking on the files flat inside the .zip or under a folder?

For the records, on github if you download a repository as .zip it puts it inside a folder.

4

I talked to Francesco in-person and the proposal is to have a folder inside the zip.

5

Regarding: ZIP vs other compression formats, ZIP is ubiquitous, so I’m not claiming it’s a weak argument, … some counter points which make me prefer formats with better compression (all things considered).

  • Bandwidth considerations:
    • It’s easy for people in countries with very fast internet to discount compression as a factor, however people in other countries (Australia in my case), may have slow download speeds depending on the location of hosting, this is often true even if you pay for fast internet as unless the hosting is within your country. My download speeds from blender.org are quiet slow.
    • Until fairly recently many internet plans in Australia had a monthly download limit, now this is mainly the case for mobile internet plans, although some peoples primary internet plans may be via mobile-internet so it’s still worth considering.
  • Regarding using a ubiquitous format: Users shouldn’t be regularly extracting these files manually (similar to compressed Z-standard .blend files). So I find the argument to use ZIP because people can easily extract it - not to be all that compelling.
  • Regarding compression speeds: I don’t think the speed of compressing files will be slow enough for it to become much of a concern - as extensions aren’t likely to be large archives, people will be downloading extensions far more often than creating them.

Finally … from the user experience perspective, the time to install a single extension isn’t likely to be all that bad whatever format we use. Consider a users could have between 50-100 extensions installed and upgrade them after some weeks/months. In this case the accumulated download times could become more of a factor, where the difference between smaller and larger extensions impacts the user experience.
In the same way we would see it as beneficial to reduce render times on older hardware, I see it as a net-benefit to reduce update times for people with slower internet.

2 Likes
6

especially in places further away from the b.org servers (like you an me) latency gets you long long before the actual datatransfer size will. That’s why SVN is such a horrid experience, being able to request those 50-100 updates in a single request rather than 50-100 individual conversations will help more there than any data compression will.

1 Like
7

The extensions website should use a CDN, I don’t think latency should be so much of a problem. As far as I know download.blender.org should be quite fast now that it’s behind one. Assuming the file has been cached, which I think will be the case if the extensions website is really used. For example try downloading a file once, and if it’s not very fast try a second time once it’s in the cache.

I think download size is important. My main point i s that I don’t expect it to make so much difference. But I might be wrong. I just expect it will be less than 10% difference on average and would not make that trade-off.

Guidelines or even automation for image file compression, making add-on update parallel to avoid latency, incremental updates, those are the kinds of optimizations that I would expect to make a big difference.

8

From my tests with existing add-ons it’s around a 20-25% improvement (zip -9 compared with xz -9). Updates are already parallel.

Regarding Delta Updates

Delta updates of course help if there are many static binary files, however unless extensions main purpose is to store static binary files this seems like unwarranted overhead for the implementation & maintaining a server.

It’s interesting that most Linux distributions aren’t using delta updates, Arch supports it but most repositories don’t enable it, Fedora supports it but is considering dropping it.

Unlike binary packages Blender extensions aren’t likely to contain many compiled-binary files, instead a large image, blend or bundled python-wheel may remain unchanged so there could be greater advantage to delta-updates in our case, nevertheless the issues noted in the linked post are worth considering.