2024-09-05 Flamenco Dependency Packing / BAT

Meetings
1

Agenda

  • What exactly is the problem?
  • What is the preferable solution?
  • How long would this take?
  • Who can work on this?
  • When can this be prioritized?

Attending

  • Bastien Montagne
  • Sebastian Parborg
  • Sergey Sharybin
  • Simon Thommes
  • Sybren Stüvel

Problem

  • Flamenco uses BAT for packing all dependencies of a job, which analyzes the .blend file separately from Blender and does not properly take nested dependencies into account.
  • So far this has not been a problem, since BAT did still catch all dependencies due to how Blender references (/referenced) nested dependencies directly
  • With recent refactors this is becoming an issue with dependencies that are missing from the pack
  • More problems will continue to come up with how BAT is analyzing dependencies without loading the .blend files

Proposed Solution Options

  • Use Blender internal functionality for packing in the Flamenco Addon instead of BAT ( → long term)
  • Make BAT use Blender’s internal dependency information for packing, rather than relying on independent analysis ( → short term)
  • Implement recursive dependency analysis in BAT

Meeting Outcome

There is a preferable long term solution to stop relying on BAT, since currently this continuously put the burden on keeping up with Blender development to find all necessary dependencies.
Since sufficiently implementing this solution would require a large effort of core development regarding multiple different big topics, this is currently not a feasible target to be prioritized.
There is a sufficient short term solution to keep using BAT but replace collecting the dependecies by exposing existing Blender functionality to the python API and using the active Blender instance when submitting a job with Flamenco.

Short Term Solution
(within few months)

  • we keep using BAT for remapping and editing the paths without loading the files
    • Bastien exposes existing filepath iterator system to python API (few days of work - either still before holiday or right after)
    • Sybren implements Flamenco (python) functionality to pass information about dependencies and filepath remapping from Blender’s already loaded instance when submitting a job to BAT

Long Term Solution
(big breaking changes for upcoming series or beyond)

  • Avoid the necessity of using BAT as a separate system that needs to be kept up-to-date with any Blender changes
  • Add required functionality to Blender itself
  • Avoid building viewport depsgraph on file-load with command line execution (for performance when recursively going through libraries)
  • Consistent pointers for hashable blend files (for shaman)
5 Likes
2

What’s meant by this? I think I get the big picture but not how we’d actually be able to avoid reallocating data on file read for example.

3

One thing I liked about Clarise was that you could ask it to list all the dependencies from a given project.
something like blender --list-dependencies myproject.blend of course this could also be a python script.

  • no need to load the full blend file. can reuse logic blender already contains.
  • output could also mention the id blocks it uses from the dependency.
  • bat could be adapted to use this information to be more aware.
2 Likes
4

I’m not super firm on the technical details here, but from what I understood this is about moving away from a full on memory dump for the .blend file format and more to a serialized format.
This is obv. a major compatibility breakage and not to be done any time soon probably. But it would be part of the ‘full and proper’ solution to this issue.

1 Like