PyClone - Asset Management Prototype

Archive User Feedback
1

Hi,

I have been working on an asset engine for Blender, and wanted to get feedback from other developers and users. My design is inspired from all of the blog posts, development tasks, and other users proposals I have read over the years.

I have created a video explaining the concept, and wrote an article with more information.

Video - https://youtu.be/1Kuvf5W_eF4

Article - https://creativedesigner3d.com/2020/05/27/pyclone-asset-manager-for-blender-2-83/

There are several improvements that I would like to make, but I am not sure where to start. Hopefully Blenders Development team can assist, or at least point me in the right direction.

  • New editor space - Create a new sub type of the file browser to store assets.
  • Better way of loading asset previews - We need a way to load assets into the asset manager region. Currently I just display a directory from your hard drive. This can work in a similar way to bpy.utils.previews, but it would be nice if it could load previews faster. https://docs.blender.org/api/2.83/bpy.utils.previews.html
  • Better way of navigating assets - This will mainly be developed in the ToyBox asset library. Once we have a way of loading custom previews then we can determine what should be displayed by using search fields, tag system, and filters.
  • Automate thumbnail previews - Blender creates a preview of the blend file, but no previews are generated for ID types (objects, collections, materials, worlds). It would be nice if Blender can automatically do this in the background.
  • Better way of saving multiple assets - This will mainly be developed in the ToyBox asset library. I have several ideas of how this can be done for the ToyBox Library. I just need more time to develop the functionality.
  • Move some python code to blenders source code - I created a repository of some of the code that I would like to move to Blenders source. https://github.com/CreativeDesigner3D/Blender_Asset_Manager_Prototype
8 Likes
2

Did you know that the Blender Foundation is working on an asset manager?

There is already a lot of stuff in design, planning and implementation:
https://developer.blender.org/T73366

Maybe you can have a deeper look into it and compare it with your project.

1 Like
3

This dev task hasn’t seen a lot of movement and has few or little commits, but the design doc is a good guide. I have tried out the build above a bit, and it has a lot of potential, and more so… it’s a working prototype.

Another blender integrated asset management is with third party systems to take inspiration from is, like ConnectorÑ

I like that system mainly due to, as he mentioned, tags, search, filters, and a harddisk or cloud based system. I think Blender seems to still be debating if they include a cloud based system or local based - both would be ideal. But providing a cloud based service would be costly and hard to implement, which might be the road bump. If it was a local based database, or in the user preferences folder - it might make sense.

I am keeping an eye on this project! I hope it eventually matures well enough to be committed as a patch. All the power to you. @Andrew_Peel

1 Like
4

When I think of Asset management I think of the following questions:

  • how are assets uniquely identified / versioned.
  • how are asset dependencies recorded.
  • how do you handle mutability of assets.
  • how do you handle multiple users committing assets at the same time.
  • how do you handle variants of assets.
  • how do renders fit into all of this.

I think you’re getting your cart before your horse at the moment, you should try to model that data in a database first, and then start looking at UI.

consider something like this (way too simple) table.

id datablock_type filepath version tags inputs
0 texture /storage/ash/texture/v001/ashDiffuse.exr 1 [“wood”, “hardwood”, “diffuse”] None
1 texture /storage/ash/texture/v001/ashSpecular.exr 1 [“wood”, “hardwood”, “specular”] None
2 texture /storage/ash/texture/v001/ashNormal.exr 1 [“wood”, “hardwood”, “normal”] None
3 material /storage/ash/material/v001/ash.blend 1 [“wood”, “hardwood”] [0,1,2]
4 model /storage/ashPlank/model/v001/ashPlank.blend 1 [“plank”, “wood”, “hardwood”] [3]

Imagine when you’re asked to bundle a scene to send to an offsite freelancer. Now you have this data recorded you can simple ask your Asset management system for a list of files to send them.
Untitled Diagram (1)

Usually your UI is going to be very basic, categories as you have them now would be configurable database queries. all it should really care about is whats the thumbnail, whats the filename. Then your trigger a callback which is picked up by your “controller” which will load the data.
Untitled Diagram

This is a very big project you’ve taken on, the reason we see so many Asset Managers is that most shops do this their own way, based on their own worklows. there is rarely a one size fits all approach to asset management.

2 Likes
5

Hi,

There is a lot of really good information here. Thank you for the well thought out reply.

The development so far was focused on the first milestone that the Blender Development Team created in the task.

https://developer.blender.org/T73366

The work I have done is not complete, but this currently works for my workflow, and for others I have spoken with.

Below are answers to your questions. I hope this helps.

  • how are assets uniquely identified / versioned.

    • In the current system they are uniquely identified by filepath, and are not versioned. When we develop the feature to connect to an online repository then we can use a system like git or svn to manage this.

  • how are asset dependencies recorded.

    • Blender already manages dependencies. If you link or append an object with materials, textures, or any other dependencies it will automatically link or append any other assets required.

  • how do you handle mutability of assets.

    • Currently the library is setup to append the assets, so all assets can be modified. I will be adding a feature to allow linking and appending, but I am waiting for the library override feature to be finished before I spend more time on this.

  • how do you handle multiple users committing assets at the same time.

    • I currently don’t. When online repositories are available the same or similar concept to git or svn will be used.

  • how do you handle variants of assets.

    • I currently don’t. I have some ideas on how this would work, but need more development time to outline my plans.

  • how do renders fit into all of this.

    • Sorry but I don’t understand the question.

  • I think you’re getting your cart before your horse at the moment, you should try to model that data in a database first, and then start looking at UI.

    • I am not currently using a database to store assets. They are simply stored on the users file system. A database table like you have listed will be needed when implementing features like tags and versioning.

  • Imagine when you’re asked to bundle a scene to send to an offsite freelancer. Now you have this data recorded you can simple ask your Asset management system for a list of files to send them.

    • All of this information is stored in the blend file. Blender already has built in commands to automatically pack/unpack all files in a Blend File. These commands can be found in the File > External Data menu. Depending on the client and project you may also be able to give them access to the project repository.

  • This is a very big project you’ve taken on, the reason we see so many Asset Managers is that most shops do this their own way, based on their own worklows. there is rarely a one size fits all approach to asset management.

    • This is very true. My first goal is to make a basic system that will work for the majority of small studios and individuals using Blender. The reason why I am focusing on having the majority of the code to save and retrieve assets available from add-ons is so larger studios can develop their own add-ons that are focused on their workflows and projects.

Thanks again for the feedback and info. As development continues concepts that you have listed will certainly need to be used. To start off I wanted to create a simple prototype that can help get this project off the ground.

2 Likes
6

Did you ever heard of this?

In case there is a question of using a database, there might be still a solution to connect your data to another system in the future.

7

Yeah this looks really helpful for teams. Thanks for the info.

1 Like
8

@Pipeliner Kitsu is a production tracking system, not an asset management system. Production tracking tends to deal with Shots, Tasks, and who is working on them. Asset Management tracks all the data generated by a production, and how that data is inter-related. Think Shotgun before they added tank.

@Andrew_Peel the reason I mention renders is that generally speaking your asset management system tracks all data, not just meshes / materials & scenes.

I would be very sceptical about Git or SVN scaling well with binary data. The power of those tools is their ability to diff text. ( do BF use GIT ? )

I’ll take a look at that link you attached! thank you!

1 Like
9

I know but I wanted to point to it anyways for purposes in the future. Kitsu is really good and open source!

10

@rfletchr I have used both Git and SVN to track binary files and found SVN handles larger files better, but as long as you only need to know ‘date_modified’, ‘author’, and ‘last_changed_by’ then it worked for my projects. I’m sure there is better technology, but it’s usable.

I am curious to know what BF and other studios currently use for asset version tracking.

11

@Pipeliner I’m speaking here as a user, which I am, not as a developer (which I am not). One of my (very few) complaints about blender is the lack of an asset management system – ive been using blender since the 2.7 series and there have been lots and lots of promises of an asset management system, and so far we have a lot of talk and a lot of fancy diagrams, and still no asset management system. Ive been using a free addon which is is BARELY adequate, and kinda clunky-- Ive been talking with Andrew quite a bit b/c this is the first system that looks like it has a chance of actually doing the job and getting us a working asset management system (see notes below for comments on CONNECTOR)

Thanks for all the work you do, and please if you can, help Andrew with getting us a working asset management system – I’d suggest cater to the individual user / small shop first and worry about the big production studios as it develops

TIM

3 Likes
12

@Draise

I tried to use Connector, and it has a lot of potential and 1 glaring flaw and one minor one that I ran into. The glaring flaw is that it only accepts .blend files – no obj, no fbx nothing else at all – that alone makes it basically unusable, because I am not going to take the time to convert every single file that I download to blend file and upload it to Connector. the minor fault is thumbnail renders which are ugly

Im really with you in commending @andrew_Peel, and I’ll ask everyone here to please support and / or help him – Im not a developer, dont speak python and dont have time to learn it, so I’ll help where I can doing testing etc, but this project really needs a boost b/c we have gone long enough without a decent asset manager in blender

Thanks
TIM

3 Likes