Lib2to3 missing , on purpose?

Archive Python API
1

Hi guys, my first post here, so hope is the right place where to have some information.
I found some posts from back in the days about this :


So for what I do understand seems has been done on purpose to keep the blender distribution as small as possible to download.

As much as I do understand this, I feel like a module that helps the backward compatibility between python 3 and python 2 would have some meaning to be in blender to simplify the porting of some modules, or at least allow to have some modules able to work on both.

So long story short, is there any hope to see this module back in ?
Thanks.
L.

2

My understanding is that lib2to3 is for converting code, and that is not something that should happen at runtime in an addon. The Python documentation also mentions that it does not have a stable API and seems to be mostly for internal usage in the 2to3 tool, so having an addon import it seems wrong?

3

Hi Brecht, thanks for the quick reply, really is appreciated.

lib2to3 allows to use the module past from the module future.
This allow to have code which works in both.

I’ve been trying to install pip and through pip install the future module, but the lacks of the lib2to3 makes the past module break badly.

Of course the best would be to have py3k only, but sometimes you just have to deal with both.

4

Blender has been using Python 3 for years. Why would you support Python 2 in your add-on?

5

Hi Sybren, I’m aware Blender has been moving to py3k many years ago, but the vfx industry hasn’t yet :slight_smile: (sure, it’ll happen in the next few years, but it’ll be quite a slow process at the beginning).

In my specific case, is because I need to keep the compatibility with 2 as the add-on I’m writing to allow blender to connect with a commercial asset management system for Vfx.

Having the ability to load python 2 (with future compatibility) to blender would simplify
as having a separate branch and tagging and testing just for py3 is not really doable at the moment, or hard to justify at least :slight_smile:

Thanks.
L.

6

It’s hard to talk about “the vfx industry” as being “not Blender”; discussions are bound to go awry then.

Python 3 has existed for about a decade now. “At the beginning” is years and years ago, and Python 2 is end-of-life in two years.

In my specific case, is because I need to keep the compatibility with 2 as the add-on I’m writing to allow blender to connect with a commercial asset management system for Vfx.

Isn’t it possible to do that using something that’s independent of the Python version? Things like shared memory or socket connections come to mind.

7

I think the Vfx industry is looking at blender with more and more interest every day, expecially with the approaching of 2.8. Up to now was relegated to just a simple tool to be kept far from the main pipelines.

That said , the same industry is really slow in adopting new technologies, especially big players.
(so even if python2 is deprecated in few years, it’ll take many mores before all the production tools will move over).

Having the ability to smooth the transition, will simplify the adoption of blender or at least would allow to start testing before time, which is important, otherwise will take +10 years more to for the offices to actually evaluate any benefit.

Of course, is possible to find workaround, but that’s not my aim to find these.
I want to be able to push blender in the vfx pipeline, and use it along with the rest of the tools.

Hope it make sense from a production point of view.

On the other hand , what problem could cause to blender to have this library back ?

Cheers.
L.

8

Can you point to a real world use case for using lib2to3 within Blender?

Where using Blender’s Python is somehow a significant gain over installing Python 3 and using that?

Edit: I see this is an indirect dependency from PIP, not something you want to use directly.

In the next years this will be used less and less, however Blender’s Python should allow PIP usage - so there is an argument to be made for including it.

Out of curiosity which library are you trying to use?

9

Well… that’s not quite accurate. Just read Use in the media industry on Wikipedia.

If you want to make claims about “the industry”, please do so based on facts and with references, not with just vague statements.

That said , the same industry is really slow in adopting new technologies, especially big players.

Yes, that’s what I’ve seen too :wink:

I want to be able to push blender in the vfx pipeline, and use it along with the rest of the tools.

That’s a laudable goal. However, I don’t think that the solution lies in supporting nearly-obsolete software in Blender.

10

I said Vfx industry, not media industry… I do not know much about the others.

Not sure it counts, but I’ve been working as lead pipeline developer in for the past ~10 years, before joining a development company. Hope by now, got a bit of a grip on how does work :wink:

Anyhow, if there’s no interest from your side, do not see why should be on mine.
Just sorry to see there’s no much room for discussion , thought having more options was a plus, not an issue.

My question though still stand, apart from not wanting, what’s the downside from a blender product point of view ?

11

We can always discuss. However, having more options also implies more effort to keep things up-to-date, to fix bugs, write & maintain documentation, fixing & testing more sources of incompatibility, so in all it means more time of developers. And developer time, that’s not something we’re in ample supply of. If you’re willing to put in that time, that’s great!

12

Hi Sybren,

this library itself, what problem brings , if any ?

All I have found is just to get it removed to save space, I have found no trace of issues brought in for having it bundled. Now if that does bring issues to blender code base I’m happy to step back, but if does not, then would like to understand why is such an issue to try helping an industry to adopt a tool made for it.
Also please do not mistake, I’ve been using blender and following its development for many years now , I like it a lot, and I’m not payed by anyone to do this. Although I’m working in this company, I’m doing this on my free and spare time, because I think Blender deserve a bigger visibility and usage, and I think is getting mature enough for being at least tested on some types of productions.

As a developer myself (please bear that in mind), I don’t honestly see how something which is almost python code base would imply spending more time testing blender or for the developers, we are talking about a backward compatibility module. In case will be up to the coder to ensure the add-on python code works nicely on all the versions of python, I’m not asking to have the blender api working on python 2 ! In that case I would he happy to call myself in to help or simply fool.

Cheers.
L.

P.s
For now I’m putting my time trying to do something for what I think is helpful for blender and its community.
might be not enough, for sure, but is what i can bring for now.
With some interest on helping the industy, might come back some interest from the industry to help blender.

13

As Campbell also asked, can you point to a real world use case for using lib2to3 within Blender?

14

Hi @sybren thanks for not giving up on me on this , and thanks to @ideasman42 to get his head in as well.

The problem is not about the 2to3 command that automatically convert the code, for me that can be haply removed.
The issue comes from the python “past” module in the “future” module, which require the presence of the lib2to3 module.

If “future” is used in python2 to be able to run code in 3, the “past” module is used to run python 3 code in 2.
The lib2to3 is a direct dependency of this module, and therefore even installing with pip future, is not possible to use the “past” module.

one of the most basic example I can think of is :

from past.builtins import basestring

you can test, installing pip in the blender python
install future through pip, and then run the above code from within blender.

without the lib2to3 this code will break.
(you need to dig in the stack trace to find it)

I hope this help.
Cheers.
L.

P.s
here some further reading for you if you are interested in this module:

15

Thanks for the info. Can you give a “big picture” example?
What are you trying to solve?

Of course you may want to import a module that uses “past” or “2to3” for any kind of module, nevertheless I’d like to know the use case here.

I’m just not sure which popular/useful module you would want to import into Blender which happens to be Python2 only _and_ will works usefully with an automated update.

Typically automated (even semi-automated) updates only work with simple test cases and aren’t enough to load more complex projects.

16

Hi @ideasman42, thanks for getting back to me.

I think the big picture you are asking for, can be found in this phrase of the link I sent earlier:

to aid in providing minimal-effort Python 3 support for applications using libraries that do not yet wish to upgrade their code properly to Python 3, or wish to upgrade it gradually to Python 3 style.

So , why is important to allow “wish to upgrade it gradually to Python 3 style” ?

Over the years there’s been a big effort to try to find a common ground on libraries and dependencies between the DCC production companies, to make sure all the software can work together, and therefore the vfxplatform has born (https://www.vfxplatform.com/) to help having all aligned. (Foundry, Disney, Autodesk etc… are participating)

Vfx offices has been investing quite lot of money in developing many custom tools, and integrating software in each other using mostly python2 (leave alone qt and pyside), and shaping this way their pipelines and workflows. And due to the fact these places are almost always in crunch time for deliveries, there’s not much of a willing to risk to break their workflow or pipeline and therefore production, so the adoption of new , and breaking technologies has to be taken with care , and abundant time ahead. Still they are willing to look into new technologies, but ensuring the effort is worth doing it. You can consider them like elephants, with a lot of memory , pretty robust, but slow to move.

And this is the point.
In order to have tools ported, you need to go in various steps.

  1. ensure your python2 code is robust and does what should be doing.
  2. make the code compatible to py3k as well as python2 (future/past module) and ensure all works.
  3. once all the DCC application and tools supports py3k , remove the past dependencies and finally be in a better place.

The step 2 and step 3 though, could be taking way longer than the livespan support for python2, as they need to wait for ALL the moving parts (mostly DCC applications) to be compatible. In the meanwhile, bugs will be found, and improvements will be added. And you do not want to have multiple branches to keep in sync.

Where does blender fit in all this, you may ask ?
Well, I can see the new blender version is getting more and more attention, but in order to actually put it into the workflows, or even just evaluating it , it has to be able to fit somehow in existing pipelines and tools, and due to the fact is the only one using python3 (for now) , the python libraries at least has to be able to work in both worlds.

Allowing blender to load backward compatible code , will allow everyone interested to start using blender as test bed for their code, and due to that, lot of code will be tested against blender , and why not , integrated into it. (which is exactly what I’m doing)

As much as I agree with this:

Typically automated (even semi-automated) updates only work with simple test cases and aren’t enough to load more complex projects.

what I’m after is the ability to load old python types in py3k (which is what past module allowes) so code can work in both, I’m not after any auto conversion of the code.

My specific case, is just an example of what is likely to happen in any vfx house in the next few years, and allowing blender to be part of this process, could bring blender closer to productions, and have productions able to better evaluate blender in their environments and tools, and why not, later on contributing back if they can use it for production, with improvements and bug fixes.

Hope it helps, if you have any other question, just let me know.
Cheers.
L.

17

OK, I see what you’re getting at, but this is speculation.

Unless you can show these tools are actually useful with a real-world non-trivial project, I remain skeptical about how useful they are in practice.

18

Hi @ideasman42

How allowing blender to use code which has been in production for years, and allow companies to better see the value of the software (as they might be able to start making it fit into their production pipelines and start using it along the others) can be considered speculation ?

I’m genuinely interested, though, on understanding if bringing back this module back could cause any harm to blender, and how.

Cheers.

L.

p.s
Being this a personal project, I’m happy to chat in PM if you like about other details and give you some of my background if you like.

19

The assumption you make is that these tools will work or give anything like a useful migration to Python3.

In practice I would guess this “half-works” but the time you spend messing with remaining broken things you could better spend on upgrading your project to Python 3 properly.

Further, why would you rely on this unofficially supported “past” module?
It had a single release 4 years ago.

Probably it’s already not well supported - so you only postpone the inevitable.

20

Guess, that’s the whole point, postpone the inevitable, but the time gained, is what is needed to have tools working in both contexts.

In practice I would guess this “half-works” but the time you spend messing with remaining broken things you could better spend on upgrading your project to Python 3 properly.

Tools has to work in both, or there’s no point now of doing a full port to python 3, as you are the only dcc application supporting it.

Let me try to ask something different then. how can I build blender with this module in so at least I can test ?
And please, can anyone let me know what problem or harm could cause to blender ?

Thanks.
L.

p.s
if past is not the answer , do you have any other module you can suggest, which can allow me to load python3 code in 2 and viceversa , I’m more than happy to change if there’s anything better !