Developer.blender.org; choice for GITEA. Reasons and timeline

I thought the start of my post would have covered this, but i in no way shape of form am trying to imply arnd is doing a bad job, or isn’t super busy. What i’m saying is, calling the reasoning shared with the world on how we made this decision “light” would be on the generous side of things.

on “the studio” i can only say one thing, shape up, and step up your communication efforts, to call things by their proper name. It may make sense in your tiny HQ bubble, outside that bubble it looks bad to the point where it concerns me every time i hear the studio mentioned in a setting it does not belong.

I don’t think we should distract this thread much further and lets focus on phab’s successor instead

12 Likes

Do you know whether all existing tasks from Phab, including all comments, can be migrated to Gitea? Is it known whether changes to workflows for bug reporting, bug triaging, patch submission and code review are to be expected?

I imagine migrating URLs and Phabs automatic links to tasks could be a challenge.

2 Likes

Why work with gitea in case blender is opensource, and exist github?
I understand that FB not supporting more phabricator… and it little hard to manage around road map, etc. maybe… esspecialy if not php as FB have own version… maybe performance issue… but… why to not start support new version of Blender on github? Each new version can repport issue direct to github, then… burn old tasks on phabricator, close problem on some time and timeline… then new issues from old software can be moved/forward to github…

Effective June 1, 2021: Phabricator is no longer actively maintained.

Github is not an open source platform.

That’s a bit reductive. For me as someone who isn’t working for BF it straight up doesn’t make a difference if I’m interacting with Gitea hosted by Blender or Github hosted by Microsoft somewhere. Both are inaccessible to me.
It doesn’t even matter if the source code is accessible or not; I can’t do anything with it – it runs on someone elses machine after all.

Now the benefit of Github is that it’s pretty much ubiquitous. Everyone uses it, lots of tools have extra features to work better with the Github ecosystem and the backing is so stable, that it’s very unlikely that Github will shut down or cease development.
But the most important thing is that everyone can fork repos for free. This enables the pull request workflow. And all of Githubs other tools like issues, kanban boards, projects, CI/CD, the APIs are therefore useable and accessible to everyone.
I mean — there is a reason why Gitea is being developed on Github.

Moving to Gitea would still be an upgrade over Phabricator, as the UI is just more familiar and clearer (it’s modelled after Githubs after all), but if everyone without commit access would still have to upload patches, instead of just forking and then pushing commits to a branch, then the whole migration effort would mostly just be a sidegrade for me.

@ogierm @ManBlender Please do not derail this thread. This has been discussed previously plenty of times. Using an open source platform that the Blender Foundation has full control over is a hard requirement.

9 Likes

Sorry to have left replying to this thread until today. I’ve been putting 100% of my time into typing code for a (basic) migration-script for Phabricator-data into Gitea.
With the aim to have a full export of issues+comments on a public-facing gitea-test box available for people to look at at the start of next week.

Having said that; the discussion above focuses on a few things:

  • The confusion between BF, Studio, Institute, etc.
  • The lack of clarity/updates
  • The choice for Gitea over other options out there
  • How will we…/How will this…/What will happen with…

Lets first address the first issue
When I used the word ‘studio’, I meant it to refer to the people who walk around in the building, in Amsterdam. Mainly developers who are based in Amsterdam. Note that my original statement there was ‘…internal discussion and development…both online as well as at the blender studio.’
I am relatively new and perhaps have used the word ‘studio’ here too lightly, I’ll admit; but it was not intended to indicate ‘Blender Studio’ per se.

Secondly:
The lack of updates and public communication/updates is mainly due to the workload that’s on my shoulders. There’s Phabricator , there’s buildbot, there’s release-tests for each release we do, there’s studio (network) infra, there’s the datacenter infra and a number of other tasks to attend to.
The youtube video presented before that lists the deliberation process, the desired/expected timeline, the concept of making PoC’s, etc… has sadly not been attainable/fulfillable. There is just simply not enough time/people available to actually do that amount of work. It turned out to be even ambitious if it’d have been one person (me) that could dedicate 100% of his/her time to it (which even in the best of weeks has been a pipe-dream).
While this was going on, the views on what options we have available to us have shifted. Where Gitlab seemed enticing beforehand, multiple people have expressed their doubts about being on something that is ‘handed to us for free, but that can be taken back at any moment’. (Gitlab Ultimate tier)
The Gitlab Community-edition does not provide the features we’d need for even some of the basic things we’d like to be able to do; and patching against an open-source product that replicates functionality of their ‘enterprise offering’ would be a rather senseless activity indeed.
At the same time, more people have been taking a look at Gitea (and Pagure), and the general view (espcially those that decide upon my time) has been to put full effort into having a migration away from Phabricator done with a self-hosted Gitea implementation as the goal.
I personally concur that this is most likely to be the choice to be the most successful and future-proof; and having the decision to go for Gitea been made makes it possible to put efforts to an end-goal, taking concrete steps, making commitments, agreements, design-decisions, etc.

As for why Gitea ?
Some of it has already been discussed/touched upon in the section above. What rests here is to re-iterate what already has been clarified by Robert: the choice of a non open-source platform for hosting our code has been rejected, is not an option.
The Blender project is about more than just building the software to be OpenSource, but wants to have its own choices further the goals and ideals of OpenSource in general, too.
The choice of its ‘forge’ is part of that.

Then the last part:
How will certain features that we use NOW work with Gitea (or any non-phabricator workflow).
These questions are not answered yet. This is a discovery-process that’s already started , but no decisions have been made about these things yet. They depend on a lot of things, part of which is the software (Gitea), but the rest of it has much more to do with Phabricator, developer-time availability and hardware/storage-resources required for any particular solution.
Some of these things are:

  • Will we do a clone/pull-request/merge workflow ? Do we have a way to support that (disk-space wise)
  • What will we do with code-review messages in Phabricator that dont belong to a branch/tag or cloned-repo (right now)
  • How will we map the ‘loose collection of tags’ that Phabricator allows to something more strict like projects/labels/priorities ?
  • Phabricator does not use blender-id, but we’d really like that for the future. Can we import data into Gitea while preserving user-identity by finding out the correlations between Phab-users and users in Blender-id (is that even feasible ?)…
    etc. etc.

Since the announcement that the choice has been made to dedicate 100% of effort to Gitea, I’ve worked to have ‘something to show’ which’ll contain all bf-blender project issues and all comments in a ‘naive way’. Just to give some kind of an idea and for a place (that is not my development machine) for people to login (using blender-id) and interact with things (in a mockup environment).

Expect this from me at the beginning of next week.

Hope i’ve touched on the things that people brought up in the thread above as clearly as possible.

23 Likes

Thank you for the reply and all the effort that goes into this!

While I’m not in the picture of what has been discussed, planned and organized with the other developers at the HQ, I’m a bit surprised that you are currently the only person working on this and alongside your regular duties as well. Given how vitally important a functioning platform for the future development is, I hope that more developers / devops get involved in this project and that you are getting the support you need.

9 Likes

It was old discus, now situation is new after decade, coz technology was changed. More possibility.
What if gitea will be not suported more after time or other problems or costs that are important around “TOTEX”… care about servers, etc. ? It’s comparable with risk of Github open source policy as if open the free, etc.?

@silex phabricator was OS, and as on links… dead. It’s not always good argument that is “open source” or free.

Arguments new are that many addons for Blenders are on github. Also when bugs can be related can be moved to this repos, not like on gitea I think.

Just opinion and words to attention.

Github can change it’s policy from one day to the other. And then you’re screwed and you need to change systems immediately. With the risk that all your old bugtracker and other history is lost because you don’t control the server it’s hosted on.

Gitea can stop it’s development. But because blender hosts the repository themselves they can take their time (like currently with phab) to look for something better while their own gitea instance just keeps on running.

And also it’s a matter of principle. Gitea is open source/libre. Github very much isn’t. Having a healthy open source ecosystem benefits all open source applications/communities. Having all needed infrastructure get monopolized by github (microsoft) is dangerous.

13 Likes

Two things I’d add:

21 Likes

Besides that, it should be known that if something closed source is being given away for free, you are likely the product , github seems to be no different in this department, their recent move to harvest OSS code for their paid copilot program is raising some eyebrows currently.

7 Likes

I don’t understand the strongly ideological arguments that the platform has to be open source for the sake of it. Being pragmatic is often just as important in my opinion. However, the points from @julianeisel (especially the second one) make a lot of sense to me.

Literally anyone can harvest open source code from GitHub to build something like Copilot. It is a platform where everyone has access to the public code and can use it. Microsoft could do that even if they didn’t own GitHub.
From my point of view, there is nothing wrong with that. The only issue is the license of the produced code which is an open legal question as far as I can see.

Its not just the license of the derived product, its if whether or not the code they included in their drag-net data-set allows for such derivatives.

If I understand GPL right then if Blender was hosted on Github and the source code was used for the Copilot project then Copilot would have to be open sourced/GPL’d too.

Correct me if Im wrong tho. I think the license landscape is fascinating! (really!)

@mods can you please split this off into a new topic on copilot? It’ll be nice to discuss it here on the platform and I’m pretty sure it’ll be brought up again somewhere else here later anyways.

if you believe that running Copilot is distributing (the individually licensed pieces of) software then yes.
Most licenses like MIT, BSD or GPL require some form of attribution and codex does not (and could never) give attribution.

I’m not convinced that Copilot is redistributing (copying) source code though.
You send copilot a prompt and it’ll respond with what it imagines to follow after the prompt. It’s autocompletion.
Copilots response will (ideally) not be copied from some code in its training dataset (overfitting), but instead it’ll be something new, generated from its ‘understanding’ of the issue.
That is, when Codex was trained, it learned to abstract code into concepts, that it can then apply to write code (in principle similar to a human!).
Therefore it’s impossible to know what exact pieces of code from the training data ‘inspired’ copilot to respond to your prompt in a certain way. Someone on Twitter described it cynically as ‘copyright laundering’ - idk about the copyright part, but it sure is a form of laundering.

For me personally this is the reason I’m fine with copilot. It learned to code - just like I did - and is therefore now able to solve problems. I didn’t violate any copyrights by reading blog posts, books, documentation or attending lectures; and in my opinion so didn’t copilot.
Now obviously the legal side of this could change - similar to how some countries have copyright exemptions to allow data mining, they could add rules to forbid machine learning. Or they could treat the outputs of an ML model as a derivative work of the training data. But they aren’t as of right now (and I don’t such a thing has been brought to courts yet?).

There’s another thing too: even if Copilot was copying verbatim and it was basically a code search engine: lots of countries have minimum thresholds under which it’s perfectly fine to just copy stuff without attribution. Basically, two lines of code from a project with 1000 lines are probably not ‘significant’. Even the help section on the GPL license home page says, that it’s not worth sticking a GPL header on a couple of lines.

And I think this is the crux: the spirit of the GPL was not to enforce copyright, but to undermine it. To proliferate free software that everyone can not only execute, but also read and modify. To prevent patent trolling, and to stop big companies from building monopolies that you cannot escape from.
Something like codex is something the GPL was never made to encourage or to deny, but let’s be real, the solution should not be to interpret copyright more strictly. Because the only ones profiting from that will not be programmers, but large IP holders such as Disney.

This is a good take by Felix Reda, a former member of the EU parliament for the pirates party:
https://felixreda.eu/2021/07/github-copilot-is-not-infringing-your-copyright/

1 Like

The Blender source code can be found on GitHub, as such it is highly likely that it was used to train Copilot. Copilot itself doesn’t contain Blender’s source code. The weights of its neural network were influenced by the code it “saw” during the training. That’s it.
So I pretty much agree with what @ogierm already replied. At the end, courts will have to decide regarding the legal situation and every country will have its own ruling and it is going to be a huge mess :slight_smile:

Thats is strong arguments around globall reach for Blender… maybe… exist developers around currently or IP around Phabricator?

1 Like

Anyhow, the arguments given for Gitea, especially the one Julian mentioned regarding access to countries under US sanctions are fair. There is definitely also value in supporting Gitea and improving libre alternatives to Github, especially since Gitea is something that can be run easily by anyone (contrary to Gitlab).

What doesn’t convince me is that Gitea has been selected, without apparently giving a thought to how contributors are supposed to work with it? Idk but stuff like “can we make a pr based workflow work, and do we want to do that”, would’ve been one of the first things on my mind when thinking about how the new platform should be used and what features it has to have.
In general it would’ve been nice to know what the hard and soft requirements were. Now we know that “Libre software that Blender will host” was one, but what were the others? And what were the reasons why Gitea specifically was chosen in the end?

My other concern is that there is apparently a lot on Arnds plate already, and now he’ll also have to run and manage the Gitea instance. If Blender will also have to develop custom extensions for Gitea, that will be a lot of time that could’ve been spent on building 3D software! (Or for devops: making the documentation & translations easier to edit, improving the test infra etc etc)

3 Likes

Hi, I want to share some clarification regarding the work process of this (and other) projects. To quote the Strategic Targets 2022 post:

The Blender Institute (BI) crew has two main tasks; the responsibility for maintaining the blender.org facilities and to initiate high priority strategic development projects – especially the ones that take long-term focus by experienced developers.

The migration from phabricator is a perfect combination of both, and as such falls within the responsibility of the Blender Institute. This includes the organization of the development process, communication strategy and decision-making.

The same goes for LTS, for the strategic targets for the BI crew, dev grants, workshops in Amsterdam, … If someone needs clarification on this let me know.


Now to some updates.

Arnd and I will have a meeting tomorrow (5/7/2022) where we go over some of the pipeline specific challenges for the new platform (e.g., how do we handle PR / diffs?). I will use both my experience as a developer involved actively at least in one module, and what I learned as a developer coordinator (specially in regard to how we tried to unify the pipeline in the different modules, with mixed results).

That should help to shine some line on how we will do things in gitea, the time-frame, the roll out of features. That should also help people to be able to give solution-centric feedback (given the gitea constrains, how else would you do it).

I will also check with him a good communication strategy. So we separate what we simply want to inform the community (e.g., a progress log of the work, the decisions that were made) versus when we need feedback (e.g., proposed workflows).

Have a fine day

8 Likes

Thanks for clarifying what the BI does, but no-one ever criticized the BI’s role here, it is as you point out literally what it is meant to do. I actually specifically called them out by name for being involved with a project like this.

The issue was the BI wasn’t mentioned at all in arnd’s opening post, he only explicitly mentioned “the studio” which I called “uncomfortable”

This later got clarified as “the HQ bubble calls everything the studio” to which my response is and can only be: Stop doing that!

If there’s anything that needs to be clarified it is for people in Amsterdam to call things by their proper name as to not muddy the waters between “the studio” and the non profit arms of blender as from the outside looking in, it’s “not a great look”

With that hopefully clarified, lets leave this ugly (yet important!) semantics mess behind and get moving forward on the actual project!

7 Likes