Developer.blender.org; choice for GITEA. Reasons and timeline

Archive
21

Github can change it’s policy from one day to the other. And then you’re screwed and you need to change systems immediately. With the risk that all your old bugtracker and other history is lost because you don’t control the server it’s hosted on.

Gitea can stop it’s development. But because blender hosts the repository themselves they can take their time (like currently with phab) to look for something better while their own gitea instance just keeps on running.

And also it’s a matter of principle. Gitea is open source/libre. Github very much isn’t. Having a healthy open source ecosystem benefits all open source applications/communities. Having all needed infrastructure get monopolized by github (microsoft) is dangerous.

13 Likes
23

Two things I’d add:

21 Likes
24

Besides that, it should be known that if something closed source is being given away for free, you are likely the product , github seems to be no different in this department, their recent move to harvest OSS code for their paid copilot program is raising some eyebrows currently.

7 Likes
25

I don’t understand the strongly ideological arguments that the platform has to be open source for the sake of it. Being pragmatic is often just as important in my opinion. However, the points from @julianeisel (especially the second one) make a lot of sense to me.

Literally anyone can harvest open source code from GitHub to build something like Copilot. It is a platform where everyone has access to the public code and can use it. Microsoft could do that even if they didn’t own GitHub.
From my point of view, there is nothing wrong with that. The only issue is the license of the produced code which is an open legal question as far as I can see.

26

Its not just the license of the derived product, its if whether or not the code they included in their drag-net data-set allows for such derivatives.

If I understand GPL right then if Blender was hosted on Github and the source code was used for the Copilot project then Copilot would have to be open sourced/GPL’d too.

Correct me if Im wrong tho. I think the license landscape is fascinating! (really!)

27

@mods can you please split this off into a new topic on copilot? It’ll be nice to discuss it here on the platform and I’m pretty sure it’ll be brought up again somewhere else here later anyways.

if you believe that running Copilot is distributing (the individually licensed pieces of) software then yes.
Most licenses like MIT, BSD or GPL require some form of attribution and codex does not (and could never) give attribution.

I’m not convinced that Copilot is redistributing (copying) source code though.
You send copilot a prompt and it’ll respond with what it imagines to follow after the prompt. It’s autocompletion.
Copilots response will (ideally) not be copied from some code in its training dataset (overfitting), but instead it’ll be something new, generated from its ‘understanding’ of the issue.
That is, when Codex was trained, it learned to abstract code into concepts, that it can then apply to write code (in principle similar to a human!).
Therefore it’s impossible to know what exact pieces of code from the training data ‘inspired’ copilot to respond to your prompt in a certain way. Someone on Twitter described it cynically as ‘copyright laundering’ - idk about the copyright part, but it sure is a form of laundering.

For me personally this is the reason I’m fine with copilot. It learned to code - just like I did - and is therefore now able to solve problems. I didn’t violate any copyrights by reading blog posts, books, documentation or attending lectures; and in my opinion so didn’t copilot.
Now obviously the legal side of this could change - similar to how some countries have copyright exemptions to allow data mining, they could add rules to forbid machine learning. Or they could treat the outputs of an ML model as a derivative work of the training data. But they aren’t as of right now (and I don’t such a thing has been brought to courts yet?).

There’s another thing too: even if Copilot was copying verbatim and it was basically a code search engine: lots of countries have minimum thresholds under which it’s perfectly fine to just copy stuff without attribution. Basically, two lines of code from a project with 1000 lines are probably not ‘significant’. Even the help section on the GPL license home page says, that it’s not worth sticking a GPL header on a couple of lines.

And I think this is the crux: the spirit of the GPL was not to enforce copyright, but to undermine it. To proliferate free software that everyone can not only execute, but also read and modify. To prevent patent trolling, and to stop big companies from building monopolies that you cannot escape from.
Something like codex is something the GPL was never made to encourage or to deny, but let’s be real, the solution should not be to interpret copyright more strictly. Because the only ones profiting from that will not be programmers, but large IP holders such as Disney.

This is a good take by Felix Reda, a former member of the EU parliament for the pirates party:
https://felixreda.eu/2021/07/github-copilot-is-not-infringing-your-copyright/

1 Like
28

The Blender source code can be found on GitHub, as such it is highly likely that it was used to train Copilot. Copilot itself doesn’t contain Blender’s source code. The weights of its neural network were influenced by the code it “saw” during the training. That’s it.
So I pretty much agree with what @ogierm already replied. At the end, courts will have to decide regarding the legal situation and every country will have its own ruling and it is going to be a huge mess :slight_smile:

29

Thats is strong arguments around globall reach for Blender… maybe… exist developers around currently or IP around Phabricator?

1 Like
30

Anyhow, the arguments given for Gitea, especially the one Julian mentioned regarding access to countries under US sanctions are fair. There is definitely also value in supporting Gitea and improving libre alternatives to Github, especially since Gitea is something that can be run easily by anyone (contrary to Gitlab).

What doesn’t convince me is that Gitea has been selected, without apparently giving a thought to how contributors are supposed to work with it? Idk but stuff like “can we make a pr based workflow work, and do we want to do that”, would’ve been one of the first things on my mind when thinking about how the new platform should be used and what features it has to have.
In general it would’ve been nice to know what the hard and soft requirements were. Now we know that “Libre software that Blender will host” was one, but what were the others? And what were the reasons why Gitea specifically was chosen in the end?

My other concern is that there is apparently a lot on Arnds plate already, and now he’ll also have to run and manage the Gitea instance. If Blender will also have to develop custom extensions for Gitea, that will be a lot of time that could’ve been spent on building 3D software! (Or for devops: making the documentation & translations easier to edit, improving the test infra etc etc)

3 Likes
31

Hi, I want to share some clarification regarding the work process of this (and other) projects. To quote the Strategic Targets 2022 post:

The Blender Institute (BI) crew has two main tasks; the responsibility for maintaining the blender.org facilities and to initiate high priority strategic development projects – especially the ones that take long-term focus by experienced developers.

The migration from phabricator is a perfect combination of both, and as such falls within the responsibility of the Blender Institute. This includes the organization of the development process, communication strategy and decision-making.

The same goes for LTS, for the strategic targets for the BI crew, dev grants, workshops in Amsterdam, … If someone needs clarification on this let me know.

Now to some updates.

Arnd and I will have a meeting tomorrow (5/7/2022) where we go over some of the pipeline specific challenges for the new platform (e.g., how do we handle PR / diffs?). I will use both my experience as a developer involved actively at least in one module, and what I learned as a developer coordinator (specially in regard to how we tried to unify the pipeline in the different modules, with mixed results).

That should help to shine some line on how we will do things in gitea, the time-frame, the roll out of features. That should also help people to be able to give solution-centric feedback (given the gitea constrains, how else would you do it).

I will also check with him a good communication strategy. So we separate what we simply want to inform the community (e.g., a progress log of the work, the decisions that were made) versus when we need feedback (e.g., proposed workflows).

Have a fine day

8 Likes
32

Thanks for clarifying what the BI does, but no-one ever criticized the BI’s role here, it is as you point out literally what it is meant to do. I actually specifically called them out by name for being involved with a project like this.

The issue was the BI wasn’t mentioned at all in arnd’s opening post, he only explicitly mentioned “the studio” which I called “uncomfortable”

This later got clarified as “the HQ bubble calls everything the studio” to which my response is and can only be: Stop doing that!

If there’s anything that needs to be clarified it is for people in Amsterdam to call things by their proper name as to not muddy the waters between “the studio” and the non profit arms of blender as from the outside looking in, it’s “not a great look”

With that hopefully clarified, lets leave this ugly (yet important!) semantics mess behind and get moving forward on the actual project!

7 Likes
33

Gitea Diaries: Part 1

6 Likes
34

aaaah, derailed thread again… i thought this post/thread would have been about focussing on the devt blender is doing, not going back to the things where conclusion was made.

what decades ago, the discussion only happened in march this year.

hey @Arnd , please add the link to that old post ( Developer.blender.org - Call for comments and participation - #78 by Alan-Reviewer ) with a notice that discussion about alternates MUST go there…

it’s very distracting and tiresome to go over same things singing in favour of github/gitlab over and over

35

discussion about alternates MUST go there…
it’s very distracting and tiresome to go over same things singing in favour of github/gitlab over and over

Just to be clear:
There is no more discussion to be had about that.
The decision has already been made that we will go and try to do a full deployment and migration to Gitea.

If that fails, then we can start discussing alternatives again.

14 Likes
36

yeah, that’s what i thought and came here. but was disappointed to see the derailing here… anyways… 'ts okay.

37

Hi @Arnd,

I work on a Gitea based import/export package and closely follow the main branch of the Gitea codebase to stay up to date and ensure my contributions can be merged upstream.

Is there a place where I can follow the progress of the work done to modify the Gitea import/export code so that Blender can migrate from phabricator to Gitea?

The blog post published about two weeks ago reads:

And, quite significantly, we’re in the finishing stages of having a support agreement with the Gitea project to have them support us in our migration by funding work on missing features, code and bugfixes that will be available 100% to Gitea users under Gitea’s MIT license.

Which suggests discussions are ongoing but I’ve not read any specifics, either here or in the Gitea forums & chat rooms. Since Gitea is not an incorporated organization, I assume “the Gitea project” means these discussions happened with one or more of the three current “owners” (that’s the top of the foodchain in the Gitea community), in private.

While this is by no mean an obligation, it would be great to have clarity and transparency on this project. It would help me and others in the Gitea community anticipate and adapt depending on what is planned and discussed. This is the first time a support contract is being negotiated within the Gitea community and I’m very happy this is with Blender, a project dear to me :slight_smile:

The work done to transition from phabricator is likely to translate in a very large pull request that will introduce many significant changes in the migration code and quite possibly the database structure as well. Watching successive draft implementations over time would also give me an opportunity to contribute.

Thanks in advance for shading some light on this exciting project :sparkles:

10 Likes
38

Hey there @loic-dachary!

You’re right in wondering ‘where the code is’.
To get-up-to-speed asap I defaulted to using a self-hosted GITEA instance I had set up on a machine before, which isnt quite production-suitable; but was available and at least ‘good enough’ to start hosting some private repo material to share between the GITEA people and Blender.
This is not ‘how it should be’ ™ and my main focus right now is to set up a good place for this in Blender’s studio-infra to host this, instead.
Both the reason why ‘this happened in the first place’ and ‘why it hasnt been fixed yet’ has to do with a dutch hacker-conference taking away time. I just arrived back yesterday night. Hopefully this week I’ll be able to get it moved to a publically accessible place for people to have a look.

Right now it consists of:

  • A customized ‘Go/Goth’ oauth version with support for Blender-ID
  • A GITEA tree that incorporates the altered goth-module
  • A phabricator-migration-process repo that is used to track issues, share considerations and pull all the parts of issues together.

Wrt. to the talks with GITEA; we are indeed talking with the people who are behind the project as well as the efforts to add an organization behind it to support users/organizations who use Gitea as well as to function as a way to receive funds to support developers. Some of this happened on the gitea discord, some of it in mail, a good bit in video-chat (jitsi)

I believe that (sadly ?) we are not entirely the first organization that the Gitea project is supporting in this manner; but we are likely the first public one in that respect.

With regards to expected code-changes, we’re hoping to have the changes be mostly in the form of features that will indeed percolate into the main Gitea repo in the form of changes to migration code and/or (optional) functionality that can be used by others. We already identified some code which’ll likely be of very little re-use value, but will be freely shared/shown so that people can see what was involved in our conversion.

Main things that we have identified so far are:

  • Extension of migration-code to allow for higher fidelity data-imports
  • Custom Phabricator->‘migration-import-data’ tool to map blender-related tags/projects/etc to equivalents in gitea
  • Possible new logic around the git-server providing for different behaviours when using ‘fork’ (to save diskspace) (under investigation)
  • Code related to tags/labels/workboards (under investigation)

These are the ‘larger topics’ at the moment; next to questions of scaling, provisioning, etc… but the above ones are the most relevant wrt. to ‘expected code-additions’ from the top of my head.

Cheers for reaching out and hope to be able to provide more, soon.!

–
Arnd

8 Likes
39
  • hey, i was just thinking today, and this incurred to me that
  • the workflow on the “issue tracker” in the phabricator, IIRC, was:

r > t > d or in other words
report > task > differential

and the workflow in other “modern?” issue trackers is:

issue > merge request

(ofcourse, these both can be preceeded by discussions in other places)

that is:

  • there was an additional level of “report” before “task”
  • where “report” is the generally user filed so, verbose, overdetailed stuff, sometimes contains multiple issues too
  • and the “task” is isolated technical tangible aim filed by the team
  • that is, there’s an extra dimension/room for it, which helps a lott in issue management
  • and, since the modern issue trackers are just markdown comments and replies,
  • so, in those ones, this can effectively be accomplished by just having an additional clone of the same tracker for issues, and giving it a different name

and if the need of advanced features like tree-graph for tasks (like in phab) arises, that can be implemented later.

Questions:

  • thoughts? do the 2 trackers for same repo make sense? will they be useful?
  • Do gitea have something like that?
  • do blender want it?

Addendum:

Umh, this suggestion is on the underlying software side or “what we have” side of things, i have some further ideas on how these (combined with availability of subgroups) can be used for even more effective organization - but that will be on “how we use it” side of things. and that is out of scope currently.

40

@anonym Phabricator doesn’t have an additional level for reports. We have tasks of different types: bug report, known issue, to do, design. Triaging is already time consuming, having multiple tasks associated with one bug report does not seem worth it, there’s already the code review for more technical discussion.

1 Like
41

@Arnd Thanks for the update :sparkles: Has there been new updates? Maybe a repository where the work is making progress that I could look at? I’m eager to follow how this is going :slight_smile: