Developer.blender.org; choice for GITEA. Reasons and timeline

Archive
1

First off, apologies for the low rate of public updates on this topic.

This did not reflect the amount of internal discussion and development that’s gone into this subject both online as well as at the blender studio.

The outcome of the process so far is that we’ve decided is that Gitea is our platform-of-choice for moving to, away from Phabricator.

There are a fair number of reasons for this and a lot of them have been voiced in the previous thread here on devtalk, but the general consensus has also become clear in talks with people we’ve spoken to as well as what are desired long-term goals and achievements the blender-foundation wants to work towards.

In a lot of ways, having a fully open-source platform available to us is something that is seen as both important as well as beneficial to Blender on the shorter and longer term. And, also importantly, not just for us.

In the process of exploring options, the first steps have already been taken towards getting a Gitea setup up and running and contain meaningful data. More will be coming in the following weeks and I hope to be able to show something by the middle of next week to the wider community.

We’ve reached out to the Gitea developers regarding our wish to move towards their software and are looking forward to explore any possibilities that’d provide mutual benefit in the form of strengthening the goals of our communities; building the best open source solution out there for our users.

As for Blender, the ambition is to have a working, usable, implemented development-forge running in the course of October. This will not be easy but seems possible with what we’ve discovered through the work already performed.

As we work towards that goal, we want to also start producing more regular updates on where we stand and what our developer-community can expect. When are there opportunities to help test things, how to provide feedback (and about what), when will the exact switchover be and how will that happen, etc.

Expect more about this , soon, as this gets dialed in. Devtalk will be part of it, for sure.

–
Arnd

30 Likes
2

I cannot imagine the complexity of migrating such a project, I imagine along with commit history etc. Back in the day I would have gone “cool for Blender”, today not only that but it’s probably a huge deal for Gitea too, haha

Anyway cool

4 Likes
3

Great news! So is the plan to use buildbot-gitea for existing ci/cd integration?

1 Like
4

That’s actually quite a good question. That hasnt been looked into yet, though the existence of that extension is known to us.
It seems like an obvious thing to start providing ways unlocking build-related (CI/CD) data from inside of our development-forge in an easy way.
In short: it’s very likely that we’ll end up enabling that. It’s just a question of ‘when’, as it also involves work on the buildbot side of things.
As for a reason why it wouldnt be a plug-n-play kind of thing for us is that we do not currently have a very good way to organize logins/authorization to our buildbot setup other than static passwords. Given that we fully intend to have Blender-ID involved in access to Gitea (have that working in a proof of concept already), it’ll involve having a look at our Buildbot setup and see what it’d take to get Blender-ID involved there too.

It’s obvious that it’ll be possible and very likely is going to be a good idea, but it may well not be a thing we’ll decide to tackle after we’ve ticked off more important tasks from our list.

Welcome to the discussion btw! Again, good question

3 Likes
5

It’s my please, and thanks for the warm welcome.
I see how the identity management can be a problem here but I guess in any case ci/cd subsystem will require to work with blenderId from a shared idP.
So buildbot-gitea seem to be a good starting point and might let this transition to be smoother.

6

What will be the workflow for contributors without commit rights going forward?

7

This is something that we’ll be looking at in the near future and thus has not been hammered out yet.

There is of course the usual ‘clone->pullrequest->merge’ workflow possible.
There seems to also, however, be support for patch-files directly, like in the current workflow used in phabricator.

We’ll likely have to leverage support for ‘patchfiles’ in Gitea in able to migrate code and patch history into Gitea in some meaningful way.

In short: undetermined, but will be one of the first things to hammer out in the process of defining what kind of customization-work will be required.

1 Like
8

Would pull requests require that the fork is hosted on Blenders Gitea instance (or one that federates with it), or could the fork be anywhere (ie GitHub)?

9

We’d have to check if the latter is even possible, and that still depends on us adopting that workflow in the first place.
A possible main issue there is that a clone of a repo leads to quite some data on disk, creating risks regarding system-requirements to support all that.

Pull-requests between forges might not be practical or even desirable; but again, this has not been decided/determined yet either.

10

I’ll be upfront, this post may seem harsh (and at some points it is) however, please don’t confuse having a critical look as something and expecting some accountability with disagreement, I’m happy to hear we have picked an OSS direction to work towards even though the gitlab offering was very seductive and generous.

It’s always good to have a large pool of opinions, what has been done with them? What is the set of core requirements for a new platform that was extracted from these discussions?

The blender studio is a commercial independent [1] entity that makes movies afaik, why are they intimately involved with development and development decisions governing the blender opensource project? The foundation [2] ? for sure! the institute [3] sounds good! the studio? Why? don’t get me wrong more opinions are always better, but it is curious for me at least to see these things happening at the studio rather than the foundation, or institute (which were not mentioned at all in your post) it implies an uncomfortable influence of what ought to be an independent entity. I’ll be upfront, this has me concerned as this is not the first time i see the waters between the blender OSS project and the blender studio muddier than i’d like them to be.

we chose gitea for “reasons” and “desires” and “goals” you discovered during “talks”… glad we cleared that up! can you elaborate how gitea is the right choice for us and how other platforms that were being considered failed to meet our bar in the…[check notes]…“desires and goals”… department?

I have to admit, i like the direction this is going, however this update is much less transparent as I’d like it to be and frankly a little skimpy on details.

[1] Blender Studio — blender.org
[2] Blender Foundation — blender.org
[3] Blender Institute — blender.org

14 Likes
11

“Blender Studio” or “the studio” has become an informal pseudonym for “the office” in Amsterdam. So here it refers to the place (that houses these multiple organizations), not the company. It’s a bit confusing, and we haven’t come up with a name yet that isn’t awkward in some contexts.
Point is, there have been countless of discussions in the office, also with remote developers, Gitea developers, Gitlab, etc.

The Blender Studio is independent in terms of finances, i.e. doesn’t rely on big investors like a lot of the industry, and doesn’t have clients telling them what to do. They are definitely not independent of Blender and its development. Maybe the wording could be a bit more clear there.

Indeed the work going on isn’t reflected well enough online, I agree there should be more public insight. But I really have to defend Arnd here. He has a bunch of responsibilities in the infrastructure department. There is so much unexpected crap happening all the time that Arnd has to deal with (e.g. the buildbot and build signing are really good at breaking), it’s impressive that he even manages to spend time on the Phabricator migration. And he’s doing this very thoroughly and efficiently. That guy just started this job a few months ago!

There have been two presentations about the migration in the office that were recorded. Arnd also wanted to do further ones, I think every two weeks? Maybe the recordings can all be published. The current ones just contain the URL of a testing setup that shouldn’t be hammered by public traffic yet, we would have to address that first.

13 Likes
12

I thought the start of my post would have covered this, but i in no way shape of form am trying to imply arnd is doing a bad job, or isn’t super busy. What i’m saying is, calling the reasoning shared with the world on how we made this decision “light” would be on the generous side of things.

on “the studio” i can only say one thing, shape up, and step up your communication efforts, to call things by their proper name. It may make sense in your tiny HQ bubble, outside that bubble it looks bad to the point where it concerns me every time i hear the studio mentioned in a setting it does not belong.

I don’t think we should distract this thread much further and lets focus on phab’s successor instead

12 Likes
13

Do you know whether all existing tasks from Phab, including all comments, can be migrated to Gitea? Is it known whether changes to workflows for bug reporting, bug triaging, patch submission and code review are to be expected?

I imagine migrating URLs and Phabs automatic links to tasks could be a challenge.

1 Like
14

Why work with gitea in case blender is opensource, and exist github?
I understand that FB not supporting more phabricator… and it little hard to manage around road map, etc. maybe… esspecialy if not php as FB have own version… maybe performance issue… but… why to not start support new version of Blender on github? Each new version can repport issue direct to github, then… burn old tasks on phabricator, close problem on some time and timeline… then new issues from old software can be moved/forward to github…

Effective June 1, 2021: Phabricator is no longer actively maintained.

15

Github is not an open source platform.

16

That’s a bit reductive. For me as someone who isn’t working for BF it straight up doesn’t make a difference if I’m interacting with Gitea hosted by Blender or Github hosted by Microsoft somewhere. Both are inaccessible to me.
It doesn’t even matter if the source code is accessible or not; I can’t do anything with it – it runs on someone elses machine after all.

Now the benefit of Github is that it’s pretty much ubiquitous. Everyone uses it, lots of tools have extra features to work better with the Github ecosystem and the backing is so stable, that it’s very unlikely that Github will shut down or cease development.
But the most important thing is that everyone can fork repos for free. This enables the pull request workflow. And all of Githubs other tools like issues, kanban boards, projects, CI/CD, the APIs are therefore useable and accessible to everyone.
I mean — there is a reason why Gitea is being developed on Github.

Moving to Gitea would still be an upgrade over Phabricator, as the UI is just more familiar and clearer (it’s modelled after Githubs after all), but if everyone without commit access would still have to upload patches, instead of just forking and then pushing commits to a branch, then the whole migration effort would mostly just be a sidegrade for me.

17

@ogierm @ManBlender Please do not derail this thread. This has been discussed previously plenty of times. Using an open source platform that the Blender Foundation has full control over is a hard requirement.

9 Likes
18

Sorry to have left replying to this thread until today. I’ve been putting 100% of my time into typing code for a (basic) migration-script for Phabricator-data into Gitea.
With the aim to have a full export of issues+comments on a public-facing gitea-test box available for people to look at at the start of next week.

Having said that; the discussion above focuses on a few things:

  • The confusion between BF, Studio, Institute, etc.
  • The lack of clarity/updates
  • The choice for Gitea over other options out there
  • How will we…/How will this…/What will happen with…

Lets first address the first issue
When I used the word ‘studio’, I meant it to refer to the people who walk around in the building, in Amsterdam. Mainly developers who are based in Amsterdam. Note that my original statement there was ‘…internal discussion and development…both online as well as at the blender studio.’
I am relatively new and perhaps have used the word ‘studio’ here too lightly, I’ll admit; but it was not intended to indicate ‘Blender Studio’ per se.

Secondly:
The lack of updates and public communication/updates is mainly due to the workload that’s on my shoulders. There’s Phabricator , there’s buildbot, there’s release-tests for each release we do, there’s studio (network) infra, there’s the datacenter infra and a number of other tasks to attend to.
The youtube video presented before that lists the deliberation process, the desired/expected timeline, the concept of making PoC’s, etc… has sadly not been attainable/fulfillable. There is just simply not enough time/people available to actually do that amount of work. It turned out to be even ambitious if it’d have been one person (me) that could dedicate 100% of his/her time to it (which even in the best of weeks has been a pipe-dream).
While this was going on, the views on what options we have available to us have shifted. Where Gitlab seemed enticing beforehand, multiple people have expressed their doubts about being on something that is ‘handed to us for free, but that can be taken back at any moment’. (Gitlab Ultimate tier)
The Gitlab Community-edition does not provide the features we’d need for even some of the basic things we’d like to be able to do; and patching against an open-source product that replicates functionality of their ‘enterprise offering’ would be a rather senseless activity indeed.
At the same time, more people have been taking a look at Gitea (and Pagure), and the general view (espcially those that decide upon my time) has been to put full effort into having a migration away from Phabricator done with a self-hosted Gitea implementation as the goal.
I personally concur that this is most likely to be the choice to be the most successful and future-proof; and having the decision to go for Gitea been made makes it possible to put efforts to an end-goal, taking concrete steps, making commitments, agreements, design-decisions, etc.

As for why Gitea ?
Some of it has already been discussed/touched upon in the section above. What rests here is to re-iterate what already has been clarified by Robert: the choice of a non open-source platform for hosting our code has been rejected, is not an option.
The Blender project is about more than just building the software to be OpenSource, but wants to have its own choices further the goals and ideals of OpenSource in general, too.
The choice of its ‘forge’ is part of that.

Then the last part:
How will certain features that we use NOW work with Gitea (or any non-phabricator workflow).
These questions are not answered yet. This is a discovery-process that’s already started , but no decisions have been made about these things yet. They depend on a lot of things, part of which is the software (Gitea), but the rest of it has much more to do with Phabricator, developer-time availability and hardware/storage-resources required for any particular solution.
Some of these things are:

  • Will we do a clone/pull-request/merge workflow ? Do we have a way to support that (disk-space wise)
  • What will we do with code-review messages in Phabricator that dont belong to a branch/tag or cloned-repo (right now)
  • How will we map the ‘loose collection of tags’ that Phabricator allows to something more strict like projects/labels/priorities ?
  • Phabricator does not use blender-id, but we’d really like that for the future. Can we import data into Gitea while preserving user-identity by finding out the correlations between Phab-users and users in Blender-id (is that even feasible ?)…
    etc. etc.

Since the announcement that the choice has been made to dedicate 100% of effort to Gitea, I’ve worked to have ‘something to show’ which’ll contain all bf-blender project issues and all comments in a ‘naive way’. Just to give some kind of an idea and for a place (that is not my development machine) for people to login (using blender-id) and interact with things (in a mockup environment).

Expect this from me at the beginning of next week.

Hope i’ve touched on the things that people brought up in the thread above as clearly as possible.

23 Likes
19

Thank you for the reply and all the effort that goes into this!

While I’m not in the picture of what has been discussed, planned and organized with the other developers at the HQ, I’m a bit surprised that you are currently the only person working on this and alongside your regular duties as well. Given how vitally important a functioning platform for the future development is, I hope that more developers / devops get involved in this project and that you are getting the support you need.

9 Likes
20

It was old discus, now situation is new after decade, coz technology was changed. More possibility.
What if gitea will be not suported more after time or other problems or costs that are important around “TOTEX”… care about servers, etc. ? It’s comparable with risk of Github open source policy as if open the free, etc.?

@silex phabricator was OS, and as on links… dead. It’s not always good argument that is “open source” or free.

Arguments new are that many addons for Blenders are on github. Also when bugs can be related can be moved to this repos, not like on gitea I think.

Just opinion and words to attention.