I’ve been managing a project for a new developer documentation platform, that seeks to optimize the documentation workflow for developers. The bigger picture is an effort towards a new culture for technical documentation.
Developer documentation will then live on developer.blender.org/docs.
This platform would replace the Wiki, which is a rather significant change. An archived version of the Wiki will likely be available still.
The transition may happen as early as next week (Dec 18-22)!
This is because of developer availability. If not, we don’t want to drag out the transition for long, so it may happen in the first weeks of January instead.
Where do personal pages & weekly reports go?
Developers writing weekly reports can create a repository on their projects.blender.org account. There you can either use the builtin repository Wiki (uses Markdown), or just create Markdown files inside the repository that you edit online (or offline even).
I’ve done the latter for a while now: JulianEisel/Weekly-Reports (see output). I personally found editing that a bit nicer over the builtin Wiki, but either way should be fine. Easy to change back and forth.
Where do the release notes go?
The complete release notes will be moved to the new platform. It’s unclear still if we will port old release notes, depends on how tangible this is.
The Wiki would be made read-only and hopefully moved to archive.blender.org (with redirects). So old release notes could stay there without issues. Clarified in the post, thanks
Ah, okay, guess it’s personal preference then, but I’d really hope we don’t end up with yet another half-ported archived wiki. I think the goal for this project should be to completely replace the wiki, or not do anything.
I think it’s fine to have old documentation archived somehow. Looking forward, Git should have that covered.
All current documentation should be on the new platform indeed, including long-standing core designs. We can also do a bit of spring cleanup while we’re doing the transition. Generally I’d say if we have to refer to archived docs, that’s a sign for badly maintained documentation, which the new platform tries to address.
May I suggest to use the Gitea convention and have a .profile repository? Gitea uses the README.md inside of that for the front page, people can put infos, links etc into it. See my profile as an example. This repo can then also be used for reports, again check my profile as an example.
Weekly reports are kinda scattered atm, and I would really love if reports are in one place in the future, making it easier to find them.
Ultimately, you only have to edit your weekly notes in your repo then. I will take care of linking to them from the devtalk weekly notes!
Any objections or concerns to that? The template can be modified of course if people prefer for example a descending chronological order or like to remove / modify the dates mentioned there. The only bit which cannot be changed is the url structure YEAR.md#calendar-week-X.
I don’t think the title and date format should be customized by people at all. I don’t think it’s a good thing that it looks different on every user page now, or that the order is randomly flipped.
One thing I’d like some feedback on is what people think about moving the docs into the main Blender repository. The advantage would be that pull requests can include release notes and design docs, and we can reasonably require all developers and contributors to do this.
If the repository is a (quasi-)submodule of the Blender repository, it’s not nearly as convenient for contributors without commit access to have to create multiple pull requests, and not as simple to review and merge.
I think this could be very helpful. Writing the release notes forces you to think about how the feature will be presented to users, and how easy to understand it really is.
But there are challenges.
Image and video file sizes. The plan already for the developer docs is to host to the images and videos outside of the developer docs repo, on an upcoming drive.blender.org where anyone with a Blender ID would be able to upload files for this purpose. If this works out, we can have only markdown files which are quite small.
Noisy commit messages. Imagine the number of commits doubles because of this, would it get too annoying for reading through git logs or bisecting? Would it stop people from making minor spelling or grammar fixes? I think I would be ok with it, but not sure how other people feel.
Unnecessary buildbot builds. We’d need to setup a filter on the buildbot to trigger Blender and developer docs build depending on the folder the changes were made it. This is possible in buildbot and common in CI systems in general.
Add-ons and other contributor permissions. If we have a separate repository, we can easily give out commit access to add-on developers, artists and any other contributor. If it’s in the Blender repository, they would have to go through pull requests. For add-ons we could potentially put the docs inside the add-ons repo, though that only solves part of the problem.
Release branches. The docs would always be built off main, but for release notes we’d probably want to put them in the release branch and then merge them in. This can lead to merge conflicts when someone edits the release notes in main. So there’s some extra overhead to this.
We can still do this after the initial migration to markdown. And probably should if we decide to do it, to avoid delaying the migration.
Another thing that would be helpful to get feedback on is the current navigation structure and content. All the pages that we didn’t intentionally leave out are there now. See:
Are there any pages that were left out that are considered important to keep? Looking for “trash” in remap.txt shows what is being removed. Note some of the content got merged into other pages.
In the features section, I grouped together some areas in the navigation structure to keep the number of sections from growing too long. Is this a reasonable structure, should we expand it out more or organize things differently?
Some features subsections have a “Module” and/or “Proposals” folder for things that are not design and implementation docs. I don’t think it’s an ideal fit, but we have to put them somewhere. Is it best to keep everything related to certain functionality together like this? Should “Module” stuff become its own section entirely, perhaps also moving the wiki pages from projects.blender.org? Should “Proposals” be an entirely separate section from “Features”, or a separate top level section under “Features”?
For the handbook section. Is it easy to find things? Is 13 subsections too much or is there a good way to merge some things?
Where to put summer of code? It’s under its own Contributor Programs at the top level, but it’s unclear it should be so prominent compared to other things. On the other hand it’s not obviously “developer handbook” or “features” either, as it has both instructions as well as information about projects.
I have nearly always had bad experiences with projects that try to bundle everything relevant to it into a big “mono repo”. (You’ve already listed quite a few of the downsides of this already)
I don’t know if having it as part of the main repo would help too much either with the problems you we are trying to solve either.
For example:
Not every change that is committed or reviewed via PRs is required to have release notes or design docs. So we still need to on a case per case basis decide if this is needed. So this won’t really automatically nag or remind people that they need to do any of those things.
I think this will make PRs and commits a lot more noisy. Not just the amount but the content/discussions in the PRs as well.
Lets say that i have made a big contribution that has 5+ doc pages that contains images and videos as well. In the PR then the conversation will be a mishmash of feedback about the code and the docs at the same time. I think this will take away from one of the most important bits of code reviews: Is the actual code good?
In the same vein as the point above: If we have a big PR with multiple reviewers assigned (4-5). How will they be able to clearly signal that they accept the code but not the docs and vise versa? Documenting and writing code are two different skill sets as well, so I have many times in the past seen this be handled by different people and at different times. (IE the code is merged first and then the developer works with one or multiple other people to write documentation that is good and easy to understand)
I think having separate repos will make it clear what the focus is of each repo is. In the doc repo it is to provide good documentation. In the Blender repo is to provide good code that solves issues.
This will make it so that reviewers don’t have to context switch between code review and doc review in the middle of the review process.
Sure, it might make it easier for people to be able to bundle everything into a single big merge. But I don’t think it is a good idea to have these “mega merges” as the docs and the code are to separate things that can function without each other.
It’s not about automating it, but having it part of the guidelines for writing PRs.
Right now the issue is not that I have to nag contributors about doing this, but that it’s not really practical to ask it all. The amount of overhead is big enough that it’s faster to just write them myself, and that would continue to be the case with a separate repo.
If the PR comes with 5 pages of docs, then the amount of code in the PR is already too big to review. The amount of docs is going to be proportional to the amount of code, so I don’t think it would end up being excessive compared to the rest.
And if it needs a lot of docs, that information is useful for reviewers too.
I don’t think this is really different than what we already do, we often approve some subset of the PR and write a comment about that. In practice it’s rare to have more than 2 reviewers.
We already have the requirement that release notes must be updated immediately along with the code. This can be minimal and refined later, but something has to be there.
Eh, I dont really agree here.
PR reviews can drag on for quite a while already. For example if these two had docs included into them as well, I think that the review would quickly lose focus and there would be two discussions going on in parallel.
I think the amount of interactions (comments, feedback) will probably be proportional if not greater than for the code for docs. So this means that something like the two PRs above would quickly balloon into something that would be in my opinion be very tedious to work with.
Right, but my point is that it will be even messier than it already is as instead of only having to keep track of what code parts are approved they also have to keep track of what is approved of the docs, in the same PR.
But if it doesn’t have to be there immediately, then having separate repos is not a big issue, no?
Both of these need like 2 or 3 sentences in the release notes, mostly copied from the PR description. I don’t think that would have been added much overhead.
It means that for me as a reviewer or module owner I have to either write the release notes myself or keep track of what is missing in the release notes and bug people when they inevitably forget.
Applying correct formatting, running tests, adding tests, writing release notes and making demo images, checking performance impact, getting user feedback, write user manual docs, … it all adds up and can make it so that my time investment as a reviewer is as much as doing the implementation. This doesn’t scale well, and we’re actively trying to improve that on various fronts. For me release notes is one part of that.
I think it would be nice to have the documentation in the same PR as the code it goes with. Much less chance of things going out of sync and also much easier as a contributor to add docs. I now often add the docs diff as a comment on the PR anyway so I don’t think it would matter much for the PR discussion.
Large files should be sorted out though or the repo will balloon way too quickly.
On the other hand… the same problems for docs are also present for tests. And adding those into the main repo seems a bad idea to me (because of all the images). And since we probably can’t really solve it for tests it might be better to just make the instructions for contributors for how to add docs and tests more clear?
Obviously docs are good, but keeping the actual process as simple as possible is still the most important thing. Asking someone to open 4 pull requests (code, release notes, tests, manual) for a single feature is just not realistic to me, no matter how well it is documented.
Thanks Brecht