Changes to the Blender.Git submodule configuration

Announcements
1

Just a quick announcement that we are changing the submodule configuration of the main blender.git repository. This will be done for both blender-v3.5-release and main branches.

The very terse version is that you’ll need to run make update and it will automatically take care of converting the submodules to the new structure.

The fuller version can be read in the design proposal which includes motivational part and the actual design of the changed status. The important parts:

  • release/scripts are moved to the scripts
  • addons and addons_contrib are simple Git checkouts, which are ignored as per .gitignore
  • source/tools are moved to the tools and are part of the main repository. The repository used for submodule will be archived.
  • release/datafiles/locale is moved to locale and is part of the main repository. The repository used for submodule will be archived as well.

Another aspect of the change is that the make update will support Github style of remote organization (origin remote pointing to thy fork, upstream remote pointing to the upstream blender/blender.git).

EDIT 1: Just to clarify: the addons and addons_contrib are still separate from blender.git repositories (blender-addons.git and blender-addons-contrib.git). They are just checked out using slightly different git commands (regular git clone instead of git submodule).

EDIT 2: The new version of the make update is needed to bring the repository to the proper state. So, git pull (or other means of updating your working branch) is needed to grab the new version of the make update script.

4 Likes
2

I keep getting these errors when trying to compile the latest Blender:

[1384/1703] Generating cs.mo
FAILED: source/creator/cs.mo C:/blender-git/build_windows_x64_vc17_Debug/source/creator/cs.mo
cmd.exe /C "cd /D C:\blender-git\build_windows_x64_vc17_Debug\source\creator && C:\Users\andre\scoop\apps\cmake\3.25.1\bin\cmake.exe -E make_directory C:/blender-git/build_windows_x64_vc17_Debug/source/creator && C:\blender-git\build_windows_x64_vc17_Debug\bin\msgfmt.exe C:/blender-git/blender/locale/po/cs.po C:/blender-git/build_windows_x64_vc17_Debug/source/creator/cs.mo"
[1386/1703] Generating ca.mo
FAILED: source/creator/ca.mo C:/blender-git/build_windows_x64_vc17_Debug/source/creator/ca.mo
cmd.exe /C "cd /D C:\blender-git\build_windows_x64_vc17_Debug\source\creator && C:\Users\andre\scoop\apps\cmake\3.25.1\bin\cmake.exe -E make_directory C:/blender-git/build_windows_x64_vc17_Debug/source/creator && C:\blender-git\build_windows_x64_vc17_Debug\bin\msgfmt.exe C:/blender-git/blender/locale/po/ca.po C:/blender-git/build_windows_x64_vc17_Debug/source/creator/ca.mo"
[1387/1703] Generating ar.mo
FAILED: source/creator/ar.mo C:/blender-git/build_windows_x64_vc17_Debug/source/creator/ar.mo
cmd.exe /C "cd /D C:\blender-git\build_windows_x64_vc17_Debug\source\creator && C:\Users\andre\scoop\apps\cmake\3.25.1\bin\cmake.exe -E make_directory C:/blender-git/build_windows_x64_vc17_Debug/source/creator && C:\blender-git\build_windows_x64_vc17_Debug\bin\msgfmt.exe C:/blender-git/blender/locale/po/ar.po C:/blender-git/build_windows_x64_vc17_Debug/source/creator/ar.mo"
[1388/1703] Generating de.mo
FAILED: source/creator/de.mo C:/blender-git/build_windows_x64_vc17_Debug/source/creator/de.mo
cmd.exe /C "cd /D C:\blender-git\build_windows_x64_vc17_Debug\source\creator && C:\Users\andre\scoop\apps\cmake\3.25.1\bin\cmake.exe -E make_directory C:/blender-git/build_windows_x64_vc17_Debug/source/creator && C:\blender-git\build_windows_x64_vc17_Debug\bin\msgfmt.exe C:/blender-git/blender/locale/po/de.po C:/blender-git/build_windows_x64_vc17_Debug/source/creator/de.mo"

I ran make update a few times to make sure it’s the latest version. Already tried deleting the build folder and compiling from scratch. Seems to be related to this change. Do you know what could be causing the errors?

3

Not really sure what is going on. Don’t have Windows handy to check, can only rely on the buildbot which successfully builds both 3.5 and the latest main.

Maybe start with checking if there are errors reported by the make update and that the locale folder exists in your checkout.

4

Also had troubles with locale module, but after switching to release build all went well.

Will troubleshoot tomorrow, I thought this was just me, because today I had experienced nothing but various random issues :confused:

1 Like
5

No errors from make update and the locale folder does exist in the root directory.

6

@pembem22 did you get a popup about tbb_debug.dll not found during this by any chance?

7

Don’t remember seeing that one.

8

Don’t see why this commit triggered the behavior, but it seems like it’s unrelated wrongness that causes it

msgfmt has a TBB dependency though bf_blenlib. In the release config the MSVC linker is smart enough to figure out it is not used, and discards the dependency hence you can run msgfmt without needing tbb.dll available, debug however leaves the dependency intact and requires some runtime dll’s.

Now if you have an existing build folder, tbb_debug.dll will already be sitting next to msgfmt.exe and all will be well, a clean folder however the runtime dll’s only get copied during the install phase, long long after msgfmt is supposed to run. causing msgfmt to error out since its runtime requirements are not met.

I have a fix, will try to get that in later today.

2 Likes
9

You are right, thanks! I manually copied tbb_debug.dll to the same folder as msgfmt.exe and the build finished without any errors.

10

After pulling down the new organization with make update, is it ok to switch to the upstream remote name convention then? Just want to double-check before I do it.

11

The build error on windows should be resolved with b874a0ead8.

12

Added an edit to state that a new version of make update script is needed.
Would be nice if the script somehow did the initial pull itself, to grab the new version of self and re-run. But that’s for another day.

13

How would this work when bisecting across this change? In such a case it’s not possible to run make update, because that would do too much.

14

Can you elaborate on the “would do too much”?

Going to a revision prior to this change brings back the old make update which had those issues for bisecting:

  • The main blender.git can not be updated because it is detached head. And you do not want to update the HEAD anyway.
  • The decision about what libraries and tests to pull did not work reliably either because the tag/branch of SVN repositories and submodules was based on the main repository branch name.

If you go back in history, run make update it will re-initialize the submodules, and most likely to pull the latest submodule state. This will be really the latest state for addons and addons_contrib. For translations and tools that’d be the state of repositories prior to this change.

If you go back to the latest main branch and run make update one of the things will happen:

  • If you never had the addons and addons_contrib initialized then the script will move the submodules over to the new location (so that if there is any local changes to them they are not lost)
  • If you had the addons and addons_contrib initialized you’ll see unstaged addons and addons_contrib folders under their old locations, and the make update will update the check-outs at the new locations.

Starting from this change and onwards, using make update in the middle of bisect will allow you to update libraries to the proper state since the code is now based on the BKE_blender_version.h.

I suggest that you jus try the bisect, see if there is anything wrong with it, and come with a proposal or workaround how to get try better.

1 Like
15

I’ve always had the libraries (and tests) in a git-svn clone for bisecting, to prevent hammering the svn server all the time. And also because svn is rather sluggish on my (mobile) internet and re-updating the complete libs folder over svn for every bisect step when you are flipping back and forth over a libs change was excruciatingly slow.

I saw some noise on the chat about git svn in make update, but can’t really find anything in make_update.py (admittedly I didn’t look too hard, just did some searches).

Now I know there are plans to switch the libs over to git lfs sometime so probably it’s no use trying to advocate the addition of a git-svn path in make_update.py, but I just wanted to throw it out here.

Since the switch to new libs in a different folder from a different repository my setup broke somewhat (still need to do manual stuff when crossing the lib switch date) but for me having the libs in git-svn made a night&day difference when bisecting because it never needed to re-download.

16

Hello, Good morning :slight_smile:

Can ‘make update’ (and build process) please recognize following tree as valid (if they exists) and, not re-download/pull these (addons/(-contrib)) repositories?

root
├─blender
├─blender-addons
├─blender-addons-contrib
├─lib, etc.

First of all, thank you for this reorganization.

The request is primarily because, with current(new) re-pulling addons into scripts make it harder (read messy) to keep archive of the these repositories as addons and addons-contrib does not actually exists within git tree (submodule or otherwise).

I use ‘git pull --all --rebase’ and previously it was ‘git pull --all --rebase && git submodule update --init --recursive --remote --rebase’.

I keep repositories with upper structure to keep all these repositories properly tracked.
So now, when ‘make update’ is called addons and addons-contrib gets re-pulled into scripts, which makes it duplicate efforts. And, these two actually does not exists within blender repository, as they are git ignored. Maybe think of it like how ‘lib/*’ are treated during build.

Hopefully I explained things somewhat coherently.

17

git svn was never used by make update.

18

You should be able to use make source_archive. This is what the release builds are using.

I do not think it is realistically to fit everyone’s workflow. Keeping the addons and addons-contrib inside of the scripts was an explicit choice to:

  • Support the workflows when developers don’t use make install and rely on symlinking to speedup iteration process.
  • Keep the Python API refactor doable form inside of a text editor without need to go into complicated setup with multi-work-tree workspaces. It is still possible to do modifications to RNA and do global search in all scripts.

So it is like a trade-offs no matter which path you take. Hopefully, the current situation is here for a not so long time, and eventually community addons will be moved to the extensions platform, and the core addons will become part of the main repository.