Blender Buildbot worker node Dockerfile

docker
buildbot

#1

Hey,

On the rack I currently have some “idle” nodes that could possible be put to (light) use running a Docker swarm. What I am looking into atm is the possibility of running some Buildbot worker nodes on a few of the nodes that I have.

My idea would be to create a temporary repo, with the ultimate goal being to include per distro Dockerfile’s in the blender build_files/build_environment/ area that would replace the monolithic install_deps.sh approach to creating a build environment , with a Dockerfile that has that OS/distro installed, along with the necessary build environment for blender so that a Buildbot worker node can start connecting and pulling our repo’s.

Imagine a design where you have a docker environment to test RHEL, Fedora, Ubuntu, multiple versions of each, all available as buildbot workers that can build blender so that our Linux platforms are a little more tested.

I am poking around with things now (it takes a while to get everything prepared), but if I get something going, I can at least share and see. If all goes well, I can hand it off to the official maintainers and delete my temp repo after it is moved to something more official.

Thoughts?


#2

It’s not entirely clear to me what you are suggesting, what would the Docker file contain exactly? A list of distribution packages to install? Where would building of libraries not available as distribution packages be handled? Would the Docker file for each distribution need changes when upgrading libraries to new versions?

If we ever move away from install_deps.sh, then I think it should be towards the make deps system that is already used on Windows and macOS. Personally I even think having a lib/linux_x86_64 directory with precompiled static libraries is ideal.


#3

The Dockerfile would act as a sort of base install that would make up the docker image itself. This would be an immutable image that is used to create all of the worker nodes and connect to the buildbot master, which would then pull the repo’s and compile blender using the dependencies available in the base docker image.

It would be ideal if the blender makefile had a target like make deps that would do this for you, as it would simplify the Dockerfile to a single step during image creation, but to use Docker for worker nodes, you need an image to run the build slave software anyway, and you would probably wouldn’t want to wait an hour or two while the worker node rebuilds the base dependency packages before it finally gets to compiling blender, and then do this for each commit. Instead, you would want a base (the installed deps), and then only compile blender every commit.

So it comes down to that if I were to use the idle nodes for Docker (which I will likely use for other stuff anyway), why not use them for buildbot slave worker nodes compiling blender, in which case we need a Dockerfile no matter what.

As a sample, see: https://github.com/buildbot/metabbotcfg/blob/nine/docker/metaworker/Dockerfile

Now replace those apt-get commands with our commands (be it packages, or manually building them from source). This is all done during the create of the immutable docker image, which acts as our base.

Make a little more sense?


#4

Ok, that’s clear. But in that case we might as well just run install_deps.sh? That way we can test that it installs all the appropriate packages, builds the remaining ones, and builds Blender on all the those distributions as expected.

Doing the whole thing from scratch might actually be quite fast if we enable ccache and keep the downloaded package files. I can rebuild all Blender dependencies from source with make deps in about 5 minutes on a quad core using ccache, install_deps.sh might be in the same ballpark.


#5

Well you can manually build a Docker image up by starting with say Ubuntu plain, then running apt-get, making edits etc, and then save out a final image, but that is a hands on approach. Ideally you want something where you tell docker to build the image, and it just goes. The problem is that our install_deps.sh is a very interactive script, I believe?

As for the caching, each step in a Dockerfile is cached, and only a changed entry and everything below it are re-run when you update the file. Basically each step in that file is saved as a filesystem like layer, or blob, that is then combined to run the final image. So of you only were to edit the last command in a Dockerfile, docker would fire up an image of the layer right before that last line, and then run the last command to create a final image. The trick then is to create the Dockerfile intelligently by structuring it such that changes that happen infrequently tend to stay near the top, and more volatile changes are near the end.


#6

Ah, I see a --no-confirm option. Maybe it would be just fine then to use the script automated. I’ll give it a shot in a bit and let you know.


#7

Well, this Dockerfile appears to at least build. I have yet to actually try starting it, but it is a starting point. Just don’t ask how big that image is :blush: (3.29 GB)

In order to build this, install docker.io, drop the contents below into a file called Dockerfile in the Blender git ./build_files/build_environment/ directory and then run docker build .

Of course, you can name the image (with -t <name> I think).

#
# Provided here as a real life example on how to make a customized
# worker Dockerfile for your project

FROM        buildbot/buildbot-worker:master
MAINTAINER  Blender Buildbot Maintainers

# Switch to root to be able to install stuff
USER root

# This will make apt-get install without question
ARG         DEBIAN_FRONTEND=noninteractive

# Switch to root to be able to install stuff
USER root

# on debian postgresql sets default encoding to the one of the distro, so we need to force it for utf8
RUN apt-get update && apt-get install -y locales && rm -rf /var/lib/apt/lists/* && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG=en_US.utf8

# install the DB drivers we need to test against databases, as well as git and nodejs + phantomjs
RUN apt-get update && \
    curl -sL https://deb.nodesource.com/setup_6.x | bash - && \
    apt-get install -y python-software-properties software-properties-common && \
    # add python2.6 and python3.6 repositories
#    add-apt-repository ppa:fkrull/deadsnakes  && \
    apt-get update && \
    apt-get install -y \
#        libmysqlclient-dev \
#        libjpeg-dev \
#        libpq-dev \
#        # selenium is a java thing
#        default-jre \
#        # chromium needs xvfb even with --headless
        xvfb \
        chromium-browser \
        git \
        gconf2 \
        python-virtualenv \
        enchant \
        libenchant-dev \
        locales \
        aspell \
        aspell-en \
        ispell \
        iamerican \
        fontconfig \
        nodejs \
#        postgresql \
        sudo && \
#        python3.4-dev  \
#        python3.5-dev  \
#        python3.6-dev  \
#        python2.6-dev  \
#        mysql-server && \
    \
    localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 && \
    curl -sL https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 | \
    tar  --strip-components=1 -C /usr/local -xj phantomjs-2.1.1-linux-x86_64/bin/phantomjs && \
    npm install -g yarn && \
    yarn global add protractor coffee-script && webdriver-manager update --chrome --no-gecko &&\
    rm -rf /var/lib/apt/lists/*
RUN mkdir /blender/
COPY install_deps.sh /blender/
COPY patches/ /blender/
RUN \
    cd /blender/ && \
    ./install_deps.sh --with-all --no-confirm
#COPY pg_hba.conf /etc/postgresql/9.3/main/pg_hba.conf
#COPY sudoers /etc/sudoers
#COPY mysql /etc/init.d/mysql
#RUN \
#    /etc/init.d/postgresql start && \
#    su postgres -c "createuser buildbot" && \
#    su postgres -c "psql -c 'create database bbtest WITH ENCODING UTF8 TEMPLATE template0 ;'"
# Switch to regular user for security reasons
USER buildbot

# generate cache for the buildbot dependencies
RUN \
    mkdir -p /tmp/bb && \
    curl -sL https://github.com/buildbot/buildbot/archive/master.tar.gz | \
    tar  --strip-components=1 -C /tmp/bb -xz && \
    virtualenv /tmp/bb/sandbox && \
    . /tmp/bb/sandbox/bin/activate && \
    pip install -U pip && \
    pip install -e '/tmp/bb/master[test,docs,tls]' && \
    pip install -e /tmp/bb/pkg && \
    pip install -e /tmp/bb/www/base && \
    rm -rf /tmp/bb

Next would be to clean up the code directories and help reduce the image size a bit.


#8

After speaking with Sergey, apparently it is a much more complicated process to build an actual “release” build, which would be needed since buildbot would be making this file available for download.

Some of the highlights were that we use non stock compiler for more speed, and other tools that are normally available as a package need to be recompiled from source as well for reasons I can’t recall. Long story short, I would probably be better off starting with a blank docker image based on debian or similar.

I’ll look into it a bit (he has a chroot available for inspection) and see if I can come up with something manageable.


#9

Yes, it depends what the purpose is. If it is to make release builds for users to download, you only need one carefully setup docker image. If it is to test that Blender can be built and run on various distributions with the default compiler and packages, you want to change as little as possible.


#10

Sergey and I were discussing this a bit actually; the idea of having an official “release” capable image, and images that are like Ubuntu/RHEL/Suse/etc. that use distro packages as much as possible, or ideally, the install_deps.sh.

The issue with compiling official releases is that a bunch of things have to be setup in Jessie that would be mandatory since users would be downloading and running this file from the website. For distros such as Ubuntu or RHEL we could possibly get away with install_deps.sh only, but we would have to take care not to make those files available to download on the builder website.

I brought up the point that we could do more frequent compiles for distro specific (ie: non universal static) builds to try catch problems, but he doesn’t sound convinced that it would be worth the effort of maintaining separate docker images for. This, I can’t say, but it would have to be discussed. I personally think it would be a step in the right direction.

I am trying to sort through the files on biserver right now, but it’s a lot of stuff to go through. Sergey is busy with other things at the moment, so it’s going to be difficult to get any help from him with the official release setup. I am also not as familiar with compiling as the core devs are, either. I can get my own compiles just fine (usually) on say Ubuntu, but doing a full on release build in a chroot of Jessie with backports and a million other tweaks, is really more complicated than I feel it should be to build blender.

I am actually curious if Jessie was only picked because it’s what is running on the server, where the builds happen. If so, we could potentially start everything over, and come up with a much cleaner implementation. Problem is, I can’t do that alone.

Edit: My point being that Docker could potentially be a game changer in that we don’t need Jessie anymore, just because the environment runs on Jessie, in the form of a Docker image.


#11

To give you an idea of what it’s like to start from Jessie, I tried the below file (well, I haven’t rebuilt with the last part with Boost yet), right off the bat nothing works because cmake isn’t 3.5 in Jessie. Why are we basing on Jessie, again?

FROM debian:jessie
MAINTAINER Blender
USER root
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y sudo
RUN mkdir -p /tmp/blender/
COPY install_deps.sh /tmp/blender/
COPY patches/ /tmp/blender/
RUN \
    cd /tmp/blender/ && \
        ./install_deps.sh --with-all --no-confirm
RUN rm -rf /root/src
COPY check_release.py /tmp/
RUN apt-get install -y python3
RUN mkdir /root/blender-git/ && \
    git clone https://git.blender.org/blender.git /root/blender-git/blender && \
    cd /root/blender-git/blender/ && \
    git submodule update --init --recursive && \
    git submodule foreach git checkout master && \
    git submodule foreach git pull --rebase origin master
RUN cd /root && \
    wget https://sourceforge.net/projects/boost/files/boost/1.60.0/boost_1_60_0.tar.bz2 && \
    tar jxvf /root/boost_1_60_0.tar.bz2 -C /tmp

My point being that, if the first thing you have to do is spend half a day getting Jessie ready to even start to compile blender, when something like Ubuntu 16.04 or 18.04, out of the box, can do 99% of everything, then I start to question the base having outgrown it’s usefulness.


#12

I believe the main reason to use older distributions is to ensure Blender is compiled against older versions of glibc and other system libraries. As far as I know it’s the only practical way to be compatible with many Linux distributions.

Anyway, I don’t see a reason to set up a new release building environment, unless the Linux platform maintainer (Sergey) has a use for it. Testing building on various distributions could be useful to catch some issues earlier, but unless it’s really easy and little work to maintain, it’s probably not worth it.


#13

I think only glibc would be applicable for a static build?

No doubt that it would need to be of use for @sergey for it to actually be used. I tried peer pressure him into taking a look, but his will is too strong :stuck_out_tongue:

One approach to getting an image up is to just rsync the exact on disk files into an image, while carving out a few Docker volumes so tar files persist on disk, but it’s around 30G of files just to rsync locally for testing. Would be easier to just do it on the server, than copy everything remotely. It’s not my call to install Docker on biserver though. If anything, we need to clean it up, not add more stuff :slight_smile:

Anyway, we can always make this a spring cleaning project, or something down the road, assuming a better approach doesn’t present itself by then.


#14

Latest version of the Dockerfile:

FROM debian:jessie
MAINTAINER Blender
USER root
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y sudo
RUN mkdir -p /tmp/blender/
COPY install_deps.sh /tmp/blender/
COPY patches/ /tmp/blender/
RUN \
    cd /tmp/blender/ && \
        ./install_deps.sh --with-all --no-confirm
RUN rm -rf /root/src
# TODO: needs to be +x
COPY check_release.py /tmp/
RUN apt-get install -y python3
RUN mkdir /root/blender-git/ && \
    git clone https://git.blender.org/blender.git /root/blender-git/blender && \
    cd /root/blender-git/blender/ && \
    git submodule update --init --recursive && \
    git submodule foreach git checkout master && \
    git submodule foreach git pull --rebase origin master
# Need cmake >= 3.5 to compile blender
RUN echo "deb http://ftp.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/jessie-backports.list && \
    apt-get update && apt-get install -t jessie-backports -y cmake
COPY build-openal.sh /root/
RUN cd /root && \
    wget http://kcat.strangesoft.net/openal-releases/openal-soft-1.16.0.tar.bz2 && \
    tar jxvf /root/openal-soft-1.16.0.tar.bz2 -C /tmp && \
    mv /root/build-openal.sh /tmp/openal-soft-1.16.0 && \
    cd /tmp/openal-soft-1.16.0 && \
    ./build-openal.sh && \
    cd /root && \
    rm -rf /tmp/openal-soft-1.16.0 && \
    rm /root/openal-soft-1.16.0.tar.bz2
RUN apt-get install -y python3-pip
#RUN cd /root && \
#    wget https://pypi.python.org/packages/38/2d/290d33417c079a5248fcd06b0b8492acdd1851e54e4bdad54c3859dab600/requests-2.8.1.tar.gz && \
#    tar zxvf /root/requests-2.8.1.tar.gz -C /tmp && \
RUN cd /root && \
    wget https://sourceforge.net/projects/boost/files/boost/1.60.0/boost_1_60_0.tar.bz2 && \
    tar jxvf /root/boost_1_60_0.tar.bz2 -C /tmp
# Set make options
RUN cd /root/blender-git/ && \
    cmake -C blender/build_files/buildbot/config/blender_linux.cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_C_COMPILER=/usr/bin/gcc -D CMAKE_CXX_COMPILER=/usr/bin/g++ -D PYTHON_VERSION=3.6 -D WITH_PYTHON_INSTALL_NUMPY=ON -D WITH_PYTHON_INSTALL_REQUESTS=ON blender && \
    cd /root/blender-git/blender/ && \
    make -j6

It appears to compile, but still not an official release. Posting for R&D purposes :wink:


#15

Would not make sense to use a Docker volume for the Blender git clone , this is going to pull the same repo over and over again.

Also is this still working fine.I might use this soon.