Building Blender with CUDA on CentOS - gcc Version?

Hi Blender Dev Talk!

I am having some trouble trying to build Blender with CUDA enabled on Linux machines. There seems (to me) that the documentation requires two conflicting dependencies:

  • CUDA SDK 10.1 or 10.2 (requires gcc < 8)
  • Precompiled libraries (requires gcc >= 9.3)

When I try to compile with -DWITH_CYCLES_CUDA_BINARIES=ON and gcc 9.3 I get the following error:

Blender Cycles Kernel Build Error
Scanning dependencies of target cycles_bvh
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh2.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_binning.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_build.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_embree.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_node.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_optix.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_sort.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_split.cpp.o
[ 10%] Building CXX object intern/cycles/bvh/CMakeFiles/cycles_bvh.dir/bvh_unaligned.cpp.o
[ 10%] Linking CXX static library ../../../lib/libcycles_bvh.a
[ 10%] Built target cycles_bvh
Scanning dependencies of target cycles_graph
[ 10%] Building CXX object intern/cycles/graph/CMakeFiles/cycles_graph.dir/node.cpp.o
[ 10%] Building CXX object intern/cycles/graph/CMakeFiles/cycles_graph.dir/node_type.cpp.o
[ 10%] Building CXX object intern/cycles/graph/CMakeFiles/cycles_graph.dir/node_xml.cpp.o
[ 10%] Linking CXX static library ../../../lib/libcycles_graph.a
[ 10%] Built target cycles_graph
Scanning dependencies of target cycles_kernel_cuda
[ 10%] Generating filter_sm_30.cubin
In file included from /usr/local/cuda/bin/../targets/x86_64-linux/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/local/cuda/bin/../targets/x86_64-linux/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
make[2]: *** [intern/cycles/kernel/filter_sm_30.cubin] Error 1
make[1]: *** [intern/cycles/kernel/CMakeFiles/cycles_kernel_cuda.dir/all] Error 2
make: *** [all] Error 2
The command '/bin/sh -c cd build_linux_bpy &&     make install' returned a non-zero code: 2

When I try to compile with -DWITH_CYCLES_CUDA_BINARIES=ON and devtoolset-7's gcc 7.3.1, I get the following:

CMake Configuration Error
cd build_linux_bpy &&  cmake ../Blender/blender -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DWITH_CYCLES_CUDA_BINARIES=ON -DWITH_CYCLES_DEVICE_OPTIX=ON -DOPTIX_ROOT_DIR=/blenderpy/NVIDIA-OptiX-SDK-7.1.0-linux64/SDK
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- Check for working C compiler: /opt/rh/devtoolset-7/root/usr/bin/cc
-- Check for working C compiler: /opt/rh/devtoolset-7/root/usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /opt/rh/devtoolset-7/root/usr/bin/c++
-- Check for working CXX compiler: /opt/rh/devtoolset-7/root/usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- WITH_DRACO requires WITH_PYTHON_INSTALL to be ON, disabling WITH_DRACO for now
-- Performing Test SUPPORT_SSE_BUILD
-- Performing Test SUPPORT_SSE_BUILD - Success
-- SSE Support: detected.
-- Performing Test SUPPORT_SSE2_BUILD
-- Performing Test SUPPORT_SSE2_BUILD - Success
-- SSE2 Support: detected.
-- Found Git: /usr/local/bin/git (found version "2.28.0") 
CMake Error at build_files/cmake/platform/platform_unix.cmake:42 (message):
  GCC version must be at least 9.3 for precompiled libraries, found 7.3.1
Call Stack (most recent call first):
  CMakeLists.txt:882 (include)


-- Configuring incomplete, errors occurred!
See also "/blenderpy/build_linux_bpy/CMakeFiles/CMakeOutput.log"

I also tried installing a more recent version of the CUDA SDK, however the problem is that the more recent versions don’t have certain device definitions (or something like that…?) that Blender’s source code depends upon so that build fails as well.

I can try this without the prebuilt libs but wanted to ask first since I figured that there must have been someone out there who encountered this same issue, or at least I hope so.

I can’t imagine that nobody else has run into this because it even seems that on the NVIDIA CUDA Linux System Requirements that none of the OSs specified for version 10.2 of CUDA use gcc 9.3, or even close to it? Which is part of my confusion.

Here's the Dockerfile if it helps
FROM quay.io/pypa/manylinux2014_x86_64 AS builder

WORKDIR /blenderpy

RUN yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo && \
    yum -y clean all

RUN yum -y install \
    cuda-toolkit-10-1.x86_64  \
    gcc \
    gcc-c++ \
    glew-devel \
    libffi-devel \
    libX11-devel \
    libXcursor-devel \
    libXi-devel \
    libXinerama-devel \
    libxml2-devel \
    libXrandr-devel \
    make \
    openssl-devel && \
    yum -y clean all

ADD https://github.com/Kitware/CMake/releases/download/v3.17.3/cmake-3.17.3-Linux-x86_64.tar.gz cmake-3.17.3-Linux-x86_64.tar.gz
RUN tar xzf cmake-3.17.3-Linux-x86_64.tar.gz
ENV PATH="/blenderpy/cmake-3.17.3-Linux-x86_64/bin:${PATH}"

COPY . /blenderpy

RUN Optix/NVIDIA-OptiX-SDK-7.1.0-linux64-x86_64.sh --include-subdir --skip-license

RUN mkdir build_linux_bpy
RUN cd build_linux_bpy && \
    cmake ../Blender/blender -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DWITH_CYCLES_CUDA_BINARIES=ON -DWITH_CYCLES_DEVICE_OPTIX=ON -DOPTIX_ROOT_DIR=/blenderpy/NVIDIA-OptiX-SDK-7.1.0-linux64/SDK
RUN cd build_linux_bpy && \
    make install

RUN /blenderpy/.vscode/tasks/unix/build_manylinux_wheels.sh

Many thanks in advance!

This would happen on windows all the time with cuda throwing a fit on the MSVC version, this got to the point where i added a nvrtc based kernel builder to decouple the cycles kernel from the host compiler (there is just device code in the cycles kernel so the host compiler, is virtually irrelevant) this is not a codepath that gets a whole lot of exercise but you can try giving it a whirl by turning on WITH_CYCLES_CUBIN_COMPILER and WITH_CYCLES_CUBIN_COMPILER_OVERRRIDE in cmake.

Well, it got a bit farther, but stopped with the following error.

Building /blenderpy/Blender/blender/intern/cycles/kernel/kernels/cuda/kernel.cu
"/usr/local/cuda/bin/ptxas" /tmp/38b8-eb3a-6990-4dea -o /blenderpy/build_linux_bpy/intern/cycles/kernel/kernel_sm_75.cubin --gpu-name sm_75 -m64 --verbose
[ 12%] Generating filter_compute_75.ptx
Error: target shader model not set (-target), required

Error: invalid parameters, exiting
make[2]: *** [intern/cycles/kernel/filter_compute_75.ptx] Error 1
make[1]: *** [intern/cycles/kernel/CMakeFiles/cycles_kernel_cuda.dir/all] Error 2
make: *** [all] Error 2
The command '/bin/sh -c cd build_linux_bpy &&     make install' returned a non-zero code: 2

Seems like a pretty annoying issue. Thanks for your help!

try adding the parameter in intern\cycles\app\cycles_cubin_cc.cpp just look for --gpu-name and you’d be in the right area

At first glance to me it looks like I could just set the target name to be the same as the device name in that source file you directed me to, but it looks like from the help doc that’s in that same file the target option should be specified externally so I’m just not sure if that’s the way to go or not.

What do you suggest that I set settings.target_arch to?

Oddly, line 162-164 seems to suggest to me that the -target is being supplied… maybe something else is going on here? I’ll look at it more later.

string ptx = "\"" + cudapath + "ptxas\" " + settings.ptx_file + " -o " + settings.output_file +
               " --gpu-name sm_" + std::to_string(settings.target_arch) + " -m" +
               std::to_string(settings.bits);

Since my build log shows

“/usr/local/cuda/bin/ptxas” /tmp/38b8-eb3a-6990-4dea -o /blenderpy/build_linux_bpy/intern/cycles/kernel/kernel_sm_75.cubin –gpu-name sm_75 -m64 --verbose

I would figure that the sm_75 portion of that was determined by the -target flag unless I am gravely mistaken.

Please see the links in this comment.

1 Like

Creating the symlink worked for me, and although it is a very strange fix in my opinion, It’s perfectly fine for me to mess with the Docker configuration since I’m just going to copy the wheels out of the image.

Thanks for your help. The build worked after symlinking.