Feedback with OIDN install_deps.sh in Kubuntu 18.04

Hello.
I have applied @StefanW denoise node and install_deps.sh patches on Base 6aef124:
https://developer.blender.org/D4304#124813

“./install_deps.sh” does not install OIDN by default.

“./install_deps.sh --with-oidn” download, compile and install OIDN successfully in /opt/lib

The resulting message to compile that “./install_deps.sh --with-oidn” shows at the end is:
make -j8 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D OPENCOLORIO_ROOT_DIR=/opt/lib/ocio -D WITH_OPENIMAGEIO=ON -D OPENIMAGEIO_ROOT_DIR=/opt/lib/oiio -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D OPENSUBDIV_ROOT_DIR=/opt/lib/osd -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D OPENVDB_ROOT_DIR=/opt/lib/openvdb -D BLOSC_ROOT_DIR=/opt/lib/blosc -D WITH_OPENIMAGEDENOISE=ON -D OIDN_ROOT_DIR=/opt/lib/oidn -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;x264;openjp2'"

But when I try to compile with that instruction, then it is not built with OIDN. This is what cmake-gui shows:

Edit:
Remember that I am not a developer, it is very likely that I did something wrong.

1 Like

Thanks for testing this. The install_deps.sh script for OIDN still is a work in progress, hopefully I should get this working a couple of days.

1 Like

I downloaded pre-built library and it works well.

I just updated and pushed a new version of install_deps.sh. Try again if it works now.

@RonanDucluzeau, Thank you. I was simply trying to give feedback for install_deps.sh process automation.

@StefanW, Do you say in master? Current install_deps.sh on master seems to contain no data about OIDN. Or, where to download the file/patch?

My mistake. I had it committed, but forgot to push. Try yet again.

Thanks.
I tried build with:
make -j8 BUILD_CMAKE_ARGS="-U *SNDFILE* -U *PYTHON* -U *BOOST* -U *Boost* -U *OPENCOLORIO* -U *OPENEXR* -U *OPENIMAGEIO* -U *LLVM* -U *CYCLES* -U *OPENSUBDIV* -U *OPENVDB* -U *COLLADA* -U *FFMPEG* -U *ALEMBIC* -D WITH_CODEC_SNDFILE=ON -D PYTHON_VERSION=3.7 -D WITH_OPENCOLORIO=ON -D OPENCOLORIO_ROOT_DIR=/opt/lib/ocio -D WITH_OPENIMAGEIO=ON -D OPENIMAGEIO_ROOT_DIR=/opt/lib/oiio -D WITH_CYCLES_OSL=ON -D WITH_LLVM=ON -D LLVM_VERSION=6.0 -D OSL_ROOT_DIR=/opt/lib/osl -D WITH_OPENSUBDIV=ON -D OPENSUBDIV_ROOT_DIR=/opt/lib/osd -D WITH_OPENVDB=ON -D WITH_OPENVDB_BLOSC=ON -D OPENVDB_ROOT_DIR=/opt/lib/openvdb -D BLOSC_ROOT_DIR=/opt/lib/blosc -D WITH_OPENIMAGEDENOISE=ON -D OPENIMAGEDENOISE_ROOT_DIR=/opt/lib/oidn -D WITH_ALEMBIC=ON -D ALEMBIC_ROOT_DIR=/opt/lib/alembic -D WITH_CODEC_FFMPEG=ON -D FFMPEG_LIBRARIES='avformat;avcodec;avutil;avdevice;swscale;swresample;lzma;rt;theora;theoradec;theoraenc;vorbis;vorbisenc;vorbisfile;ogg;x264;openjp2'"

Apparently there are problems with these two NOTFOUND variables:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
OPENIMAGEDENOISE_COMMON_LIBRARY (ADVANCED)
    linked by target "blender" in directory /media/disk1/@Blender_Git/blender/source/creator
OPENIMAGEDENOISE_MKLDNN_LIBRARY (ADVANCED)
    linked by target "blender" in directory /media/disk1/@Blender_Git/blender/source/creator

-- Configuring incomplete, errors occurred!
See also "/media/disk1/@Blender_Git/build_linux/CMakeFiles/CMakeOutput.log".
See also "/media/disk1/@Blender_Git/build_linux/CMakeFiles/CMakeError.log".
GNUmakefile:292: recipe for target 'all' failed
make: *** [all] Error 1

Wait. Now I remember that some time ago I had installed mkl-dnn on my system when I was trying to compile OIDN for myself. Maybe files installed on my system is the problem.
Edit:
I just searched the system and I don’t have mkl-dnn installed by myself.
Edit 2:
Probably some garbage that was left in my system when I tried to compile OIDN by myself in Kubuntu 18.04. Now I’m going to try in 19.04, but currently having some problems with “ninja” dependency. I will report later.
Thank you and sorry for inconvenience if this is due to my mistakes

While I try to solve “ninja” problem in 19.04, I show the differences of OIDN installation with install_deps.sh between 18.04 and 19.04 in the folders “oidn-1.0.0” and its corresponding symbolic link “oidn” folder.

** 18.04 **

$ ls oidn-1.0.0/lib
cmake libOpenImageDenoise.so libOpenImageDenoise.so.0 libOpenImageDenoise.so.1.0.0

$ ls oidn/lib
cmake libOpenImageDenoise.so libOpenImageDenoise.so.0 libOpenImageDenoise.so.1.0.0

** 19.04 **

$ ls oidn-1.0.0/lib
cmake libcommon.a libmkldnn.a libOpenImageDenoise.a

$ ls oidn/lib
cmake libOpenImageDenoise.so libOpenImageDenoise.so.0 libOpenImageDenoise.so.1.0.0

** cmake-gui 18.04 **

** cmake-gui 19.04 **

Weird thing (for me that I don’t understand much of this), in 19.04 it is finding those 18.04 missing libraries inside /opt/lib/oidn/lib/, when they appear inside /opt/lib/oidn-1.0.0/lib/ (only on 19.04)

Edit:
This is driving me crazy. The content with “ls” for 19.04 that I showed earlier is running “ls” from Kubuntu 18.04 pointing 19.04 folders. But doing it from “19.04” it shows the same content for oidn-1.0.0 folder and oidn sym link:

$ ls oidn/lib
cmake libcommon.a libmkldnn.a libOpenImageDenoise.a

$ ls oidn-1.0.0/lib
cmake libcommon.a libmkldnn.a libOpenImageDenoise.a

My 19.04 file system is Btrfs and maybe that was the problem about this when explore from 18.04.

If I download the prebuilt lib from here https://github.com/OpenImageDenoise/oidn/releases the directory structure is identical to the one that gets built on Linux Mint 19.2 (based on Ubuntu 18.04) with install_deps
No mkldnn and no libcommon to be found.

So, finally I was able to solve ninja problem and build successfully on 19.04. OIDN is working on 19.04!
So guys in Mint 19.2 are having the same problem that I have in Kubuntu 18.04.
Those that I have published above are the differences about how install_deps.sh installs OIDN in /opt.
Apparently 18.04 installs the binaries and 19.04 the sources?
Edit:
Ok, .so vs .a differences:

What version of CMake are you using? This needs 3.13 or higher.

On Kubuntu 18.04:
$ cmake --version
cmake version 3.10.2

So the only solution is to try to update cmake in the distro we use?

Edit:
No problem really, but it would be nice if you devs could solve the problem with buildbot builder so that it can build with OIDN support on Linux :grinning:

Updated cmake to 3.15.2 - Same problems as before…

It has almost worked in Kubuntu 18.04, but I get an error when running built blender. What I did from the terminal:

*Update cmake with pip:
pip install --upgrade cmake

*Verify the cmake version:
cmake --version
I get: cmake version 3.13.3

*Build OIDN again from install_deps.sh:
./install_deps.sh --with-oidn --build-oidn --force-oidn

I am sure that one of the above options (with/build/force) is not necessary, but just in case.

When install_deps.sh finishes, you copy the instruction that script shows at the end to build blender with oidn suport.
Also make sure that there are files with “.a” extension in the path “/opt/lib/oidn/lib/”.

*Build Blender with install_deps.sh instructions.
It builds successfully, but when I execute Blender I get:

$ ./blender
./blender: symbol lookup error: ./blender: undefined symbol: _ZN3tbb10interface78internal15task_arena_base24internal_max_concurrencyEPKNS0_10task_arenaE

I remember having this kind of strange error in other occasions.

Edit:
About “pip” to upgrade cmake. I think this is related to python (I had already installed python components with it). You may have to install it from your distro repositories (I don’t remember how I installed it)

It’s hard to diagnose from a distance, but it sounds to me like, in both of your cases, you have multiple versions/builds of OIDN and/or TBB on your system. Also, mkl-dnn, which OIDN is based on, requires TBB 2017 or newer.

1 Like

That’s right, I had garbage referred to TBB in my /usr/local/lib when I tried to install TBB by myself (libtbb.so.2). I have removed the file and now Blender starts (only TBB-related packages installed from ubuntu repos remain on my system.), OIDN is working! Thank you Stefan!

But I think that @SteffenD problem may be something else. You make sure that cmake update was correct with “cmake --version”. You then build oidn again with “./install_deps.sh --with-oidn --build-oidn --force-oidn”. And make sure that there are files with “.a” extension in the path “/opt/lib/oidn/lib/”.

@StefanW any specific reason we look for common in build_files/cmake/Modules/FindOpenImageDenoise.cmake ?

I’ve installed oidn system wide and it seems like this is marked as not found for me.
However, if I remove it from the cmake file, oidn support is build properly and is working. So this seems to be unneeded

When you build OIDN using make deps or install_deps.sh and you are using CMake 3.13 or newer, OIDN gets built as several .a files, including a libcommon.a which then is necessary to link. If you install OIDN through other methods, then that is something I have no control over and cannot prepare FindOpenImageDenoise.cmake for.

So finally I found some time to update cmake to 3.15.2 and rebuilt install_deps.sh --with-oidn --force-oidn.
Then I manually set the paths to all required libraries and the include folder and now it’s building and working!
Great!!

1 Like