Cycles AMD HIP device feedback

well done @bsavery, thanks for getting some past support going!

6 Likes

ROCm 5.1 is released With CRIU Support, More RadeonPro RDNA Enablement

3 Likes

Hello, i just read about proprietary repository of ROCm 5.1.

  • Can Blender work only with the open source repository of ROCm 5.1 ?
  • Is possibile to install just the open source components of ROCm 5.1 ?

https://docs.amd.com/bundle/ROCm-Release-Notes-v5.1/page/About_This_Document.html

1 Like

ROCm 5.1 src

1 Like

I read the documentation thank you.
For what i understand maybe we just need:

amdgpu-install -y --usecase=hip

and to control there is nothing proprietary check:

base-url// proprietary sub-folder.

Succe… not yet.
I installed AMDGPU 22.10 on Ubuntu 20.04.4 for my RX 5600XT and compiled the latest Blender including the changes mentioned in ⚙ D14360 Linux and GL Interop enablement for Cycles on HIP.

Now HIP appears (besides CUDA) in Settings > System. But it just shows a warning:
No compatible GPUs found for Cycles
Requires discrete AMD GPU with Vega architecture

The debug output complains about missing HIP files, although the are installed at /opt/rocm/hip/bin/
I0331 21:51:08.719480 53272 device.cpp:32] HIPEW initialization succeeded
I0331 21:51:08.720350 53272 device.cpp:42] Neither precompiled kernels nor HIPCC was found, unable to use HIP

Edit:
One step further. Adding /opt/rocm/hip/bin to $PATH enables the HIP settings for my RX 5600XT card.
Rendering starts… and crashes (Memory access fault by GPU node-1).

1 Like

I got as far as compiling and running blender, but it crashes for me when I try to render anything. I think I will wait for a proper release :slight_smile:

edit: crash log

# Blender 3.2.0, Commit date: 2022-03-31 17:14, Hash 42853bacc973
Read library:  '/home/xxx/rendering/classroom/assets/lamps/lamps.blend', '//assets/lamps/lamps.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/chairs/chairs.blend', '//assets/chairs/chairs.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/coatStand/coatStand.blend', '//assets/coatStand/coatStand.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/desks/desks.blend', '//assets/desks/desks.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/dustBin/dustBin.blend', '//assets/dustBin/dustBin.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/radiator/radiator.blend', '//assets/radiator/radiator.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/suitcase/suitcase.blend', '//assets/suitcase/suitcase.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/wallClock/wallClock.blend', '//assets/wallClock/wallClock.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/wastes/wastes.blend', '//assets/wastes/wastes.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/books/books.blend', '//assets/books/books.blend', parent '<direct>'  # Info
Read library:  '/home/xxx/rendering/classroom/assets/officeSupplies/officeSupplies.blend', '//assets/officeSupplies/officeSupplies.blend', parent '<direct>'  # Info
bpy.context.space_data.search_filter = "pose"  # Property
bpy.context.space_data.search_filter = "pose"  # Property
bpy.context.space_data.filter_text = "pose"  # Property
bpy.ops.outliner.item_activate(deselect_all=True)  # Operator
bpy.ops.outliner.delete()  # Operator
bpy.ops.outliner.item_activate(deselect_all=True)  # Operator
bpy.ops.outliner.item_activate(deselect_all=True)  # Operator
bpy.ops.outliner.delete()  # Operator
bpy.ops.outliner.delete()  # Operator
bpy.ops.outliner.item_activate(deselect_all=True)  # Operator
bpy.data.scenes["_mainScene"].(null) = True  # Property
bpy.data.scenes["_mainScene"].(null) = True  # Property
bpy.data.scenes["_mainScene"].(null) = True  # Property
bpy.data.scenes["_mainScene"].(null) = True  # Property
bpy.context.space_data.search_filter = ""  # Property
bpy.context.scene.cycles.device = 'GPU'  # Property
bpy.context.space_data.params.filename = "classroom2.blend"  # Property
Saved "classroom2.blend"  # Info

# backtrace
./blender(BLI_system_backtrace+0x33) [0x364d7f3]
./blender() [0xd468be]
/usr/lib/libc.so.6(+0x42560) [0x7f40bfbc3560]
/opt/rocm/hip/lib/libamdhip64.so(+0x1e8b3d) [0x7f4051de8b3d]
/opt/rocm/hip/lib/libamdhip64.so(hipTexObjectCreate+0x8e1) [0x7f4051df1901]
./blender(_ZN3ccl9HIPDevice9tex_allocERNS_14device_textureE+0x899) [0x2642849]
./blender(_ZN3ccl12ImageManager17device_load_imageEPNS_6DeviceEPNS_5SceneEiPNS_8ProgressE+0x336) [0x2b9e9b6]
./blender() [0x2dcd86f]
/usr/lib/libtbb.so.12(+0x20abe) [0x7f40d4f0cabe]
/usr/lib/libtbb.so.12(+0x22d44) [0x7f40d4f0ed44]
/usr/lib/libc.so.6(+0x8d5c2) [0x7f40bfc0e5c2]
/usr/lib/libc.so.6(clone+0x44) [0x7f40bfc93584]

# Python backtrace
2 Likes

Work is underway to do that. Vega support has been added to Cycles HIP (it does have issues at the moment and an investigation into fixing it is underway). And there is a patch for enabling Linux support here: ⚙ D14360 Linux and GL Interop enablement for Cycles on HIP

2 Likes

The latest build from @brecht (⚙ D14360 Cycles: enable HIP binaries on Linux) seems to work. It renders the classroom scene without a problem on an RX 5600XT.
Great, great, great!

2 Likes

It’s not working for me in Arch Linux and 5700XT, I get the same crashes. There is a chance that the package for Arch Linux is to blame (which is why I’m testing alpha blender versions). Maybe someone else has better luck.

I don’t think Arch packages have been fully updated to 5.1 yet, for example this one is on 5.0.2 still.
https://aur.archlinux.org/packages/hip-runtime-amd

For Vega on Linux, we still have to investigate errors there, it’s not expected to be working yet.

1 Like

Installing HIP on Arch is confusing. It should be documented on the arch blender wiki when someone finds out what works.

hip-runtime-amd is derived from the open source GitHub version of HIP. I would think Blenders building and testing process uses the amdgpu-install version from repo.radeon.com, which is equivalent(ish) to opencl-amd.

I tested D14360 on Arch 5.17.1 with opencl-amd and a RX6800 and got a GPU crash on the default cube.

kernel: [drm:amdgpu_dm_atomic_commit_tail [amdgpu]] *ERROR* Waiting for fences timed out!
kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=129284, emitted seq=129286
kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process blender pid 3117 thread blender:cs0 pid 3147

@Luciddream did you use the pre-compiled D14360 for testing?
@tschipie what version of Linux and amdgpu-install method did you use?

Thanks for the hard work.

2 Likes

@brecht I maintain opencl-amd which is using the Ubuntu compiled binaries for 5.1, I don’t use the Arch Linux source packages.

@L_S Yes I used D14360. I’m running it with AMD_SERIALIZE_KERNEL=3 AMD_SERIALIZE_COPY=3 AMD_LOG_LEVEL=4 ./blender and the error I get is

||:3:hip_texture.cpp          :1453: 0524239113 us: 2918 : [tid:0x7f6ac77ff640] hipTexObjectCreate (   0x7f6,ac5,c82,de8, 0x7f6,ac7,7b9,140, 0x7f6,ac7,7b9,0d0, char array:<null> )|
||:4:rocdevice.cpp            :2034: 0524239276 us: 2918 : [tid:0x7f6ac77ff640] Allocate hsa device memory 0x7f690a400000, size 0x1e4000|
||:3:rocdevice.cpp            :2073: 0524239284 us: 2918 : [tid:0x7f6ac77ff640] device=0x7f6acedbe000, freeMem_ = 0xf9f5d340|
||:4:rocdevice.cpp            :1899: 0524239402 us: 2918 : [tid:0x7f6ac77ff640] Allocate hsa host memory 0x7f6ad4e1e000, size 0x110|
||Writing: /tmp/classroom.crash.txt|
fish: Job 1, 'AMD_SERIALIZE_KERNEL=3 AMD_SERI…' terminated by signal SIGSEGV (Address boundary error)

When I compiled blender myself the day before I also got unhandled SGPR spill to memory errors. But at the same time HashCat HIP is also not working, so maybe I was overconfident with opencl-amd package. At this point what I’m thinking is setting up Ubuntu in a different SSD to see if any HIP software is working there.

Interesting, when I run the following on the default cube, I get over 5000 lines with the last few shown below. Might have to try Ubuntu to see if its not a hardware fault.

AMD_SERIALIZE_KERNEL=3 AMD_SERIALIZE_COPY=3 AMD_LOG_LEVEL=4 ./blender &> ./terminal.txt

:4:command.cpp              :339 : 1658509288 us: 4869 : [tid:0x7fdf8087f640] command is enqueued: 0x7fde77fab300
:4:rocvirtual.cpp           :984 : 1658509291 us: 4869 : [tid:0x7fdf8087f640] HWq=0x7fdfab21c000, BarrierAND Header = 0x1503 (type=3, barrier=1, acquire=2, release=2), dep_signal=[0x0, 0x0, 0x0, 0x0, 0x0], completion_signal=0x7fdfdc620180
:4:rocvirtual.cpp           :525 : 1658509294 us: 4869 : [tid:0x7fdf8087f640] Host wait on completion_signal=0x7fdfdc620180
:3:rocvirtual.hpp           :62  : 1658509296 us: 4869 : [tid:0x7fdf8087f640] Host active wait for Signal = (0x7fdfdc620180) for -1 ns
amdgpu: amdgpu_cs_query_fence_status failed.
amdgpu: The CS has been cancelled because the context is lost.

I quickly hacked together a PKGBUILD for hip-runtime-amd-5.1.0 and got Blender to enable HIP support on my 5700XT. But when I try to actually render anything with GPU compute enabled, Blender either crashes immediately or just hangs with the GPU at 100% load. If I try to use the Cycles viewport with GPU compute enabled, it seems to draw one frame, then Blender freezes.

Those are the last couple of lines before Blender locks up (with AMD_LOG_LEVEL=4):

:3:rocvirtual.cpp           :2738: 6857117724 us: 23934: [tid:0x7f55f45b9640] ShaderName : kernel_gpu_integrator_shade_surface
:4:rocvirtual.cpp           :827 : 6857117728 us: 23934: [tid:0x7f55f45b9640] HWq=0x7f57301d0000, Dispatch Header = 0x302 (type=2, barrier=1, acquire=1, release=0), setup=3, grid=[176128, 1, 1], workgroup=[1024, 1, 1], private_seg_size=8080, group_seg_size=0, kernel_obj=0x7f55f240d040, kernarg_address=0x7f5730080180, completion_signal=0x0
:3:hip_module.cpp           :427 : 6857117732 us: 23934: [tid:0x7f55f45b9640] hipModuleLaunchKernel: Returned hipSuccess : 
:3:hip_context.cpp          :236 : 6857117735 us: 23934: [tid:0x7f55f45b9640] hipCtxPopCurrent ( char array:<null> )
:3:hip_context.cpp          :249 : 6857117738 us: 23934: [tid:0x7f55f45b9640] hipCtxPopCurrent: Returned hipSuccess : 
:3:hip_context.cpp          :253 : 6857117742 us: 23934: [tid:0x7f55f45b9640] hipCtxPushCurrent ( context:0x7f5,726,4d5,000 )
:3:hip_context.cpp          :263 : 6857117744 us: 23934: [tid:0x7f55f45b9640] hipCtxPushCurrent: Returned hipSuccess : 
:3:hip_memory.cpp           :1186: 6857117750 us: 23934: [tid:0x7f55f45b9640] hipMemcpyDtoHAsync ( 0x7f5,719,071,c30, 0x7f5,700,222,000, 4, stream:0x7f5,710,714,900 )
:4:command.cpp              :339 : 6857117753 us: 23934: [tid:0x7f55f45b9640] command is enqueued: 0x7f5719021900
:4:rocvirtual.cpp           :969 : 6857117758 us: 23934: [tid:0x7f55f45b9640] HWq=0x7f57301d0000, BarrierAND Header = 0x1503 (type=3, barrier=1, acquire=2, release=2), dep_signal=[0x0, 0x0, 0x0, 0x0, 0x0], completion_signal=0x7f573016da00
:4:rocvirtual.cpp           :524 : 6857117761 us: 23934: [tid:0x7f55f45b9640] Host wait on completion_signal=0x7f573016da00
:3:rocvirtual.hpp           :61  : 6857117764 us: 23934: [tid:0x7f55f45b9640] Host active wait for Signal = (0x7f573016da00) for -1 ns

Here’s what the viewport render looks like when Blender freezes. It seems to start, so something appears to be working:

1 Like

Unfortunately I have to backpedal a bit here, while I was able to active my GPU in the preferences and change it to GPU in the render settings, Blender actually still seems to render only via my CPU.
Rendering times don’t change between CPU or GPU settings. I didn’t notice yesterday right away, as my first render was just a simple default cube - which was fast enough to fool me into believing the GPU rendered it (and I might have underestimated my new CPU :slight_smile:

So while all settings seem to say GPU it still uses the CPU and otherwise crashes like everyone else. Actually I feel a bit like an idiot, but i was so thrilled to have first light that I forgot to do some sanity checks.

2 Likes

When I enable GPU rendering the GPU is listed in the verbose settings, but there seems to be problem with HIP using it (hipErrorInvalidValue in hip_device_runtime.cpp):

:3:rocdevice.cpp            :414 : 2518353392 us: 20809: [tid:0x7fc613447180] Initializing HSA stack.
:3:comgrctx.cpp             :33  : 2518380086 us: 20809: [tid:0x7fc613447180] Loading COMGR library.
:3:rocdevice.cpp            :205 : 2518383646 us: 20809: [tid:0x7fc613447180] Numa selects cpu agent[0]=0x7fc5d5ce1e00(fine=0x7fc5d65f5880,coarse=0x7fc5d6a655c0) for gpu agent=0x7fc5d5e82e00
:3:rocdevice.cpp            :1603: 2518383762 us: 20809: [tid:0x7fc613447180] HMM support: 1, xnack: 0, direct host access: 36

:4:rocdevice.cpp            :1899: 2518383793 us: 20809: [tid:0x7fc613447180] Allocate hsa host memory 0x7fc5ee85e000, size 0x28
:4:rocdevice.cpp            :1899: 2518384131 us: 20809: [tid:0x7fc613447180] Allocate hsa host memory 0x7fc5aff00000, size 0x101000
:4:rocdevice.cpp            :1899: 2518384537 us: 20809: [tid:0x7fc613447180] Allocate hsa host memory 0x7fc5afd00000, size 0x101000
:4:runtime.cpp              :82  : 2518384773 us: 20809: [tid:0x7fc613447180] init
:3:hip_context.cpp          :50  : 2518384777 us: 20809: [tid:0x7fc613447180] Direct Dispatch: 1
:3:hip_context.cpp          :145 : 2518384785 us: 20809: [tid:0x7fc613447180] hipInit: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :507 : 2518384792 us: 20809: [tid:0x7fc613447180] hipGetDeviceCount ( 0x7ff,f8a,594,130 )
:3:hip_device_runtime.cpp   :509 : 2518384796 us: 20809: [tid:0x7fc613447180] hipGetDeviceCount: Returned hipSuccess : 
:3:hip_device.cpp           :117 : 2518384801 us: 20809: [tid:0x7fc613447180] hipDeviceGetName ( 0x7ff,f8a,594,1d0, 256, 0 )
:3:hip_device.cpp           :139 : 2518384804 us: 20809: [tid:0x7fc613447180] hipDeviceGetName: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :141 : 2518384809 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,150, 23, 0 )
:3:hip_device_runtime.cpp   :331 : 2518384812 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :141 : 2518384816 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,180, 61, 0 )
:3:hip_device_runtime.cpp   :331 : 2518384819 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :141 : 2518384823 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,144, 69, 0 )
:3:hip_device_runtime.cpp   :328 : 2518384826 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipErrorInvalidValue : 
:3:hip_device_runtime.cpp   :141 : 2518384830 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,148, 67, 0 )
:3:hip_device_runtime.cpp   :331 : 2518384832 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :141 : 2518384836 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,14c, 68, 0 )
:3:hip_device_runtime.cpp   :331 : 2518384839 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipSuccess : 
:3:hip_device_runtime.cpp   :141 : 2518384844 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute ( 0x7ff,f8a,594,134, 18, 0 )
:3:hip_device_runtime.cpp   :331 : 2518384847 us: 20809: [tid:0x7fc613447180] hipDeviceGetAttribute: Returned hipSuccess : 
I0402 21:40:41.477293 20809 device.cpp:174] Device has compute preemption or is not used for display.
I0402 21:40:41.477303 20809 device.cpp:177] Added device "AMD Radeon RX 5600 XT" with id "HIP_AMD Radeon RX 5600 XT_0000:0b:00".
1 Like

Update for 3.1 open data as per late March 2022. Might update when open data supports 3.2 to add OneAPI / HIP Linux / HIP ray-tracing if available.

Additions

  • Now reports “Score”
  • Metal was added - interesting to see it out perform HIP and support Polaris (RX 580).

Warning about the following chart

  • Opendata has very few entries for 3.1 only devices 4+ samples are shown
  • Opendata by design has multiple uncontrolled variables between samples including supporting hardware and operating system.
  • Laptop, professional and Apple silicon GPU’s removed.
  • There could be errors, I didn’t check it and I’m not a statistician.
  • RX 6800 data was split, data around 1460 was used as the values in the 400’s didn’t make sense.

EDIT: at request of bsavery price data removed and sorted.

2 Likes

I kind off agree on removing laptop GPUs, but proffesional would be nice on this chart.

Apple Metal outperforms AMD HIP on their own devices ? Has this been verified ? Plus adding support to unsupported cards. What are the issues that prevent AMD from supporting the same cards ?