Rendertime explodes in 3.1 with shadowcatcher object compared to 3.01

Hello there,

I was about to transfer a scene over to 3.1 from 3.01. It´s containing a shadowcatcher object and rendertimes are exploding. Also it seems not possible to render the same scenario like before. Will elaborate on that later.

Rendertime comparison scene with shadowcatcher as floor object:

3.01 : 0:52 min
3.10 : 18:32 min

I will strip down the scene and hand it over. See below. No idea what I am doing wrong or what has changed so drastically.

Thanks
Oliver

EDIT:
Here is a simple scene. Built in 3.1.

Rendertime 2x2080 super:
3.01: 0:13 min
3.10: 1:38 min