Does Blender have a post-mortem debugging infrastructure set up? By that I mean hosting a source server and a symbol server that we can use to augment crash/minidumps.
Hi, there is none at the moment…
Too bad. Would be very handy if people could just share a minidump / DSYM with developers when things crash. Should give so much more insight in crashes.
there was a lot of progress made toward this, but the patch never got merged in… it was going to be 2.81, then 2.82, then it was abandoned. not sure why, I guess people got busy with other things and there wasn’t enough interest? maybe @LazyDodo has some insights.
Sure I can share some of my insights on the patch you linked, but i have to point out that patch is/was substantially different from what @LZandman is asking about here (symbol+source server)
There were resource (people wise) constraints on the server side of things and this kept being kicked down the road till the point where there was just no interest from anyone to get it in, knowing a losing battle when i see one, the patch was abandoned.
But that left a gap, reports were still coming in from users with crashes we could not reproduce and we had no idea what could be causing them.
So in D7520 support was added on windows for writing usable stack traces to blenders crash log, so we could request those from end users. Sadly due to size restrictions we could not ship full debugging symbols, and had to settle for a reduced set, so the line information is missing for the blender.org builds (but if you were so inclined you can turn off WITH_WINDOWS_STRIPPED_PDB in cmake for your own builds, to get that information) but at-least we’d get a stack trace with the right function names.
So yeah, it’s nowhere close to what i envisioned, but it’s definitely better than what it was.
If the blender admins ever decide this is something worth having I’d be delighted to work on it again, but at this point in time it doesn’t seem very likely.
Yeah, wasn’t asking about a Rolls Royce. Of course using an online crash reporting/analysis service would be great. I think Microsoft and Apple even offer such things for apps distributed through their online stores.
But just having symbols for official Blender builds available would be very valuable, IMHO. We can create symbols ourselves while building, but then we’d also have to distribute the accompanying private builds to users. That’s not really advisable.
Well some good news there, ever since D7520 we ship stripped symbols with blender, so although not on a server, and not quite as useful as the private symbol file, it’s a whole lot better than what was there previously. (and good enough to cover our immediate needs on the bug tracker)
Blender currently doesn’t write out any dumps while it crashes on windows, stack trace into a text file only, so there technically isn’t anything yet to augment with a symbol/source server. There’s plenty of improvements that can be done in this area, but unless the admins change their minds there is very little I can do given it’s more of an organizational/priority problem than a technical one.
So currently the entire build output of BuildBot’s builds is thrown away, except for the final downloadable installers/packages?
Yes, no, half, due to the high build times of blender, incremental builds are used in the normal CI pipeline where possible (Except naturally for official releases which use fully clean build) but for the purposes of this conversation, yes, only the artifacts we require are kept and the rest gets disposed of.