Would be great to have such AI upscaling technique inside blender renderers in general.
5 Likes
Hmm. That’s not such a bad idea. I could try to do it in the meantime. However that would probably require a bit more research as this AI was made as a proof of concept for now in around 2 days.
9 Likes
How about AI to upscale actual volumetric grids? There are quite a few image upscaling algorithms already out there, so I am not sure it makes sense to reinvent the wheel. Volumetric grids on the other side, that could be interesting. The best we have so far is just wavelet noise. I think it would be really cool to have machine learning model, which would take low res volumetric grids and at least double their resolution in all 3 dimensions.
Rendertimes aren’t that much of an issue these days, but simulation times still are. And there’s also that annoying stereotype of simulating a lowres grid to find the simulation shape you like, just for it to end up completely different if you half the voxel size. The AI model could serve as a “I like the way this looks, I just want more detail” kind of button, but without the generic wavelet noise visual tradeoff 
7 Likes
Shhhh, not that loud 
. But honestly it was the main plan when implementing Neural Engine inside JFlow. I also have some ideas how to do this, as AI Research is my main job. It’s just quite hard and computationally heavy so I’ll have a lot of work to optimize it. But it is planned.
That’s also why I’ve tried first 2D upscaling as it’s much simpler for a concept.
Also I didn’t use third party AI upscaling models as they were much slower than my model and the only reason it is as a feature is that I wanted faster 4K renders than with native rendering. ESRGAN etc. just took too long to compute
18 Likes
I’ve been thinking, that since most of the fluid simulators suffer from the issue of the grid resolution changing the outcome (shape and look) of the simulation, a good idea for a dataset for a hypothetical AI model could be just simulating a bunch of different high res simulations as output/result training data, and their respective inputs being the very same simulations, just for example each 2nd or 4th voxel. So the low res input sample would be synthetic, downsampled version of the high res output sample. With such training data set, I’d expect the resulting network being really good at keeping the simulation shapes nearly identical, just more detailed. This could also save some time and effort generating the training set.
I don’t do any AI research or programming, but I thought it could be a good idea. Maybe it’s a common approach everyone knows already, and I’ve just reinvented a wheel
5 Likes
Don’t worry and thanks for help. Yes, it is a common technique and that’s exactly what I’m doing right now but good job with coming with such idea yourself 
.
The good thing is that with current JFlow generating high-res simulations is a matter of minutes so there shouldn’t be problem with the generation itself, but the bad thing is handling such amounts of data during the training and in terms of hard-drive space.
Anyways I’ll experiment with different techniques and hopefully come up with something performant and good looking.
7 Likes
Cycles render preview. Later there will be probably an animation but it takes ages in 4K (around 4minutes/frame)
29 Likes
Aaaand there it is. A Cycles render. It was rendered in around 720p and then upscaled to 4k with the custom AI model as a test.
34 Likes
Wow man, as i can only script a bit, it blows my mind, that you can do all that stuff. Physics, AI and so fast. I really hope you might end up as blender dev. That would be awesome ! Is the upscaling done in a separate program ?
5 Likes
Thanks,
For now the upscaling is in a C++/CUDA code of JFlow to be faster during rendering,
But for the Cycles renders I have a separate python script that I launch after, but it could be turned into C++ easly as I have the codebase already done.
And I would love to be a blender dev but for now I don’t meet the requirenments so I’m trying to create a nice demo so maybe it could work somehow
8 Likes
You don’t have to be a blender developer. If you successfully build this into the blender code, you can suggest this to be added to it.
I wouldn’t count on the design to stay the same, but it is certainly interesting.
1 Like
Few more tests of AI upscaling with Cycles:
18 Likes
Shader artefacts.
I’ve set the sampling in Cycles a bit wrong to get usable render times, because I didn’t have time for 15min/frame render times
Michal, have you seen this ?
2 Likes
Yup, it’s quite amazing. I’ll try to implement it in some future or do something similar
7 Likes
Last few days I’ve worked on better export passes to allow better and more realistic shading in blender. Still not ideal, but I think it’s slowly getting better
25 Likes
Beautiful as always! I was curious how it fares with some more delicate use-cases. Something like smoking a cigarette or a small campfire. Where you could really see the silhouette of each smoke/flame trail.
Something like this:
8 Likes
Okey, I’ll try it in the near future
4 Likes