I wonder if there is a defined list of things that could be improved in Blender’s code to ease the implementation of third party render engines like LuxCore.
I think you, @BYOB, may be the most expert developer with an idea of what could be improved, and maybe little by little if could be improved by the community (if not by core devs)
Provide a ctype / numpy array of vertices or image textures without having to iterate through loops etc.
(most of export time is spent here in python code)
I am really not into the details, but I saw that the Cycles addon uses a depsgraph.as_pointer() - isn’t that already doing exactly what you are suggesting?
OTOY Octane render distributed as Client-Server package.
Custom Blender build inherit GPL license. And you can ask for source code from OTOY if you are legal owner of license.
But having this source code give you zero profit. Only things they changed is some optimization on IPR part required for faster live view and may be some specific file format for own flavor of alembic they used.
Main GPU renderer as not used library linking but using sockets is not inherit GPL, and OTOY can ask you walk somewhere instead of give you a code…
if you are legal owner of license.
I want to specify, not only if you are legal owner of the license, also if you have, by any means, that binary of Blender, no matter how you got it, you must have received a document where it explains how to get that source code, or the source code from the person that gave you that binary, otherwise they would be breaking the GPL.
Since Octane has a free version for Blender it should not be a problem to ask them for the source code of that Blender version, of course not for Octane 
I personally am interested in the display driver version of Remderman, TBH it seems a pretty clever and efficient way to connect a render engine with any other package, not just Blender
Some relief, yes
We (ProRender developers) are the ones who committed that 
so yes it did, but I’d call it more self relief.
The biggest limiting factors for Python addon render engines are, in my opinion:
depsgraph.object_instances and other potentially large depsgraph data structures. Currently they have to be iterated in Python, and there’s one element for each particle (easily millions). Best would be if such huge lists had a C++ API or would support the Python buffer interface, so they could be passed to a C++ Python module.as_pointer() and re-interpreting it as a Blender mesh struct in C++, but because this is an internal Blender datastructure, it can change between Blender versions.particle_system.co_hair(object=obj, particle_no=pindex, step=step) for each hair point (easily millions).Images are actually the least of our troubles, because we read them directly from disk.
But I also have a wishlist for image pixel access, which is important for packed files and for image formats unsupported by our own image loading libs:
The new image.pixels.foreach_get() is nice, but if the image data is not yet loaded into Blender, it behaves as a blocking call and freezes everything until the image is loaded (which can take a few seconds for large images on slow storage). It would be great if there was an asynchronous API to avoid the freezing, for example:
image.pre_load() could be a non-blocking call that triggers image loading in a background thread and immediately returns. Maybe it would also make sense if we could pass a callback function to it that gets notified once the loading is done: image.pre_load(on_loading_done)
image.is_loaded() could return wether the image was loaded yet, so we can poll for it.image.is_loaded() returns True or the callback is called, it would be safe to call image.pixels.foreach_get() and the only freeze happening would be the time it takes to copy the data.It’s a great list. Thank you for sharing this.
I believe BtoA (Arnold for Blender integration) development would benefit a lot from these improvements as well.
I think some of those things could be a problem for private engines like Arnold, since the GPL does not allow to use the exact same structures, however those things would be super useful for render engines like LuxCore, and we need better performance, Cycles is awesome, but having access to other technologies and render engines, specialized ones and such, is an important thing, specially if they are open source. and at the same time Blender would be supporting those open source projects by allowing a better integration for what already it’s an awesome engine
Now, the question is: is this something that has to be done by the core devs?.
I mean, Blender is not an easy thing to modify, specially in some parts, are these things mentioned here part of the “only core devs will do it properly”, or can it be improved by community developers?.
I am thinking not just in @BYOB of course, it can’t be his responsibility, but many others that may be interested in this kind of improvements, and for sure this is super interesting from a user perspective.
At this point I think the @brecht or @sergey point of view could be of help.
@MetinSeven if others agree, can we split this thread into another one that could be more focused on the new topic?
I mean this has nothing to do with Octane or the GPL anymore, we are talking about improvements that could improve integration of engines like LuxCore or Appleseed into Blender.ç
I would say we can split the thread since this post:
https://devtalk.blender.org/t/octanerenderer-and-gpl/19188/13
Or since the previous one where I ask about this, I don’t care, it’s just to stay on topic and not relate it to Octane or the GPL.
@JuanGea, this is not new information, we’re aware the API can be improved in these areas. It just has not become a priority for core devs yet.
Thanks, I was asking for your opinion more towards the idea of an external developer / contributor doing this, or if this is something that has such complexity (and it’s potentially dangerous since it touches the API) that should be done by a core dev.
Basically because there are several experienced developers working in several render engines, so I’m pretty sure that amongst them there are developers that are capable of doing such changes, but maybe the complexity of the changes inside the project is an obstacle and it’s something better done by core devs.
Anything can be done by a core or external developer, there’s just degrees of how much time people want to invest. It’s hard to estimate how much that is without knowing the developer.
For hair and particles, the least work would be to wait for new particle and hair systems. The new data structures have plain arrays of floats and don’t need any special conversion. A change that could be made sooner would be to add a utility function to convert the old particles to new hair and point cloud data structures. That will be useful in general, at a minimum for backwards compatibility.
For meshes ideally internal data structures would also move to such plain arrays, but that’s very hard.
For instances, it’s possible to add an API function that returns a flat array of all instances somehow, but this may need some deeper changes as some data gets invalidated after every iteration.
A C++ API in general has a big maintenance cost, and so would also require a commitment to maintain it.
For image loading, Blender itself does not support doing this in parallel, there is a mutex lock for that so even doing it in the background would not really work. This could be solved with some refactoring though.
I don’t know if async loading APIs are the right solution. You could imagine this for all kinds of operations, image loading, volume loading, generating instances, … and adding an async API for each potentially expensive operation seems not great to me. I think it would be better to use mutlithreading here, I think that’s possible if the Python GIL is released while the image loading happens.
I don’t know if this has already changed or is being planned, but linking to another thread:
The amount of book keeping we have to do as a third party render add-on, could be better. Even a simple callback, like a pre-delete object handler would be nice i.e.: the user has requested an object be deleted, make a callback to the add-ons to notify them that the object is about to be deleted, so do whatever necessary clean up you need to do, before this happens.
Reading between the lines here I’m not sensing the high priority on Blender’s side to fix these issues as they don’t affect Cycles or Eevee (@brecht that’s not meant to be a criticism, totally understand). But it would help make other render engines improve.
So I suppose these changes would have to come from add-on developers in this thread. Going back to the original it sure would be nice if octane contributed back their changes to blender source.
Yes, I think the core devs have bigger and wider priorities right now than to improve third party engines integration.
I agree, companies like Otoy / Octane or RedShift could help improve this, they have the funding and I’m sure for Otoy would be better to avoid having to maintain their own fork, and for RedShift, if they dedicate part of their main devs team to solve some of the issues for everyone they will benefit a lot from that too.
While they are commercial engines, doing something that will benefit everyone will also benefit them, and in the end their leverage is not in the Blender integration itself (that will improve) but in the render engine itself, so could be a great contribution.
Other team that may be able to improve this is the Radeon Pro Render team, since AMD is actively involved with Blender, maybe their team can allocate a bit of time to improve some of these things.
LuxCore dev team has very little funding right now and it’s a ton of work to maintain and develop not just the engine but also the addon, however I’m sure that if something is feasible by them they will contribute it no doubt.
I don’t mention other smaller engines mainly because they have very very little funding up to my knowledge so I’m sure it’s hard for them to maintain the engine, it must be harder to improve these things.
All this are just ideas