I’m not sure if this is supported but needs some extra steps or if it would need changes in Blender itself to work.
Right now if I enable “Use Freestyle” and render with a custom RenderEngine it seems that it actually computes the results, but they are not composited into the final render.
If this requires a patch I would be willing to do the work myself as long as I recieve some direction.
The way Freestyle works is that it creates a scene with meshes and shader nodes to be rendered by the render engine. Maybe the simplest way to make this work for custom engines is to always use Eevee for that instead.