Network Render Test with flamenco

Hi everyone,

I’m trying to perfrom a render test using flamenco with two computers one as a master and the other as a worker. Everything seems to work ok but I get this error on the worker:


I’m confused because the path is already created and I have acces with an explorer.

Does anybody have an idea how to solve it?

1 Like

Do both machines have Z: mapped to the same shared storage?

Yes, Z is our NAS drive.

I also noticed that beside if it has access or not to the path. It has a odd number (5) of back slashes.

This is wierd. I set the output path in the Blender Cloud Addon:

That’s fine: note how the single quotes are also escaped. It’s basically an escaped version of 'Z:\\'\'Z:\\\\\'. The double backslash is strange, though.

@Marc_C Could you try using forward slashes for all paths instead?

I just tried it and I got the same error. I’m going to try in two machines with Linux.

Hello again!,

So, we have succeed running flamenco using Linux in two PCs as you suggested in order to check everything works fine in our network. The problem is that we have all our workstations and in-house software with Windows. So we still need to find a solution with “workers” not being able to access the job storage path. We suspect the issue is in the “worker windows installer”, what are your thoughts about it? If you don’t have spare time to check this issue we would be happy to do so but we haven’t find the sorce code of the “worker installer”, can you share it or tell us where to look for it?

P.D. I’m the one who tweeted you last Monday. :wink:

Extending your comment, this is the error in the command shell of flamenco-worker with two computers on Windows:

2020-06-10 14:56:33,328  WARNING flamenco_worker.runner.TaskRunner Command 0 of task 5ee0d876e5b84e87e6b7f458 was not succesful, aborting task.
2020-06-10 14:56:33,329     INFO flamenco_worker.runner.TaskRunner Aggregated task timing info: {}
2020-06-10 14:56:33,331    ERROR flamenco_worker.worker.FlamencoWorker Task 5ee0d876e5b84e87e6b7f458 failed
2020-06-10 14:56:33,334     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager, queue size is 2
2020-06-10 14:56:33,354     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager, queue size is 1
2020-06-10 14:56:36,435     INFO flamenco_worker.worker.FlamencoWorker.fetch_task Received task: 5ee0d876e5b84e87e6b7f459
2020-06-10 14:56:36,435     INFO flamenco_worker.worker.FlamencoWorker Task changed status to active, pushing to manager
2020-06-10 14:56:36,439     INFO flamenco_worker.worker.FlamencoWorker Updating task 5ee0d876e5b84e87e6b7f459 with status 'active' and activity Activity(activity='', current_command_idx=0, task_progress_percentage=0, command_progress_percentage=0, metrics={'timing': {}})
2020-06-10 14:56:36,450  WARNING flamenco_worker.commands.blender_render.(task_id=5ee0d876e5b84e87e6b7f459, command_idx=0) Error in settings: blender_cmd 'C:/Program' does not exist
2020-06-10 14:56:36,451     INFO flamenco_worker.worker.FlamencoWorker Task changed status to failed, pushing to manager
2020-06-10 14:56:36,452     INFO flamenco_worker.worker.FlamencoWorker Updating task 5ee0d876e5b84e87e6b7f459 with status 'failed' and activity Activity(activity="blender_render.(task_id=5ee0d876e5b84e87e6b7f459, command_idx=0): Invalid settings: blender_cmd 'C:/Program' does not exist", current_command_idx=0, task_progress_percentage=0, command_progress_percentage=0, metrics={'timing': {}})
2020-06-10 14:56:36,461  WARNING flamenco_worker.runner.TaskRunner Command 0 of task 5ee0d876e5b84e87e6b7f459 was not succesful, aborting task.
2020-06-10 14:56:36,463     INFO flamenco_worker.runner.TaskRunner Aggregated task timing info: {}
2020-06-10 14:56:36,465    ERROR flamenco_worker.worker.FlamencoWorker Task 5ee0d876e5b84e87e6b7f459 failed
2020-06-10 14:56:36,466     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager, queue size is 2
2020-06-10 14:56:36,482     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager, queue size is 1
2020-06-10 14:56:51,470     INFO flamenco_worker.worker.FlamencoWorker Updating task 5ee0d876e5b84e87e6b7f459 with status '' and activity Activity(activity='', current_command_idx=0, task_progress_percentage=0, command_progress_percentage=0, metrics={'timing': {}})
2020-06-10 14:56:51,481     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager, queue size is 1

This error has occurred on Windows with the same configuration that on Linux has worked, obviously changing the value paths to the correct ones.

1 Like

When using spaces in a filename, you need to quote that filename. So configure the Blender path as "C:/Program Files/blablabla/blender.exe" --factory-startup --other-options, including the quotes.

What is this “worker windows installer”? On the Flamenco download page there is only the ZIP file that you just unzip to get the worker. There is no installer.

1 Like

You are right, there is no installer, what I meant is that everything in the zip file is compiled.

It may look that way, but most of the code is just Python sources inside one big zip file (so a zip within a zip).

I finally solved the problem changing the path of the blender.exe to “C:/PROGRA~1/BlenderFoundation/Blender_2.82/blender.exe” note there are no spaces as you said. With that everything worked OK.
Anyway is a little bit confusing that the suggested installation path of Blender on Windows is under “Program Files/Blender Foundation/ Blender XX/blender.exe” but Flamenco worker doesn’t support spaces in path.

:joy: :joy: :joy: sorry

That is confusing indeed, and would be nasty if this was intentional. Fortunately, the note in the documentation that states that paths with spaces should be quoted indicates that this is not :wink:

Flamenco has been made primarily for the Blender Animation Studio. As such, it has seen much less use on Windows in general, and hasn’t been used on Windows at all by its developers. Windows is such a different system than Linux or macOS, that I’m not surprised that Flamenco has some issues there. I’m confident that, given a properly detailed bug report on how to reproduce this with a minimal amount of effort, and enough development time, every issue can be fixed.

2 Likes

Hi, I just started setting up a small render farm here at work, it’s made of 4 PCs running windows.
I can get the renders running alright on some computers, so the setup is mostly fine. However, if I happen to have one computer that struggles when connecting to the NAS, it gives a failed task. As it should. But what I am concerned about is that if I manually fix the error on the worker (usually retyping my credentials to connect to the network), this particular worker will get ignored throughout the whole job. It connects fine to the manager, but no more tasks are being dispatched to this worker. The only solution I have found so far is to cancel the whole job and start over. This is something that can become tricky for big jobs.
Generally speaking I really enjoy the flamenco web interface, but what is lacking in my opinion is a way to manage/check status on the different workers.
Also, perhaps a way to manually set different tile sizes for each workers ? I have a i9+2080ti that works best at 256px tile size in FHD and 4 i7+1060 that work best at 64px tile size.
I’m not sure if this is the right place to post those bugs/suggestions, or if I should go the RCS or file a bug report

1 Like

This is intentional. When a Worker fails a particular task type on a particular job, it gets blacklisted. This is to prevent situations where a Worker has a bit too little memory to render a particular shot, crash as soon as the shot is loaded, mark the task as failed, and race through all the tasks until there are so many failures the entire job is cancelled.

The Worker will only be blacklisted for that particular task type. So, if rendering is to heavy for it, it can still help moving files around, running ffmpeg to convert image sequences to a video file, etc.

You can prevent this by configuring the pre-task check on the Worker. Add something like this to your flamenco-worker.cfg:

[pre_task_check]
write.0 = /render/flamenco/output
read.0 = /render/flamenco/input

This makes the Worker check whether it can write to /render/flamenco/output and read from /render/flamenco/input, before asking the Manager for a task. Both are optional, so use whatever you want. You can also add more directories there, with write.1, write.2, etc.

In the Workers overview of Flamenco Manager, click on the little triangle to open up the worker details. There you’ll find the blacklisted jobs & task types for that worker. Click on the red cross to remove a blacklist entry.