The Rush render queue allows users to manage jobs. A 'job' is
usually just a range of frames that need to be rendered. To start a job
you need to run a Submit Script,
which contains: the instructions defining the job, the frame range to be
rendered, which machines are to be used for rendering, the 'priority' the
job should run at, the pathname of the
Render Script for rendering frames, etc.
There are several premade GUI submit interfaces
which can be customized, such as the Maya submit script.
Or you can write your own Submit Script
in the language of your choice;
Python,
Perl,
Csh,
Bash.
The one script can double as both the script the user invokes
to submit the job, and can also contain the logic that invokes
the render to run on each machine, setting proper environment variables
for the renders, checking the renders for error codes and messages,
handling retry logic, etc.
To render each frame, Rush runs the script on each machine to
invoke whatever UNIX commands are necessary to run the renderer or compositor.
A frame number is passed via an environment variable,
$RUSH_FRAME,
to tell the script which frame to work on. The script can invoke
any 'command line based' programs: renderers, compositors, custom C programs,
other perl/python/csh scripts, etc.
The purpose of the script during rendering is to ensure the correct
environment variables are configured to run the render, determine
what the render command line should look like, and then check
the renderer for errors, and return one of the
'rush exit codes' to indicate success or failure; 0=Done ok, 1=Failed, 2=Retry.
The script will be executed on all the machines the job requests to use,
invokes with the same pathname on each machine. So the script must be
accessible via a file server (such as an NFS path on Unix, or a UNC path
on Windows).
When the render script runs, various
environment variables
are passed from Rush which may be useful to intermediate or advanced script programmers.
The script is executed to start the job running.
As the job runs, frames are started on the various networked machines as needed,
eating through the frame list until there are no more frames to render.
After each frame renders, the system records how long the frame took to run,
which is normally viewed
within "irush", but can also be viewed from the command line,
with the render output for each frame saved in the Frame Logs
which can also be viewed from within irush by
double clicking on the frame.
The render queue uses Priority Values
to allow important jobs to take precedence over lower priority jobs. When priorities
of different jobs are equal, a 'round robin' approach is used to allow jobs to vie for cpus.
Priority flags allow a job to fight other lower priority jobs, instead
of passively waiting for idle cpus to become available ('k', the
Kill flag). A job can also have a priority such that no other jobs
can kill it ('a', the Almighty flag). Combined, these flags enable
a job to kill off other jobs without allowing other jobs to kill it. Sysadmins
can monitor audit logs for the use of such flags to prevent misuse, and there
are rush configuartion flag settings
that let the admin limit which users can use these flags.
Rush has several GUI programs; irush lets users monitor
and control jobs, rushtop which lets users monitor
processor and ram use in realtime, onrush gives users
a simple 'light switch' for disabling their workstations from rendering, and
rushadmin which lets administrators administer
various aspects of rush.
