|
|||||
Use $RUSH_PADFRAME, it is created for you automatically to do 4 digit padding.To do your own custom frame number padding, use this unix technique:
set padframe = `perl -e 'printf("%04d",$ENV{RUSH_FRAME});'`
To use different padding widths, just change the '4' (in '%04d') to a different number.
The most common problem is a render script that does not properly handle returning exit codes. Make sure your render script is correctly returning an appropriate render script exit code: 0=OK, 1=FAIL, 2=RETRY.Also, check the frame logs being generated by your render script. Frame logs contain the error messages for each rendered frame which should help you determine the problem. Make sure your submit script has LogDir pointing to a valid directory, which is where your frame logs will be found.
See Retrying Frames.
Use 'rush -lc' and check the Notes column for messages.If you know the remote cpus aren't just busy with other jobs, then list your cpus and check the 'NOTES' column to see if the system is giving you reasons why your cpus are being rejected.
The job might be in Pause, there are no more frames to render, all the available machines don't have as much ram as your job needs, etc. Here are some typical situations:
[erco@howland]% rush -lc CPUSPEC[HOST] STATE FRM PID JOBTID ELAPSED NOTES placid=3@100k Idle - - 1 00:04:37 Job state is 'Pause' tahoe=1@1 Idle - - 2 00:02:08 No more frames superior=1@1 Idle - - 3 00:02:08 Not enough ram waccubuc=1@1 Idle - - 4 00:02:08 This is a 'neverhost' ontario=1@1 Idle - - 5 00:02:08 Failed 'criteria' check |
Use the Criteria submit script command.This command allows you to build a list of platforms, operating systems, or other general criteria to limit which machines will run your renders.
You can see the different criteria names in the output of 'rush -lah'. It is up to your sysadmin to maintain the criteria names.
With clever scripting. See Batching Multiple Frames for how to render several frames at a time.Sometimes it pays to render several frames at a time rather than one at a time, to decrease the amount of time the renderer spends loading files.
If you have existing script filters which monitor the progress of renders to determine which frames are rendering, you can probably easily modify these scripts to work with rush to reflect changes in the frame list, using either frame notes (rush -notes) or frame state change operations (rush -que/rush -done).
For a job to bump another off a cpu, these things must be true:When a frame is bumped, the bumped frame will show a message in its frame list indicating the job that bumped it, e.g.:
- A job only bump other jobs of lower priority (i.e., not same priority)
- A job can't be bumped if almighty flag is set ('a').
- A job can't be bumped unless its entry in the -tasklist is either in the Avail or Run state.
You can use eval `submit` to automatically set it, or a simple alias to set it manually. However, cutting and pasting the 'setenv' command is not so hard.Some people like to use this alias to make it easy to set new jobid variables:
Then you can use it on the command line to set one or more jobids:
If you want to have the RUSH_JOBID variable set automatically in your shell whenever you invoke your submit script, then use 'eval':
...the shell automatically parses the 'setenv RUSH_JOBID' command rush prints on stdout when a job is successfully submitted. Error messages are not affected by 'eval', so you don't have to worry about losing error messages when using this technique.
Rush is not an acronym, it was named after film 'rushes', and the fact effects people are usually in a rush to get things done.
Usually the desire to do this stems from wanting to clean up left over temporary files generated by renders. In most cases, you can avoid left over files by putting temporary files in $RUSH_TMPDIR, which rush cleans automatically, even after bumps.
Bumps and dumps use SIGKILL to kill the render script and its children. This signal is NOT trappable. There's a reason:
Since bumps can happen just as readily as dumps, both use SIGKILL, untrappable, and always effective (except in pathological cases where the process is hung).
So do not expect to be able to trap interrupts to detect bumps/dumps.
If you need a way to determine if you are re-rendering a frame that was previous killed mid-execution (i.e., bumped by a higher priority job), you can put some logic into your render script:
#!/bin/csh -f
..
if ( -e /somewhere/$RUSH_FRAME.busy ) then
echo We are picking up a frame that was killed.
echo Do pickup stuff here..
endif
# Create a 'busy' file for this frame
# If we are bumped, busy file is left behind
# so that the above logic can detect it.
#
touch /somewhere/$RUSH_FRAME.busy
echo Do rendering here..
rm -f /somewhere/$RUSH_FRAME.busy
|
Also, see DependOn to have a job wait for
frames in another job to get done, i.e., rather than wait
for the entire job to complete.
If you are trying to include 'handles' and 'slates' by using negative
numbers, don't.
...and on WinNT, if you don't have grep(1):
..and on WinNT, if you don't have grep(1):
But if the machine never reboots, the frame will stay in the Busy
state indefinitely, unless you take the following action.
Assuming you're *sure* the machine is down, and not just 'slow',
use the following command:
...where 'hosta' is the name of the machine that is down,
and 'hostb' is the name of the machine that's the server for the
job(s) with the hung frame(s).
Beware; if the remote machine is not really down, and is still
running the frame, doing the above will start the frame running
on another machine, and the two frames will overwrite each other.
For instance, to print all the hosts in the "+foo" hostgroup:
If the umask is incorrect, make sure your render script is sourcing
the $RUSH_DIR/etc/.render file, and make sure the sysadmin
configured this file correctly to include the umask approriate
render jobs. This way everyone will have the correct umask by default,
since all render scripts should be sourcing that file.
You can also explicitly specify a umask command in your render script.
Just include the command somewhere before your first render commands.
Can I chain separate jobs together, so that one waits for the other to get done?
Yes, see the submit script command WaitFor
to have a job wait for others to dump before starting.
Is it possible to use negative frame numbers in rush?
No. You are evil.
Is there a way to see just the cpus busy running my job?
Yes. In Unix:
rush -lc | grep Busy
rush -lf | grep Run
rush -lc | findstr Busy
rush -lf | findstr Run
Is there a way to see what jobs a machine is busy rendering?
In Unix:
rush -tasklist host | grep Busy
rush -tasklist host | findstr Busy
Is there a way to requeue a busy frame for a host that is down?
If a machine goes down while rendering a frame,
the frame stays in the Busy state until the machine is rebooted.
Once rush realizes the remote machine rebooted, it requeues
the frame.
% rush -down hosta hostb
How do I list all the machines in a hostgroup?
Just grep the output of 'rush -lah', or parse the contents
of the $RUSH_DIR/etc/hosts file.
rush -lah | grep +foo
...or to precisely parse them from the hosts file with awk
(which you should be able to cut and paste into a Unix tcsh shell):
awk 'BEGIN { s="+foo"; } \
{ if (match($0,"^#")) next; n = split($5,arr,","); \
for (i=1; i<=n; i++) { \
if(arr[i]==s) { print $1; break; } \
} \
}' < /usr/local/rush/etc/hosts
Why is rush creating files owned by 'ntrush'?
Under Windows, rush assumes all jobs submited on windows machines
to be owned by 'ntrush'.
If the job is told to render on unix machines, it will run renders
as the user 'ntrush'.
Why are there weird file permissions on my rendered images?
Under unix, the file permissions are determined by your umask.
Is there a way to start a job at a particular time?
Yes, you can now specify a time for the job to waitfor, either
in the submit script with the WaitFor
command, or from the command line with
'rush -waitfor', e.g:
waitfor +8h -- wait 8 hours from now
waitfor 7:30pm -- wait until 7:30pm
See WaitFor for more info.