|
|
|
AutoDump Command Cpus Criteria DependOn DoneCommand DoneMail Frames FrameFlags ImgCommand LogDir LogFlags MaxTime NeverCpus Nice Notes Priority Ram State Title WaitFor |
Dump job on completion Render script to execute Hosts (or hostgroups) to use for rendering Criteria for matching hosts Set inter-job frame dependencies Command to run when job done Send mail when job done Frame ranges to render Disables the automatic clearing of the framelist Command used to display images (in irush) Directory for log files Controls logfile behavior Set maximum time for renders Cpus to never use for rendering nice(2) value for running frames (Unix only) Job notes Default priority Ram job expects to use (max) Initial state for job Title for job Wait for other jobs to complete |
AutoDump (rush -autodump) |
Enables a job to automatically dump itself. Can be any of:
autodump off | # Don't autodump; job remains when frames are DONE |
autodump done | # Job dumps itself when all frames are DONE |
autodump donefail | # Job dumps itself if all frames are DONE or FAIL |
In the case of 'autodump done', the job will not dump if there were any FAIL frames. If you want it to dump anyway, then use 'autodump donefail'.
Command (rush -command) |
Sets the command (and optional arguments) that is executed on a per frame basis on all the remote machines.
Usually, this is always an absolute NFS path to the Render Script.
It can, however, be an absolute path to any executable or script, provided it returns rush exit codes (0,1,2), and knows how to access RUSH_FRAME to determine which frame it's working on.
command /job/MARINER/DIVE/rush/render-script 640 480
WinNT Note - Use UNC paths for the absolute path
to the render script. This prevents problems with inconsistently mapped
drive letters. A UNC example:
|
Cpus (rush -ac/-rc) |
This command tells rush which remote machines are to be used during rendering. It is an error not to specify any cpus.
When specifying a cpu, your are telling rush at least three things:
The number of cpus defaults to 1 if unspecified.
If unspecified, the priority value defaults to the Priority value for the job.
Priority is a value between 1 and 999, with 999 being highest priority, 1 being lowest. Priority values can be followed by optional flags 'k' and/or 'a'. See Priority Description for a full description of how the priority mechanism works.
cpus pabst | # 1 cpu on pabst, default priority |
cpus pabst=4 | # 4 cpus on pabst, default priority |
cpus pabst=4@900 | # 4 cpus at 900 priority |
cpus pabst=4@900,2@500 | # 4 cpus at 900 priority, 2 cpus at 500 |
cpus +any=10@1 | # use up to 10 cpus on 'any available machine' |
cpus +farm=50@1 | # use up to 50 machines on the 'farm' hostgroup |
Note that '+any' and '+farm' are hostgroup specifications which represent more than one hostname. "+any" is a special hostgroup that includes all machines on the network.
Hostgroups are configured by your sysadmin in the Hosts file.
Criteria (rush -criteria) |
Criteria contains a list of words separated by logical operators (&) and (|). The strings are configured by your systems administrator, and are to be values in the Criteria column of the 'rush -lah' reports.
[erco@howland] % rush -lah IP Hostname Ram Cpus Pri Criteria 192.168.10.3 rotwang 100 2 0 +any,linux,linux6.0,intel,+dante 192.168.10.2 how 256 2 0 +any,sgi,irix,irix6.2 192.168.10.1 nt 256 1 0 +any,winnt,+dante |
When you specify hosts to render, any Criteria you specify will
limit which machines your renders will run on; if the criteria you specify
don't match a particular host, even if the host is specifically requested
by a Cpus command, frames will be turned away from
rendering on that machine.
For instance, if your job depends on using only Linux machines or sgis
running IRIX 6.2, you might submit your job with a criteria line that reads:
The above presumes your sysadmin uses 'linux' and 'irix6.2' as qualifiers
in the host list. If you need new criteria strings configured, ask your
sysadmin to add them to the rush system's hosts
file.
Only one Criteria command should appear in a submit script; multiple instances of the command are not commutative.
Here are some more examples:
|
DependOn (rush -dependon) |
Make the current job's frames depend on the completion of frames in other jobs first. This lets one set up a dependency tree of jobs.
When a job is submitted with 'dependon', all the frames in the job enter the 'Hold' state. (Frames in the 'Hold' state will not be rendered until switched back to the 'Que' state.) As a particular frame in the jobs that are depended on enters the 'Done' state (i.e., finishes rendering successfully), it is then that the corresponding frame in the current job will switch from Hold to Que, allowing the frame to begin rendering, as resources become available.
You can have a job depend on several others, the only stipulations being that the dependon jobs:
Frames will *not* switch from Hold -> Que until *all* the jobs depended on have their corresponding frames in the Done state. Otherwise those frames will remain in the Hold state.
See Chaining Jobs for scripting techniques to do this. Example usage:
DoneCommand (rush -donecommand) |
When a job dumps, after the 'framelist' and 'jobinfo' files are saved in the LogDir, the optional DoneCommand script can be executed. This command can be used to compress logfiles, or parse them for errors, or yield customized web page reports, etc.
This command is executed only once, on the job server.
If the done command is set to '-', or if donecommand is not specified at all, it will be disabled.
For the DoneCommand to be executed, the job must dump. For automatic invocation, you will need to have the AutoDump command enabled, for the job to dump when all the frames are done. If AutoDump is disabled, the only way the DoneCommand will execute is if someone manually invokes 'rush -dump'.
DoneCommand scripts are passed the jobid in the RUSH_JOBID environment variable, so it's possible for the script to use rush(1) commands to query the job. Exit codes are currently ignored. The stdout and stderr output from the DoneCommand is written to a file called 'done.log' in the LogDir.
#!/bin/csh -f # EXAMPLE 'DoneCommand' SCRIPT set $wwwreport = /somewhere/MYPROJECT/html/`logname`/jobreport.html # CREATE A CUSTOMIZED WEBPAGE REPORT set logdir = `dirname $RUSH_LOGFILE` cat $logdir/framelist | \ my_report_generator > $wwwreport # MAIL THE REPORT TO SOMEONE Mail -s "$RUSH_JOBID Html Report" `logname` < $wwwreport |
The DoneCommand should avoid doing anything to the job that might make it continue running. Though possible, this would confuse someone manually trying to dump the job, only to find it requeuing itself.
donecommand - | # Disable done commands |
donecommand $cwd/cleanup | # Setup script to run before job dumps itself |
(rush -donemail) |
When a job completes, rush can be configured to send mail to a list of people using DoneMail. Only one DoneMail command should appear in a submit script; multiple instances of the command are not cumulative.
Arguments should all be valid email addresses. If more than one address
needs to be specified, separate with commas. There should be no spaces
in the list of addresses. Use '-' to disable sending completion mail (default).
Some possible settings for DoneMail:
donemail - | # Mail disabled |
donemail erco@3dsite.com | # Send mail to erco@3dsite.com |
donemail fred,jack | # Send mail to Fred and Jack |
Keep in mind that if LogDir is configured, framelist and jobinfo reports will be written into its directory, so sending mail might be redundant.
Frames (rush -af/-rf) |
Defines the range(s) of frames to be rendered by this job. There can
be several Frames commands in a submit script; they are cumulative.
frames 1-10 | # Frames 1 thru 10 |
frames 100-150,2 | # Frames 100 thru 150 on twos |
frames 500 507 615 | # Frames 500, 507 and 615 |
Frame States. You can set the initial
state for the frame on a per-frame basis. Possible frame state values are
Done|Fail|Hold|Queue:
frames 1-5=Done | # Frames 1 thru 5 in DONE state |
frames 6-10=Fail | # Frames 6 thru 10 in FAIL state |
frames 11-15=Hold | # Frames 11 thru 15 in HOLD state |
frames 16-20=Queue | # Frames 16 thru 20 in QUEUE state |
Notes In Frames. On a per-frame basis,
frames can contain notes, which show up in the last column of frame
lists. Notes can be initialized to a particular string as part of the
Frames command:
frames 1-10:Black | # Notes for frames 1 thru 10 is "Black" |
frames 11:Fade_up_on_sc17 | # Frame 11 has note "Fade_up_on_sc17" |
The above example creates a frame list that looks like:
[erco@howland]% rush -lf STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Que 0001 0 - 0 00/00,00:00:00 00:00:00 Black Que 0002 0 - 0 00/00,00:00:00 00:00:00 Black Que 0003 0 - 0 00/00,00:00:00 00:00:00 Black Que 0004 0 - 0 00/00,00:00:00 00:00:00 Black Que 0005 0 - 0 00/00,00:00:00 00:00:00 Black Que 0006 0 - 0 00/00,00:00:00 00:00:00 Black Que 0007 0 - 0 00/00,00:00:00 00:00:00 Black Que 0008 0 - 0 00/00,00:00:00 00:00:00 Black Que 0009 0 - 0 00/00,00:00:00 00:00:00 Black Que 0010 0 - 0 00/00,00:00:00 00:00:00 Black Que 0011 0 - 0 00/00,00:00:00 00:00:00 Fade_up_on_sc17 |
Caveats:
FrameFlags |
Disables the automatic clearing of the framelist 'NOTES' field whenever a frame starts rendering.
By default, the NOTES field is automatically cleared before each frame is rendered. This prevents stale error messages from being left behind when a frame is requeued, since the notes field is usually utilized by TDs to advertise error conditions via their render script.
This default behavior can be disabled via the submit script command:
frameflags keepnotesThis keeps the frame notes to whatever the user changes them to, regardless of frames being requeued, bumped, etc.
(rush -imgcommand) |
This command enables IRUSH to display the rendered image when you double click on one of the frames in the 'Frames' report.
Normally, this should be an actual command that brings up one of your job's images in the background. The only difference being that you use %d or %04d in place of the frame number, e.g.:
The image viewer command should background itself. If it doesn't, it will hang IRUSH until you exit the viewer. You can force the command to background itself in these platform specific ways:
Examples:
imgcommand imgworks /job/images/att.%04d.sgi | # Unix: Invokes imgworks(1) to view the image. # (imgworks backgrounds itself; no need for '&') |
imgcommand xv /job/images/att.%04d.sgi & | # Unix: Invokes 'xv' to view the image. # ('&' assures 'flip' runs in the background.) |
imgcommand start /b flip //ntserver/images/att.%04d.sgi | # WINNT: Invokes 'flip' to view images. # 'start /b' assures it runs in the background |
imgcommand - | # Disable image commands |
LogDir (rush -logdir) |
Sets the directory where logs containing the stdout/stderr of rendered frames are written to, one file per frame. The filenames are the four digit padded frame number for the rendered frame.
When a job is dumped, two other files appear in this directory:
The directory must exist relative to both the job
server and all machines participating in rendering, and the directory must
be read/writable by the user submitting the job.
logdir /jobs/myjob/logs | # Logs are dumped in this directory |
logdir - | # Disable log files |
WinNT Note - Use a UNC path for the absolute path to the logdir.
This prevents problems with inconsistently mapped drive letters. A UNC example:
|
LogFlags (rush -logflags) |
Sets optional flags that control how log files are created.
The default behavior is to overwrite frame logfiles, each time a frame renders.
KeepLast tells the system to always keep the previous logfile, if there is one. It does this by renaming the previous log to an ".old" file, before creating the new log for a running frame, similar to running the command:
mv logs/0055 logs/0055.old |
KeepAll is similar to KeepLast, with the additional behavior that all 'previous' logs are kept; before a framelog is overwritten, it is concatenated to the .old file, similar to running:
cat logs/0055 >> logs/0055.old |
Beware: if your logfiles are long, KeepAll will cause
significant use of disk space, since the logs will accumulate. A good
reason to use KeepLast instead.
logflags - | # Logs are overwritten (Default) |
logflags keepall | # Keep all logs; concatenate old logs in 0000.old |
logflags keeplast | # Like 'keepall', but only keeps last log (don't concatenate) |
MaxTime (rush -maxtime) |
(New in rush 102.30c and up)
Optional.
Sets a maximum time limit for rendering frames.
Useful for setting an upper limit for how long frames can take to render. If a frame is still busy rendering when the maxtime is exceeded, the frame is automatically requeued.
The time value is in HH:MM:SS.
NOTE: Time checks are at a resolution of approximately
30 seconds; don't expect time accuracy any more precise than that.
Accuracy is also affected if the jobserver host is under load.
maxtime 00:00:00 | # Disables maxtime (Default) |
maxtime 01:30:00 | # Frames requeue if elapsed time exceeds 1 hour 30 minutes |
NeverCpus (rush -an/-rn) |
Sets hostnames that should never be used for rendering. Overrides any
Cpus specifications, ensuring hosts are not used even if specified by name
in the Cpus list. More than one NeverCpus command
can appear in a Submit Script; multiple instances of the command are cumulative.
nevercpus tahoe rotwang | #Never use tahoe or rotwang for rendering |
Nice (rush -nice) |
Sets the default nice(2) value for the job. Higher values increase the 'niceness' to others, letting the rendering frames yield to other processes and interactive use. A value of '0' means the process runs as usual.
If unspecified, the default nice value is 10.
Note: It has been noted under RedHat 6.1 Linux that nice values greater than zero will negatively affect render times considerably. Values above 10 are not recommended.
nice 0 | # run with 'normal user' not-so-niceness |
nice 10 | # default niceness |
Notes (rush -notes) |
Each job has a free form 'notes' field, which can be used for various purposes, such as passing informational notes to other programs, or possibly to other users who will be taking over this job in a hand-off.
#!/bin/csh -f ### SUBMIT SCRIPT ### rush -submit << EOF priority 10 : notes Please don't dump this job until you have visually notes verified the matte transition at frames 205-219. notes Call me at home if there are problems! -fred EOF |
When submitted, these notes appear in the 'rush -ljf' report for the job:
[erco@howland]% rush -ljf : Elapsed: 00:47:58 Frames: 22 Cpus: rotwang=2@100k Cpus: how=3@100k Notes[0]: Please don't dump this job until you have visually Notes[1]: verified the matte transition at frames 205-219. Notes[2]: Call me at home if there are problems! -fred |
(rush -priority) |
The 'priority' value is used when cpus commands don't specify a priority value, i.e., 'cpus tahoe=4'. In such a case, the value set by 'priority' is used. Example:
priority 10 : : cpus erie=2@100 cpus tahoe=4 cpus +any=10
See Priority Description for a description
of how priority values work.
Ram (rush -ram) |
All jobs must have a Ram value that tells the system how much memory each process is expected to use. Values are in MB, and are compared against the values shown in the Ram column of List All Cpus reports ('rush -lah').
While job is running, the configured Ram value is compared against the
available ram on the remote processors. If the amount of ram your job wants
is more than the remote machine has available, then the frame will not
be started. This behavior prevents swapping the remote machines.
ram 128 | # Only run on machines that have at least 128MB of ram available |
State (rush -pause/-cont) |
You can set the initial state for the job on submission. The only current option is to submit the job in the Pause state, to prevent the job from starting up right away.
: state Pause # Submit job in paused state :After you submit the job, the job will be in the paused state:
[erco@howland]% rush -lj # List Jobs to see job's 'Pause' state STATUS JOBID TITLE OWNER %DONE BUSY NOTES ------ ----------- ------------ -------- ----- ---- ----------- Pause how-857 THX/LOGO erco %0 0 Job paused. |
For no good reason, arguments to 'state' are case sensitive;
you have to use 'Pause', not 'pause'.
Title (rush -title) |
Each job has a free form 'Title' which appears in the job reports, audit logs, etc. Some shops use this information for billing purposes.
title THX/LOGOHere's an example report showing where the title might show up:
[erco@howland]% rush -lj # List Jobs to see title STATUS JOBID TITLE OWNER %DONE BUSY NOTES ------ ----------- ------------ -------- ----- ---- ----------- Run how-857 THX/LOGO erco %0 0 00:00:05 |
Titles cannot contain spaces, and should be short (<15 characters)
to prevent reports from losing columnar formatting.
(rush -waitfor) |
Sets up the current job to wait for other jobs to dump.
When the current job has 'waitfor' configured, it won't begin
rendering frames until the jobids specified have dumped themselves.
In this way, jobs can be chained together so that one waits for the other.
See Chaining Jobs for
scripting techniques to do this. Example usage:
|
|
-ac -af -an -autodump -checkconf -checkhosts -command -cont -cp -criteria -dcatlog -deltaskfu -dexit -dexitnow -dependon -dlog -done -donecommand -donemail -down -dump -end -fail -fu -getoff -hold -imgcommand -jobnotes -lac -lacf -lah -lahf -laj -lajf -lc -lcf -lf -lff -lfi -lhg -lhc -lj -ljf -log -logdir -logflags -maxtime -nice -notes -offline -online -pause -ping -que -ram -rc -reserve -reorder -rf -rn -rotate -status -submit -tasklist -title -trs -try -tss -uping -waitfor |
Add Cpus, adds to job cpus to use for rendering Add Frames, adds new frames to a job Add Nevercpus, adds to job names of cpus to never use Automatically dump job on completion (done|donefail|-) Checks a rush.conf file for errors Checks a rush hosts file for errors Sets job's command to run each frame Continue a paused job Change Priority for job's existing cpus Sets criteria qualifiers for the job Views the remote daemon's log file Deletes a task from a job or from a cpu server Tells a daemon to exit immediately via TCP Tells a daemon to exit immediately via UDP Makes a job dependent on frames in other job(s) Enables certain daemon debug logging flags Changes frames into the Done state Sets command to run when job is done Sets who to email when job is done Tells rush a machine is down. Dumps a job, kills running frames Ends a job, lets running frames complete Changes frames into the Fail state Allows you to change other people's jobs Kills renders, offlines the cpus Changes frames into the Hold state Sets the command irush uses to display images Changes the notes for the job List All Cpus, lists status of all cpus on the net List All Cpus Full, full report List All Hosts, showing #cpus, ram, criteria, hostgroups List All Hosts Full, full report List All Jobs, all jobs on the network List All Jobs Full, full report List Cpus, all cpus requested by a job List Cpus Full, full report List Frames, shows the job's frame list List Frames Full, full report List Frame Info, a short report of render statistics List Host Groups (new in 102.30f) List Host Criteria (new in 102.30f) List Jobs. Lists jobs on local or remote host(s) List Jobs Full, full report Shows the log file for frame(s) in job(s) Changes the job's logging flags (keepall|keeplast|-) Sets maximum time for rendering frames Set job's unix nice(1) value Changes notes field for specified frame(s) Offlines the local or remote host from rendering Onlines the local or remote host for rendering Pauses the job Pings the local or remote rush daemon(s) using TCP Changes the default priority value Requeues a frame Changes the job's ram use value Removes a cpu from the current job Reserves cpus on local or remote machines Reorders the frames in the frame list Removes frames from the frame list Removes 'nevercpus' Rotates the local or remote daemon's log file Status of jobs and cpus on local or remote hosts Submits a job Shows the list of scheduled tasks on local or remote host Changes the title of the job Shows the Template Render Script Changes the Try count for frame(s) Shows the Template Submit Script Pings the local or remote rush daemon(s) using UDP Make a job Wait For other jobs |
rush -ac <cpuspec..> [jobid..] |
|
See the Cpus submit command for more info. To remove cpus, see 'rush -rc'.
rush -af <framerange..> [jobid..] |
|
See Frames submit command for more info.
rush -an <hostname|+group..> [jobid..] |
|
See Nevercpus submit command for more info.
rush -autodump <off|done|donefail> [jobid..] |
rush -checkconf <filename> |
#!/bin/csh -f ### ### Script to edit the rush.conf file, and rdist it out ### set TMPFILE=/usr/tmp/rush.conf.$$ cp /usr/local/rush/etc/rush.conf $TMPFILE vi $TMPFILE rush -checkconf $TMPFILE if ( $? ) then echo You lose, game over. set err=1 else foreach i ( tahoe superior erie ) rdist -c $TMPFILE ${i}:/usr/local/rush/etc/rush.conf end set err=0 endif rm -f $TMPFILE exit $err |
rush -checkhosts <filename> |
rush -command 'cmd [args..]' [jobid..] |
|
Note that a full path to the command is normally required. Spaces are used to separate arguments, but the entire command and arguments must be quoted.
See also the submit script docs on Command for more info.
rush -cont [jobid..] |
rush -cp [jobid..] <.JobTid [..]> <@pri> |
Priorities can be changed while cpus are busy. If a busy cpu's priority is changed, the busy task(s) inherit the new priority, which is used for all subsequent scheduling and priority battles.
Example:
rush -criteria 'criteria strings' [jobid..] |
See Criteria submit command for more info on how criteria is specified.
rush -dcatlog [host..] |
Caveat: for reasons strange and unusual, all daemon 'catlog' output has a '>' prepended to each line.
rush -deltaskfu [..] |
This command has two forms; in one case it takes two arguments
where a task (or cpu) is forcibly removed from the job's
cpu report, and in the second case a task (or cpu reservation)
is forcibly removed from the cpu server's tasklist (i.e. 'rush -tasklist'). Examples:
rush -deltaskfu vaio.55 117 | # Delete JOBTID #117 from job vaio.55 |
rush -deltaskfu vaio.55 183 tahoe | # Delete CPUTID #183 from the tasklist on host tahoe |
Caveat: This command is not intended for casual use. It is to be used by administrators in unusual situations where cpus are stuck due to communications problems between machines. Use this command carefully.
rush -dependon [-|depjobid[,..]] |
Read the examples carefully, to avoid confusing jobid(s) to affect with new jobids for the dependon command; remember to separate all 'dependon' jobids with commas instead of spaces.
Setting dependon to '-' will disable it, but you will have to un-Hold any frames manually.
All 'dependon' jobids must have the same hostname as the job being modified.
To differentiate between the new value for 'dependon', and the jobid(s) to be affected, the new value for 'dependon' must be a comma-separated list of jobids with no spaces. Examples:
|
See also, rush -waitfor.
rush -dexit [remotehost..] |
rush -dexitnow [remotehost..] |
rush -dlog <flags> [remotehost..] |
These are flags that can be used with rush -dlog, rushd -d, and the rush.conf file's LogFlags. These flags can be combined to accumulate logging verbosity. All flags can be enabled by specifying 'a'. |
a - all b - bump mechanism logging d - log duplicate/redundant receipt of packet drops e - events (time oriented, async) f - fork h - hostname lookups j - Log job submissions k - Log bumped/killed/usurped tasks l - Logical string evaluations o - connect()/open()/close()/bind()/socket() (low level) p - parse command line arguments, submit scripts m - memory calculations (RAM) during priority battles, etc n - network commands (udp/tcp) r - reboot management/transactions s - signals t - tcp u - udp w - 'waitfor' checks y - yp lookups C - class ToWords/FromWords F - File loading line-by-line debugging E - Errors not normally displayed (benign, but suspect) T - task/taskack transactions U - update (scheduling, priority mechanism, idle cpu management) R - Reaper msgs S - Server/Client context switches X - Random UDP message dropping -- TESTING ONLY!! ('a' does not affect this option, it must be specified) |
rush -done <framerange|framestate..> [jobid..] |
The frames to affect can either be specified as a frame range (ie. 1-100) or as a frame state, for which all frames matching that state will be changed. Examples:
|
rush -donecommand [jobid..] <command|-|> |
Setting the donecommand to '-' will disable it. Examples:
|
rush -donemail [email[,email..]] [jobid..] |
See DoneMail submit command for more info and examples.
rush -down downhost[,downhost..] jobhost |
This will 'free up' stuck frames that won't requeue because a remote machine is down. Normally frames will requeue when the remote has rebooted, but if it won't be coming up for a while, the user can manually tell the rush system the remote is down.
This also helps jobs that won't dump because some frames are 'stuck' on a host that is down.
Warning: This command should only be used in extreme situations where the remote machine(s) are either down, or it has been verified for certain the frames are not running on the remote.
rush -down huron,erie tahoe |
# tell all jobs on tahoe # that machines 'huron' and 'erie' are down |
rush -dump [jobid|user|user@host..] |
If no arguments are specified, the RUSH_JOBID variable is used to determine which job to dump.
If jobid(s) are specified, those jobs will be dumped.
If a user's name is specified, all jobs owned by that user on the local machine will be dumped (e.g., 'rush -dump fred').
In the case of user@host, all jobs owned by 'user'
submitted to 'host' will be dumped.
i.e., 'rush -dump fred@tahoe' will dump all
jobs owned by fred that were submitted to host tahoe.
rush -end [jobid..] |
rush -fail <framerange|framestate..> [jobid..] |
The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:
|
rush -fu [option] |
For instance, if you want to control another person's job, you might get an error, e.g.:
|
See also the RUSH_FU environment variable.
Note: This feature may be disabled by the system administrator via the DisableFu flag in the Rush Configuration File.
1 This acronym is rumored to have an alternate, pejorative expansion.
rush -getoff [remotehost] |
rush -hold <framerange|framestate..> [jobid..] |
When a frame is in the Hold state the frame will not be rendered, and a job will not autodump if there are any Hold frames in the frame list.
The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:
|
rush -imgcommand '<command [args..]>' [jobid..] |
This should be a complete command to view an image in your favorite image viewer, with %d or %04d in place of the frame number.
The image viewing command must background itself, or you must force it by using '&' (Unix), or 'start /b' (Windows). Examples:
% rush -imgcommand 'xv /job/images/foo.%04d.sgi &' | # (Unix) Invokes 'xv' to view images |
C:\> rush -imgcommand "start /b flip //nt/images/s1.%04d.png" | # (Windows) Invokes 'flip' in background to view images |
See the ImgCommand submit command for more info.
rush -jobnotes '<notes>' [jobid..] |
See Notes submit command for more info.
rush -lac [-s secs] [-c count] |
This report is useful for seeing which cpus are running whose jobs.
[-s secs] is an optional argument to specify the number of seconds to wait for responses from the remote hosts. Default is 5.
[-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1. Example report:
% rush -lac HOST OWNER JOBID TITLE FRM PRI PID ELAPSED tahoe - - - - - - Online tahoe - - - - - - Online superior erco vaio-50 FOREST_RENDER 0034 100 4026 00:10:01 superior erco vaio-50 FOREST_RENDER 0035 100 4027 00:10:02 superior tamu ibm4-208 AUTO_CAD 0101 300k 2031 02:43:31 superior tamu ibm4-208 AUTO_CAD 0102 300k 2035 02:43:31 erie erco vaio-50 FOREST_RENDER 0041 900k 14560 00:10:15 erie erco vaio-50 FOREST_RENDER 0042 900k 14561 00:10:15 michigan - - - - - - Offline michigan - - - - - - Offline michigan - - - - - - Offline michigan - - - - - - Offline *** NO RESPONSE FROM: *** ontario huron |
Note that machines that did not respond are listed in the summary at the bottom of the report; all lines of these summaries are prefixed by '***'.
rush -lacf |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
rush -lah [hostname..] |
% rush -lah IP Hostname Ram Cpus MinPri Criteria 10.100.100.1 tahoe 512 2 0 +any,irix 10.100.100.2 superior 512 2 0 +any,irix,+dante 10.100.100.3 erie 512 2 0 +any,irix,+dante 10.100.100.4 ontario 512 2 0 +any,irix,+dante 10.100.100.5 vaio 128 1 0 +any,linux,intel 10.100.100.6 rotwang 128 1 100 +any,linux,intel |
If hostname is specified, i.e., 'rush -lah hostname', the output comes from the cache of the daemon running on the specified host. This is useful in determining hostname caching problems.
rush -lahf [hostname..] |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
rush -laj [-s secs] [-c count] |
-s secs optionally sets the amount of time to wait for responses from remotes, default is 3.
-c count optionally specifies the number of times to continuously update the report, default is 1. -c 0 creates a continuously (indefinite) updating report.
% rush -laj STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED Run erie-167 100/340/master tamu %18 %0 8 01:15:28 Run erie-169 100/410/master tamu %75 %0 20 01:11:21 Run erie-170 100/390/master tamu %10 %34 4 01:10:44 Run erie-171 100/360/master tamu %10 %34 4 01:02:43 Run huron-454 W:090/440/all krement %100 %0 0 26:25:31 Run huron-457 W:090/425/master krement %100 %0 0 26:11:59 Run huron-476 W:090/425/master krement %100 %0 0 23:40:37 Run huron-519 Q:090/460/master krement %5 %0 8 00:21:27 Pause tahoe-76 230/250/layout_lock erco %2 %0 0 01:52:08 Run allek-790 qr2 tje %80 %1 4 06:58:20 Run allek-794 180/260/DAY tje %80 %1 2 06:38:48 Run allek-796 180/250/DAY tje %0 %0 0 06:34:21 Run allek-801 qr2 tje %0 %0 0 05:27:40 Run vaio-663 230/tdcheck/roomBC woz %80 %20 0 00:44:53 |
rush -lajf |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
rush -lc [jobid..] |
% rush -lc # List Cpus assigned to the current job CPUSPEC[HOST] STATE FRM PID JOBTID ELAPSED NOTES huron=4@800k JobPass - - 178 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 179 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 180 00:00:00 This is a 'neverhost' huron=4@800k JobPass - - 181 00:00:00 This is a 'neverhost' +any=10@100[howland] Busy 0080 14733 165 00:00:01 +any=10@100[howland] Busy 0078 14731 166 00:00:01 +any=10@100[howland] Busy 0079 14732 167 00:00:01 +any=10@100[toronto] Idle/Nak - - 171 00:00:00 - +any=10@100[toronto] Idle/Nak - - 172 00:00:00 - +any=10@100[toronto] Idle/Nak - - 173 00:00:00 - +any=10@100[vaio] Busy 0074 4107 174 00:00:03 +any=10@100[vaio] Busy 0071 4100 175 00:00:03 +any=10@100[vaio] Busy 0072 4101 176 00:00:03 +any=10@100[vaio] Busy 0073 4106 177 00:00:03 |
|
rush -lcf [jobid..] |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
rush -lf [jobid..] |
% rush -lf # List Frames for the current job STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Run 5000 1 vaio 18092 02/26,02:15:40 00:00:21 Run 5001 1 vaio 18093 02/26,02:15:40 00:00:21 Run 5002 1 vaio 18100 02/26,02:15:50 00:00:11 Que 5003 0 - 0 00/00,00:00:00 00:00:00 Que 5004 0 - 0 00/00,00:00:00 00:00:00 Que 5005 0 - 0 00/00,00:00:00 00:00:00 Que 5006 0 - 0 00/00,00:00:00 00:00:00 Que 5007 0 - 0 00/00,00:00:00 00:00:00 Que 5008 0 - 0 00/00,00:00:00 00:00:00 |
The columns contain the following information:
|
rush -lff [jobid..] |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
rush -lfi [jobid..] |
% rush -lfi vaio.139 # List Frame Info for job 'vaio.139' Jobid State Total Perc Average Average ETA ------------ ----- ----- ---- ---------- ------------------------ vaio-139 Que 7 %69 - - vaio-139 Run 1 %10 - - vaio-139 Done 2 %20 00:32:16 Thu Sep 28 00:41:43 2000 vaio-139 Fail 0 %0 - - vaio-139 Hold 0 %0 - - |
Some description of the average columns:
oldest_start = "The oldest start time for all Done frames" recent_end = "Most recent end time for all Done frames" time_spent = ( recent_end - oldest_start ) time_per_frame = time_spent / total_done_frames time_to_go = time_per_frame * ( total_que + total_hold ) |
Warning: the ETA is meant for ballpark estimates only, and is not meant to be taken literally.
rush -lhg [+hostgroup..] |
List Host Groups.
Shows all [specific] hostgroups
your sysadmin has configured in rush.
If no arguments are specified, all hostgroups are listed. Otherwise, arguments can be the names of hostgroups to limit the report to display.
Both hostgroups and hostnames are sorted alphabetically.
% rush -lhg # Lists all hostgroups when no arguments specified +any: erie ontario rotwang superior tahoe vaio +dante: erie ontario superior +intel: rotwang tahoe vaio +irix: erie ontario superior +linux: rotwang tahoe vaio % rush -lhg +dante # Lists only the '+dante' hostgroup +dante: erie ontario superior |
rush -lhc [criteria..] |
List Host Criteria.
Shows all [specific] host criteria your sysadmin has configured in rush.
If no arguments are specified, all host criteria are listed. Otherwise, arguments can be the criteria names to limit the report to display.
Both criteria and host names are sorted alphabetically.
% rush -lhc # Lists all criteria when no arguments specified intel: vaio rotwang irix: erie ontario superior linux: rotwang tahoe vaio % rush -lhc linux # Lists only hosts matching 'linux' criteria linux: rotwang tahoe vaio |
rush -lj [remotehost..] |
[you@erie] % rush -lj # List Jobs on local workstation STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED Run erie-167 100/340/master tamu %18 %0 8 01:15:28 Run erie-169 100/410/master tamu %75 %0 20 01:11:21 Run erie-170 100/390/master tamu %10 %34 4 01:10:44 Run erie-171 100/360/master tamu %10 %34 4 01:02:43 [you@erie] % rush -lj tahoe # List Jobs on remote host tahoe STATUS JOBID TITLE OWNER %DONE %FAIL BUSY ELAPSED Run tahoe-5 100/540/master izzie %25 %0 4 00:30:01 Run tahoe-6 100/540/comp izzie %25 %0 4 00:30:02 Run tahoe-10 100/640/master izzie %10 %0 4 00:22:48 Run tahoe-11 100/640/comp izzie %12 %0 4 00:22:49 |
The columns contain the following information:
|
rush -ljf [jobid..] |
This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.
|
rush -log <framerange..> [jobid..] |
|
rush -logdir pathname [jobid..] |
This does not affect commands that are already running or have run. i.e., old logs are not moved to the new location, or renamed on the fly.
When changing the logdir with -logdir, no checks are made to see if 'path' is a valid directory.
See LogDir for more information.
rush -logflags <-|keeplast|keepall> [jobid..] |
|
See LogFlags for more information.
rush -maxtime <hh:mm:ss> [jobid..] |
(New in rush 102.30c and up)
Sets maximum time for rendering frames.
The system will automatically requeue frames that exceed
the specified maxtime. See the MaxTime
submit command for more info.
rush -nice <niceval> [jobid..] |
rush -notes <framerange>:'notes..' [jobid..] |
|
rush -offline [remotehost|+group..] |
Lets frames that are already rendering complete before taking the processors offline. Once offline, no new frames will be started.
'rush -offline' is similar to 'rush -getoff', differing only in that -offline allows any currently rendering frames to complete before taking the processors offline. 'rush -getoff' will bump any running renders, freeing up the processors right away. The bumped frames will be requeued.
Examples of using 'rush -offline':
|
To return processors to the online state, use rush -online.
rush -online [remotehost|+group..] |
To take hosts offline, use either rush -offline or rush -getoff.
rush -pause [jobid..] |
To pause a job and kill the running frames, use:
To continue a job from pause, use rush -cont.
rush -ping [remotehost|+group..] |
rush -priority [jobid..] |
Example-- If a job submitted the following 'cpus' specifications:
priority 10 : : cpus erie=2@100 cpus tahoe=4 cpus +any=10
..this would create a job with 2 cpus on erie at a priority of 100, and all other cpus at a priority of 10.
Changing this job later with 'rush -priority 20' would cause all cpus (except erie, which specified a priority) to change their priorities to 20.
See the Priority submit command for setting the
initial default priority.
See Priority Description
for a description of how priority values work.
rush -que <framerange|framestate..> [jobid..] |
The frames to affect can either be specified as a frame range (ie. 1-100) or as a frame state, for which all frames matching that state will be changed. Examples:
|
rush -ram <ramval> [jobid..] |
rush -rc <cpuspec|tidspec|hostname> [jobid..] |
Cpus can be removed in one of several ways:
To remove a cpu via 'JOBTID' values (e.g., 'rush -rc .334') you must precede each value with a period. When you delete by JOBTID, you are deleting single cpus from the 'rush -lc' report. Note: this report includes the JOBTID values so you can see which values to delete. If the cpu you delete is part of a larger specification, (e.g., tahoe=4@12), then the cpu count for the spec will be modified and the cpu count in that spec will be decremented as well (e.g., tahoe=3@12)
If you remove a hostname (e.g., 'rush -rc tahoe') then all cpu specifications that have that host name (e.g., tahoe=3@100) will be removed. Also, any hostgroups that expand to include that host will have that host removed from the expansion (e.g., +any=3@100, which includes tahoe).
If you remove a cpu specification (e.g., 'rush -rc +any=3@100'), it must match character-for-character the entry shown in the 'rush -lc' report for the job:
|
rush -reserve <cpuspec> [ramval] |
'cpuspec' indicates which processors are reserved, and at what priority. It is an error not to include a priority value in the cpuspec.
A reservation job with a priority of '500' will prevent jobs with priorities less than 500, but yields to jobs higher than 500k. Including the 'k' flag in your reservation will also bump any renderings that are running on your machine that are lower than 500 priority. See Priority Description for more info on how rush priorities work.
As an example of reserving cpus on your workstation, let's assume you sit at a dual proc workstation called 'tahoe':
% rush -reserve tahoe=2@500k # Reserve both cpus at 500k priority |
tahoe=1@510k # Request one cpu on tahoe |
tahoe=2@510k # Request both cpus on tahoe |
The optional 'ram value' allows one to reserve an amount of ram per processor. This is useful on multiproc machines where you want to prevent large jobs from rendering on the few processors left unreserved. Increasing the ram value you reserve will decrease the size of jobs rush will allow to run on the unused processors.
If 'cpuspec' doesn't include the number of processors, one processor is assumed. If 'ramval' is unspecified, a value of '1' is assumed.
|
rush -reorder <framerange..> [jobid..] |
Changing the order of the framelist affects the order frames are rendered in, since frames are issued from the top of the list, down.
Example-- If you have a job with 10 frames in the frame list that are in normal 1 thru 10 order, you can use 'rush -reorder' to get different orderings, such as these:
|
rush -rf <framerange..> [jobid..] |
rush -rn <hostname|+group..> [jobid..] |
rush -rotate [remotehost|+group..] |
Logs can be automatically rotated with the LogRotateHour command in the rush.conf file.
Only root and the rush adminuser can use this command.
rush -status [-s secs] [-c count] [remhost|+hostgroup..] |
Commonly used as 'rush -status +any'.
Output is intended for other programs to parse and regenerate.
-s secs optionally sets the amount of time to wait for responses from remotes, default is 3.
-c count optionally specifies the number of times to continuously update the report, default is 1. -c 0 creates a continuously (indefinite) updating report.
The output contains several records of information for each host, one record per line. Host records start with an 'h' record, and terminate with a line of '---'.
4 different types of data records are possible. Data record types are defined by the first character in each record line, and can be one of:
h - hostname header. Leads off records for the specified host. d - daemon information. Info about the running daemon. j - job information. One line per job. p - processor status. One line per processor.Each record has its own fields:
h <hostname> d <sequence-id> <daemon> <version> <PID=pid> <online state> <jobs> <busy procs> <total procs> j <owner> <jobid> <job title> <job state> <elapsed> <percent done> <percent fail> <# frms busy> p <owner> <jobid> <job title> <frame> <priority> <pid> <elapsed>
rush -submit [remotehost] |
rush -tasklist [remotehost..] |
Tasks are sorted in the order they will be executed; the topmost task is the next one to be rendered.
Tasks that are busy are sorted to the bottom. Tasks that were denied by either the job, or by cpu criteria, are also sorted to the bottom.
Tasks are sorted into 'groups' of the same priority, and will round robin.
For a 'full' version of a tasklist report, use 'rush -tasklistfull'.
rush -title <text> [jobid..] |
rush -trs |
Caveat: the template render script can be modified and customized by the sysadmin via $RUSH_DIR/etc/templates.
rush -try <count> <framerange..> [jobid..] |
A 'try count' is maintained for each frame in a job's frame list. This value is shown in the 'Try' column of 'rush -lf' reports, and is passed to render scripts via the $RUSH_TRY environment variable.
% rush -lf # List Frames to see try counts STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Done 0001 4 superior 4144 10/04,03:29:10 00:00:18 Done 0002 4 tahoe 4160 10/04,03:29:30 00:00:16 Done 0003 4 placid 4163 10/04,03:29:47 00:00:16 Done 0004 6 huron 4168 10/04,03:30:04 00:00:16 Done 0005 4 finger 4171 10/04,03:30:21 00:00:16 % rush -try 0 1-5 # Reset Try count to zero 0001: tries was 4, now 0 0002: tries was 4, now 0 0003: tries was 4, now 0 0004: tries was 6, now 0 0005: tries was 4, now 0 % rush -lf # List Frames to see changes STAT FRAME TRY HOSTNAME PID START ELAPSED NOTES Done 0001 0 superior 4144 10/04,03:29:10 00:00:18 Done 0002 0 tahoe 4160 10/04,03:29:30 00:00:16 Done 0003 0 placid 4163 10/04,03:29:47 00:00:16 Done 0004 0 huron 4168 10/04,03:30:04 00:00:16 Done 0005 0 finger 4171 10/04,03:30:21 00:00:16 |
rush -tss |
Caveat: the template submit script can be modified and customized by the sysadmin via $RUSH_DIR/etc/templates.
rush -uping [-c count] [remotehost..] |
rush [jobid] -waitfor [-|waitjobid,[..,..]] |
Read the examples carefully to avoid confusing jobid(s) to affect with new jobids for the waitfor command. Remember to separate all 'waitfor' jobids with commas instead of spaces.
Setting 'waitfor' at '-' will disable the job from waiting for anything.
All 'waitfor' jobids must have the same hostname as the job being modified.
To differentiate between the new value of 'waitfor' and the jobid(s) to be affected, the new value of 'waitfor' must be a comma-separated list of jobids with no spaces. Examples:
rush -waitfor - | # Disable all waitfor's |
rush -waitfor tahoe-37,tahoe-38, | # Job waits for 'tahoe-37' and 'tahoe-38' to dump |
rush tahoe-40 -waitfor tahoe-37,tahoe-38, | # tahoe-40 now waits for 'tahoe-37' and 'tahoe-38' to dump |
rush tahoe-40 -waitfor tahoe-37, |
# tahoe-40 now waits for 'tahoe-37' to dump # Trailing comma on tahoe-37 is important! |
See also, rush -dependon.