RUSH RENDER QUEUE - COMMAND LINE OPTIONS
(C) Copyright 1995,2000 Greg Ercolano. All rights reserved.
V 102.41a 05/27/04
Strikeout text indicates phased out or obsolete operators.


Rush Command Line


  rush -ac  
rush -ac <cpuspec..> [jobid..]

Add Cpus.

Adds cpus to the running job. Examples:

    Add Cpu Examples
    rush -ac tahoe@300 Add one cpu on tahoe at 300 priority
    rush -ac tahoe=2@300 +farm=10@100k Add two cpus on tahoe, 10 cpus for +farm hostgroup
    rush -ac +any=10@30 Add any 10 available cpus at priority 30

See the Cpus submit command for more info. To remove cpus, see 'rush -rc'.

  rush -af  
rush -af <framerange..> [jobid..]

Add Frames.

Adds frames to the current framelist. Adding frames that already exist yields an error. Example:

    Add Frames Examples
    rush -af 10-15 Add 10 thru 15 to the current job
    rush -af 20-25=Hold Add 20 thru 25 in Hold state
    rush -af 30-35:"To be done" Add 30 thru 35, Notes "To be done"
    rush -af 40-45=Done:"To be done" Add 40 thru 45 as Done with Notes

See Frames submit command for more info.

  rush -an  
rush -an <hostname|+hostgroup..> [jobid..]

Add Nevercpus.

Prevents the job from rendering on the specified hosts. Example:

    Add Never Cpus Examples
    rush -an tahoe +farm Add host tahoe and all hosts in hostgroup
    +farm as machines to never use.

See Nevercpus submit command for more info.

  rush -autodump  
rush -autodump <off|done|fail|donefail> [jobid..]

Automatic Dump.

Configures job to automatically dump when there are no more frames to render.

    Autodump Examples
    autodump off Don't autodump; job remains when frames are DONE (default)
    autodump done Job dumps itself when all frames are DONE
    autodump fail Job dumps itself if job finishes with any FAIL frames (New in 102.40)
    autodump donefail Job dumps itself if all frames are DONE or FAIL

See AutoDump submit command for more info.

  rush -checkconf  
rush -checkconf <filename>

Check for errors in rush.conf file..

(Administrative) Checks a rush.conf file for errors. Returns an exit code indicating success or failure; 0=ok, 1=error(s) found. Example:

#!/bin/csh -f

###
### Script to edit the rush.conf file, and rdist it out
###

set TMPFILE=/usr/tmp/rush.conf.$$
cp /usr/local/rush/etc/rush.conf $TMPFILE
vi $TMPFILE

rush -checkconf $TMPFILE
if ( $? ) then
    echo Config file has errors
    set err=1
else
    foreach i ( tahoe superior erie )
        echo --- $i
	rdist -c $TMPFILE ${i}:/usr/local/rush/etc/rush.conf
    end
    set err=0
endif
rm -f $TMPFILE
exit $err
	

  rush -checkhosts  
rush -checkhosts <filename>

Checks a hosts file for errors.

(Administrative) Returns an exit code indicating success or failure; 0=ok, 1=error(s) found.

  rush -command  
rush -command 'cmd [args..]' [jobid..]
rush -command [jobid..] < file

Set Render Command.

Changes the command that is executed every frame, normally set in your Submit Script by the Command. Example:

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

    Command Examples
    rush -command '/bin/sleep 30' Set command to sleep 30 seconds per frame
    rush -command < file Set command to contents of 'file' on stdin
    rush -command '/job/run/myrender' tahoe.21 tahoe.22 Change command for two jobs

Note that a full path to the command is normally required. Spaces are used to separate arguments, but the entire command and arguments must be quoted.

See also the submit script docs on Command for more info.

The command is usually a simple Render Script. Here are some example 3rd Party Render Scripts.

  rush -cont  
rush -cont [jobid..]

Continue a Paused Job.

Continues a job that was paused with rush -pause.

(New in rush 102.40h and up)
No longer cancels 'Waitfor'; it was decided the old behavior (102.40g and back) of having 'rush -cont' cancel 'WaitFor' was a bug, not a feature. To properly cancel WaitFor (to get the job running) is to use 'rush -waitfor -'.

  rush -cp  
rush -cp [jobid..] <.JobTid [..]> <@pri>
rush -cp [jobid..] <cpuspec [..]> <=#cpus>

Change Priority.

Changes priority for the cpus specified by 'JobTid' to the new priority '<@pri>'. JobTid values (Job Task IDs) can be found in the 'rush -lc' report.

Priorities can be changed while cpus are busy. If a busy cpu's priority is changed, the busy task(s) inherit the new priority, which is used for all subsequent scheduling and priority battles.

Example:


% rush -lc
CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED 
+any=10@100[vaio]    Busy        0007 1030    131     100   00:01:45 
+any=10@100[vaio]    Busy        0008 1031    132     100   00:01:45 
+any=10@100[tahoe]   Busy        0004 19829   133     100   00:00:05 
+any=10@100[tahoe]   Busy        0005 19831   134     100   00:00:05 

% rush -cp .131 .132 @350k
vaio.21 +any=10@100: priority was 100, now 350k
vaio.21 +any=10@100: priority was 100, now 350k

% rush -lc
CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED
+any=10@100[vaio]    Busy        0007 1030    131     350k  00:02:03 
+any=10@100[vaio]    Busy        0008 1031    132     350k  00:02:03 
+any=10@100[tahoe]   Busy        0004 19829   133     100   00:00:28 
+any=10@100[tahoe]   Busy        0005 19831   134     100   00:00:28 
        

Caveats:

  • Currently the CPUSPEC column does not reflect the change, though the effective priority (shown in the PRI column) does.

  • GUIs (like IRUSH) make best use of this option, since they hide the mundane task of visually indexing JOBTIDs from the -lc report, and inserting dot prefixes on the rush command line.

'rush -cp' can be used to change the number of cpus in a cpuspec. For instance, 'rush -cp +any=50@100 =40' will change '+any=50@100' to '+any=40@100' without killing any running frames.

Example:


% setenv RUSH_JOBID vaio.36
% rush -lc
CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
+any=10@500[tahoe]     Busy        0012 11443   343     500   00:00:10 
+any=10@500[tahoe]     Busy        0013 11445   344     500   00:00:10 
+any=10@500[nt-1]      Busy        0008 293     345     500   00:00:11 
+any=10@500[nt-1]      Busy        0009 246     346     500   00:00:11 
+any=10@500[vaio]      Busy        0015 11489   347     500   00:00:07 
+any=10@500[vaio]      Busy        0014 11484   348     500   00:00:09 
+any=10@500[imac]      Busy        0011 22530   351     500   00:00:11 
+any=10@500[imac]      Busy        0010 22528   352     500   00:00:11 

% rush -cp +any=10@500 =3@500
vaio.36 +any=3@500: priority was +any=10@500, now +any=3@500

	 NOW SET TO 3 CPUS, BUT RENDERS STILL RUNNING, NOTHING IS KILLED..

% rush -lc
CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
+any=3@500[tahoe]      Busy        0021 11453   343     500   00:00:41 
+any=3@500[tahoe]      Busy        0020 11451   344     500   00:00:41 
+any=3@500[nt-1]       Busy        0016 280     345     500   00:00:43 
+any=3@500[nt-1]       Busy        0017 71      346     500   00:00:43 
+any=3@500[vaio]       Busy        0022 11499   347     500   00:00:39 
+any=3@500[vaio]       Busy        0023 11501   348     500   00:00:39 
+any=3@500[imac]       Busy        0018 22534   351     500   00:00:42 
+any=3@500[imac]       Busy        0019 22536   352     500   00:00:42 

	 TIME PASSES, RENDERS START FINISHING, 3 CPU LIMIT TAKES EFFECT
% rush -lc
CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
+any=3@500[tahoe]      Busy        0026 11459   343     500   00:00:04 
+any=3@500[tahoe]      JobPass     -    -       344     500   00:01:01 Cpu max (3/3 busy)
+any=3@500[nt-1]       Idle        -    -       345     500   00:01:01 Cpu max (4/3 busy)
+any=3@500[nt-1]       Idle        -    -       346     500   00:01:01 Cpu max (4/3 busy)
+any=3@500[vaio]       JobPass     -    -       347     500   00:01:01 Cpu max (3/3 busy)
+any=3@500[vaio]       JobPass     -    -       348     500   00:01:01 Cpu max (3/3 busy)
+any=3@500[imac]       Busy        0024 22540   351     500   00:00:07 
+any=3@500[imac]       Busy        0025 22542   352     500   00:00:05 

        

  rush -criteria  
rush -criteria 'criteria strings' [jobid..]

Job Criteria.

Sets the criteria string for the job. Use '-' to specify no criteria.

See Criteria submit command for more info on how criteria is specified.

  rush -dcatlog  
rush -dcatlog [host..]

Daemon cat(1) Log File.

View the rushd.log daemon on the local [remote] host.

  • rush -dcatlog prints the current rushd.log (usually today's)
  • rush -dcatoldlog prints the last rotated log (usually yesterday's).

Caveat: for reasons strange and unusual, all daemon 'catlog' output has a '>' prepended to each line.

  rush -deltaskfu  
rush -deltaskfu [..]

Delete Task, Force Unconditionally

(Administrative) Users should not use this command. Use 'rush -rc' to remove cpus from your job.

Deletes individual 'tasks' from a job server or cpu server. Only root and the rush adminuser can use this command.

This command has two forms; in one case it takes two arguments where a task (or cpu) is forcibly removed from the job's cpu report, and in the second case a task (or cpu reservation) is forcibly removed from the cpu server's tasklist (i.e. 'rush -tasklist'). Examples:

    Delete Task Examples
    rush -deltaskfu vaio.55 117 Delete JOBTID #117 from job vaio.55
    rush -deltaskfu vaio.55 183 tahoe    Delete CPUTID #183 (from 'rush -tasklist') on host tahoe

JOBTID values can be found in -lc, -lcf and -tasklistfull reports.
CPUTID values can be found in -lcf and -tasklistfull reports.

Caveat: This command is not intended for casual use. It is to be used by administrators in unusual situations where cpus are stuck due to communications problems between machines. Use this command carefully.

  rush -dependon  
rush -dependon [-|depjobid[,..]]

Set Dependency On Other Jobs.

Sets new 'dependon' jobids for the current [specific] job. See the submit script docs on DependOn for more info.

Read the examples carefully, to avoid confusing jobid(s) to affect with new jobids for the dependon command; remember to separate all 'dependon' jobids with commas instead of spaces.

Caveats:

Setting 'dependon' to a new value disables all previous values (i.e., clears them first).

Setting dependon to '-' will disable it, but you will have to un-Hold any frames manually.

All 'dependon' jobids must have the same hostname as the job being modified.

See also, rush -waitfor.

To differentiate between the new value for 'dependon', and the jobid(s) to be affected, the new value for 'dependon' must be a comma-separated list of jobids with no spaces. Examples:

rush -dependon - # Disable all dependencies
 
rush -dependon tahoe.37,tahoe.38, # Job now depends on 'tahoe.37' and 'tahoe.38'.
# Trailing comma required
 
rush tahoe.40 -dependon tahoe.37,tahoe.38,     # tahoe.40 now depends on 'tahoe.37' and 'tahoe.38'
 
rush tahoe.40 -dependon tahoe.37, # tahoe.40 now depends on job 'tahoe.37'
# Trailing comma on tahoe.37 is required

  rush -dexit  
rush -dexit [remotehost..]

Daemon Exit.

(Administrative) Tells a daemon to exit immediately via TCP. Acknowledges success. Only root and the rush adminuser can use this command.

(New in rush 102.41 and up) Automatically saves job checkpoint before exiting.

  rush -dexitnow  
rush -dexitnow [remotehost..]

Daemon Exit Now.

(Administrative) Tells daemon to exit immediately, via UDP. There is no acknowledge. Only root and the rush adminuser can use this command. Use only in case of emergency.

(New in rush 102.41 and up) Automatically saves job checkpoint before exiting.

  rush -dlog  
rush -dlog <flags> [remotehost..]

Daemon Debug Logging

(Administrative) Enables certain daemon debug logging flags. Only root and the rush adminuser can use this command. Use 'rush -dlog -' to disable all debugging.

Logging Flags



These are flags that can be used with:

  • 'rush -dlog' to enable daemon debugging messages in the rushd.log, eg: 'rush -dlog utE'
  • 'rush -d' to enable client side debugging messages to the user's screen, eg: 'rush -d t -lfi'
  • In the rush.conf file's LogFlags

These flags can be combined to accumulate logging verbosity.

  • All flags can be enabled by specifying 'a', e.g. 'rush -dlog a'.
  • All flags can be disabled by specifying '-', e.g. 'rush -dlog -'


    - - A single dash is used to disables all debugging flags
    a - all
    b - bump mechanism logging
    c - select() info
    d - log duplicate/redundant receipt of packet drops
    e - events (time oriented, async, lic unpause)
    f - fork/wait
    h - hostname lookups
    j - Log job submissions and job state changes
    k - Log bumped/killed/usurped tasks
    l - Logical string evaluations
    m - memory calculations (RAM) during priority battles, etc
    n - network commands (udp/tcp)
    o - connect()/open()/close()/bind()/socket() (low level)
    p - parse command line arguments, submit scripts
    q - sequence id errors are printed (default:off)
    r - reboot management/transactions
    s - signals
    t - tcp
    u - udp
    w - 'waitfor' checks
    y - yp lookups
    C - class ToWords/FromWords (PROBABLY OBSOLETE)
    D - dumpsig debugging (Bluesky: 1.21)
    F - File loading line-by-line debugging
    E - Errors not normally displayed (benign, but suspect)
    J - Job Objects (win2k job objects kill mechanism msgs)
    K - Actual kill commands executed
    L - Locks on files
    M - Microsoft specific debugging
    P - server: Priority mechansim, client: log 'rush -push' mechanism
    R - Reaper msgs
    S - Server/Client context switches
    T - task/taskack transactions
    U - update (scheduling, priority mechanism, idle cpu management)
    W - 'Wakeup' pass states; pass->unpass state changes
    X - Random UDP message dropping -- TESTING ONLY!!
        ('a' does not affect this option, it must be specified explicitly)
    Z - reserved for on the fly debugging problems

    @ - PROTOCOL TEST FOR TASK_TIMEOUT. 
        Induces 60 second sleep before sending TASK_START.
        ('a' does not affect this option, it must be specified explicitly)
    

Examples:

rush -dlog ut tahoe      Enable UDP (u) and TCP (t) debugging for daemon on host tahoe
rush -dlog - tahoe Disable all debugging flags for the daemon on host tahoe

  rush -dlogstats  
rush -dlogstats [host|+hostgroup..]

(New in rush 102.40f and up)

Daemon Log Statistics

(Administrative) Useful for easily determining problems with machines on large networks.

Checks the local [remote] daemon log for particular kinds of errors, and logs the number of times each error appears in a table summary.

% rush -dlogstats +any
#                        ********* Problems ********** **Security* ************* Normal **************
# SID   Hostname         Alert  Fail Error  Exit  Fork  Bump Secur Start   Rot  Info JobSt JobEn JobKi
l 0     geneva               2     0     0     0     0     0     0     0     1     0     0     0     0
l 0     rotwang              0     0     0     1     0     0     1     1     1     2     0     0     0
l 0     ontario              0     0     0     0     0     0     0     1     1     2     1     0     0
    

This report shows that geneva has two 'ALERT' errors in its daemon log, which might indicate a problem worth investigating with 'rush -dcatlog geneva | grep ALERT'.

  rush -done  
rush -done <framerange|framestate..> [jobid..]

Done Frames.

Changes frame(s) into the Done state for current [specific] job. If frames are running, they are killed and enter the 'Done' state on completion.

The frames to affect can either be specified as a frame range (i.e. 1-100) or as a frame state, for which all frames matching that state will be changed. Examples:

% rush -done 1-100 # 'Done' frames 1 through 100
% rush -done fail # 'Done' all frames currently Fail
% rush -done fail que    # 'Done' all frames currently Fail or Que

  rush -donecommand  
rush -donecommand <command|-> [jobid..]
rush -donecommand [jobid..] < file

PHASE OUT.

Synonym for -jobdumpcommand, which is what future programs should use in its place.

  rush -donemail  
rush -donemail [email[,email..]] [jobid..]

Set Job Done Mail.

Changes who to send mail to when job is done. In version 102.40h+, this means when the job renders its last frame, or when the job is dumped. Which means if you requeue frames after a job is done, you will receive an email each time the last frame is rendered.

See DoneMail submit command for more info and examples.

See also DumpMail submit command if you want mail sent only once, when the job is actually dumped.

  rush -down  
rush -down downhost[,downhost..] jobhost

A Machine Is Down.

Tells rush jobs a machine is down.

If a machine hangs or goes down, rush has no way to know it.. for all rush knows, the remote machine is just under heavy load, and is being unresponsive.

Use '-down' to requeue frames in jobs that are stuck in the Run or Start state, because the remote machine is either hung or has been turned off unexpectedly.

Normally frames will requeue when the remote is rebooted, but in cases where the machine needs to stay off for a while, the user can use '-down' to manually tell rush the remote is down, and to requeue the hung frames elsewhere.

This also helps jobs that won't dump because some frames are 'stuck' on a host that is down.

Warning: This command should only be used in extreme situations where job(s) show the frames running on a remote machine that is known to be down, or not running the frame.

rush -down huron,erie tahoe     # tell all jobs on tahoe
# that machines 'huron' and 'erie' are down
rush -down huron,erie +any     # tell all hosts that 'huron' and 'erie' are down

  rush -dump  
rush -dump [jobid|user|user@host..]

Dump Job.

Dumps the job. Busy frames are killed right away.

If no arguments are specified, the RUSH_JOBID variable is used to determine which job to dump.

If jobid(s) are specified, those jobs will be dumped.

If a user's name is specified, all jobs owned by that user on the local machine will be dumped (e.g., 'rush -dump fred').

In the case of user@host, all jobs owned by 'user' served by 'host' will be dumped.
i.e., 'rush -dump fred@tahoe' will dump all jobs owned by fred that were submitted to host tahoe.

% rush -dump tahoe.37 tahoe.38    # Dumps jobs tahoe.37 and tahoe.38
% rush -dump fred # Dumps all jobs owned by 'fred' served by the local machine
% rush -dump fred@superior # Dumps all jobs owned by 'fred' served by host 'superior'

  rush -end  
rush -end [jobid..]

End Job.

Ends a job. Similar to rush -dump, but waits for running frames to finish (i.e., running frames are not killed).

  rush -fail  
rush -fail <framerange|framestate..> [jobid..]

Fail Frames.

Changes frame(s) into the Fail state for current [specific] job. If frames are running, they are killed and enter the 'Fail' state on completion.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

% rush -fail 1-100 # Fail frames 1 through 100
% rush -fail done # Fail all frames currently Done
% rush -fail done hold    # Fail all frames currently Done or Hold

  rush -frameflags  
rush -frameflags <-|keepnotes..> [jobid..]

Change Frame Flags.

    Frame Flags
    rush -frameflags keepnotes Disables automatic clearing of 'NOTES" field in framelist
    when a frame starts rendering.
    rush -frameflags - No flags. (Default)
    Default behavior is for NOTES field in framelist to be cleared
    whenever a frame starts rendering.

See also the submit script docs on FrameFlags for more info.

  rush -fu  
rush [args..] -fu

Force User1

The -fu flag augments other operations. It overrides warnings pertaining to lack of permission to do an operation because the user invoking it is not the owner of the job.

For instance, if you want to control another person's job, you might get an error, e.g.:

% rush -an vaio va-229 # Attempt to add 'vaio' as a nevercpu to someone's job
rush: va-229: you're not owner!   # Fails because you're not the job's owner
 
% rush -an vaio va.229 -fu         # Same command with -fu to force it...
Add nevercpu 'vaio' # ...now it works

See also the $RUSH_FU environment variable.

Note: This feature may be disabled by the system administrator via the DisableFu flag in the Rush Configuration File.

1 This acronym is rumored to have alternate, pejorative expansions.

  rush -getoff  
rush -getoff [-msg 'text'] [remotehost|+hostgroup..]

Get Off A Machine.

Similar to 'rush -offline', but kills and requeues any currently running frames before taking the localhost [remotehost] offline.

If no arguments are specified, assumes the localhost.

(New in rush 102.40 and up) An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg:

% rush -getoff -msg 'Client Demo' ontario
ontario: Offline (killing 1 procs)

% rush -lac
HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED REMARKS
---------- -------- -------------- ------------------------- ---- ----- ------- ---------- --------
ontario    -        -              -                         -    -     -          Offline Client Demo
:

Examples of 'rush -getoff':

% rush -getoff # Kills frames running on localhost (requeing them elsewhere)
# leaving the localhost offline (won't accept any more renders)
% rush -getoff tahoe erie      # Kills frames running on tahoe and erie (requeing the frames elsewhere)
# leaving both hosts offline (won't accept any more renders)
% rush -getoff -msg 'Bad Memory' huron # offline 'huron' with the remark 'Bad Memory' (New in 102.40)

  rush -hold  
rush -hold <framerange|framestate..> [jobid..]

Hold Frames.

Changes frame(s) into the Hold state for current [specific] job. If frames are running, they are killed and enter the 'Hold' state on completion.

When a frame is in the Hold state the frame will not be rendered, and a job will not autodump if there are any Hold frames in the frame list.

The frames to affect can either be specified as a frame range (i.e., 1-100) or as a frame state, for which all matching frames will be affected. Examples:

% rush -hold 1-100 # Hold frames 1 through 100
% rush -hold fail # Hold all frames currently Fail
% rush -hold fail done    # Hold all frames currently Fail or Done

  rush -imgcommand  
rush -imgcommand 'command [args..]' [jobid..]
rush -imgcommand [jobid..] < file

Set Image Display Command

Used by IRUSH to determine the command to execute whenever the user double clicks on a frame in the IRUSH 'Frames' report. Normally, the command invoked should display the rendered image for that frame.

This should be a complete command to view an image in your favorite image viewer, with %d or %04d in place of the frame number.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

The image viewing command must background itself, or you must force it by using '&' (Unix), or 'start /b' (Windows). Examples:

% rush -imgcommand 'xv /job/images/foo.%04d.sgi &' # (Unix) Invokes 'xv' to view images
C:\> rush -imgcommand "start /b flip //nt/images/s1.%04d.png" # (Windows) Invokes 'flip' in background to view images

See the ImgCommand submit command for more info.

  rush -jobcheckpoint  
rush -jobcheckpoint [host|+hostgroup..]

Save all jobs to the checkpoint file

This tells the local host (or remotehost|+hostgroup if specfied) to save a checkpoint now. Checkpoint will be loaded on reboot automatically, if the checkpoint.onboot is set to '1' in the rush.conf file (which is the default).

  rush -jobnotes  
rush -jobnotes 'notes..' [jobid..]
rush -jobnotes [jobid..] < notes.txt
rush -jobnotes '-' [jobid..]

Set Job Notes.

Appends free form strings of 'job notes' to the current [specific] job. Job notes are viewable in 'rush -ljf'.

If 'notes..' are not specified, the notes can be supplied on stdin.

If 'notes..' are specified on the command line, they must be quoted if they contain spaces.

Specifing 'rush -jobnotes -' will clear all previous notes.

See Notes submit command for more info.

rush -jobnotes < foo.txt # Set job notes to contents of foo.txt
# (Previous notes are completely cleared)
rush -jobnotes 'This is a test' # Set job notes to 'This is a test'

You can dynamically modify the job notes (e.g. insert, remove, reformat), using these steps:

    1) Use 'rush -ljf' to get the current notes into a local variable or file.
    2) Change the notes in your variable or file as needed.
    3) Use 'rush -jobnotes -' to clear the previous notes.
    4) Use 'rush -jobnotes < file' or pipe in the new notes.

  rush -jobstartcommand  
rush -jobstartcommand 'command [args..]' [jobid..]
rush -jobstartcommand [jobid..] < file
rush -jobstartcommand '-' [jobid..]

(New in rush 102.40 and up)
Set the command executed when job starts running.

See JobStartCommand submit command for more info.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

Setting the jobstartcommand to '-' will disable it. Examples:

rush -jobstartcommand 'perl //server/share/rush/start.pl' # Set job start command to run a perl script
rush -jobstartcommand < foo.txt # Set job start command to contents of foo.txt
rush -jobstartcommand - # disable the start command (Default)

  rush -jobdonecommand  
rush -jobdonecommand 'command [args..]' [jobid..]
rush -jobdonecommand [jobid..] < file
rush -jobdonecommand '-' [jobid..]

(New in rush 102.40 and up)
Set the command executed when job done running.

See JobDoneCommand submit command for more info.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

Setting the jobdonecommand to '-' will disable it. Examples:

rush -jobdonecommand 'perl //server/share/rush/done.pl' # Set job done command to run a perl script
rush -jobdonecommand < foo.txt # Set job done command to contents of foo.txt
rush -jobdonecommand - # disable the job done command (Default)

  rush -jobdumpcommand  
rush -jobdumpcommand '[-nolog] command [args..]' [jobid..]
rush -jobdumpcommand [jobid..] < file
rush -jobdumpcommand '-' [jobid..]

(New in rush 102.40 and up)
Set the command executed when job dumps.

See JobDumpCommand submit command for more info.

If 'cmd [args]' are not supplied, the command will be read from stdin. This is useful to avoid shell quoting issues. If multiple lines are specified, they will be joined into one single line.

    (New in rush 102.40f and up)
    Specifying the '-nolog' option in front of 'command' will disable the creation of the 'jobdumpcommand.log' file in the LogDir. This is useful to prevent 'file in use' errors on Windows if you want to remove the entire logdir as part of a cleanup operation.

Setting the jobdumpcommand to '-' will disable it. Examples:

rush -jobdumpcommand 'perl //server/share/rush/dump.pl' # Set job dump command to run a perl script
rush -jobdumpcommand '-nolog perl //server/share/rush/dump.pl' # Same as above, disables creation of 'jobdumpcommand.log'
rush -jobdumpcommand < foo.txt # Set job dump command to contents of foo.txt
rush -jobdumpcommand - # disable the dump command (Default)

  rush -lac  
rush -lac [-s secs] [-c count]

List All Cpus.

Lists the status of all rush cpus on the network. Report should come up quickly, even on large networks.

This report is useful for seeing which cpus are running whose jobs.

[-s secs] is an optional argument to specify the number of seconds to wait for responses from the remote hosts. Default is 5.

[-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1. Example report:

% rush -lac
HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED
tahoe      -        -              -                         -    -     -           Online
tahoe      -        -              -                         -    -     -           Online
superior   erco     vaio.50        FOREST_RENDER             0034 100   4026      00:10:01
superior   erco     vaio.50        FOREST_RENDER             0035 100   4027      00:10:02
superior   tamu     ibm4.208       AUTO_CAD                  0101 300k  2031      02:43:31
superior   tamu     ibm4.208       AUTO_CAD                  0102 300k  2035      02:43:31
erie       erco     vaio.50        FOREST_RENDER             0041 900k  14560     00:10:15
erie       erco     vaio.50        FOREST_RENDER             0042 900k  14561     00:10:15
michigan   -        -              -                         -    -     -          Offline
michigan   -        -              -                         -    -     -          Offline
michigan   -        -              -                         -    -     -          Offline
michigan   -        -              -                         -    -     -          Offline
*** NO RESPONSE FROM:
*** ontario huron

Note that machines that did not respond are listed in the summary at the bottom of the report; all lines of these summaries are prefixed by '***'.

  rush -lacf  
rush -lacf [-s secs] [-c count]

List All Cpus Full.

Lists the full info of all rush cpus on the network. This report may take a while to run.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

[-s secs] is an optional argument to specify the number of seconds to wait for responses from the remote hosts. Default is 5.

[-c count] is an optional argument to specify the number of times to automatically regenerate the report (for continuous updates). A value of 0 indicates infinite updating. Default is 1.

  rush -lah  
rush -lah [-t timeout] [hostname..|+hostgroup..]

List All Hosts.

Lists the Ram/Criteria configuration for all hosts in the render queue, which is loaded directly from the $RUSH_DIR/etc/hosts file. Example:

  % rush -lah
  IP               Hostname   Ram  Cpus MinPri Criteria
  10.100.100.1     tahoe      512  2    0      +any,irix
  10.100.100.2     superior   512  2    0      +any,irix,+dante
  10.100.100.3     erie       512  2    0      +any,irix,+dante
  10.100.100.4     ontario    512  2    0      +any,irix,+dante
  10.100.100.5     vaio       128  1    0      +any,linux,intel
  10.100.100.6     rotwang    128  1    100    +any,linux,intel    

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

If hostname is specified, i.e., 'rush -lah hostname', the output comes from the cache of the daemon running on the specified host. This is useful in determining hostname caching problems.

(New in rush 102.40h and up)
If a +hostgroup is specified, i.e. 'rush -lah +farm', the output comes from the cache of the daemons in that hostgroup.

  rush -lahf  
rush -lahf [-t timeout] [hostname..]

List All Hosts Full.

Full hosts report for all hosts on local [remote] host.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -laj  
rush -laj [-s secs] [-c count]

List All Jobs.

Queries all hosts on the network for what jobs are running.

-s secs optionally sets the amount of time to wait for responses from remotes, default is 3.

-c count optionally specifies the number of times to continuously update the report, default is 1. -c 0 creates a continuously updating report.

% rush -laj
STATUS JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED
Run    erie.167       100/340/master            tamu     %18   %0    8       01:15:28
Run    erie.169       100/410/master            tamu     %75   %0    20      01:11:21
Run    erie.170       100/390/master            tamu     %10   %34   4       01:10:44
Run    erie.171       100/360/master            tamu     %10   %34   4       01:02:43
Run    huron.454      W:090/440/all             krement  %100  %0    0       26:25:31
Run    huron.457      W:090/425/master          krement  %100  %0    0       26:11:59
Run    huron.476      W:090/425/master          krement  %100  %0    0       23:40:37
Run    huron.519      Q:090/460/master          krement  %5    %0    8       00:21:27
Pause  tahoe.76       230/250/layout_lock       erco     %2    %0    0       01:52:08
Run    allek.790      qr2                       tje      %80   %1    4       06:58:20
Run    allek.794      180/260/DAY               tje      %80   %1    2       06:38:48
Run    allek.796      180/250/DAY               tje      %0    %0    0       06:34:21
Run    allek.801      qr2                       tje      %0    %0    0       05:27:40
Run    vaio.663       230/tdcheck/roomBC        woz      %80   %20   0       00:44:53

  rush -lajf  
rush -lajf [-t timeout]

List All Jobs Full.

Full job report for all jobs on the network. This report may take a while to run.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -lc  
rush -lc [-t timeout] [jobid..]

List Cpus.

Lists cpus for current [specific] job. Lists a job's cpus, and what they're doing (or not doing).

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

Example:

% rush -lc           # List Cpus assigned to the current job
CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  ELAPSED  NOTES
huron=4@800k         JobPass     -    -       178     00:00:00 This is a 'neverhost'
huron=4@800k         JobPass     -    -       179     00:00:00 This is a 'neverhost'
huron=4@800k         JobPass     -    -       180     00:00:00 This is a 'neverhost'
huron=4@800k         JobPass     -    -       181     00:00:00 This is a 'neverhost'
+any=10@100[howland] Busy        0080 14733   165     00:00:01 
+any=10@100[howland] Busy        0078 14731   166     00:00:01 
+any=10@100[howland] Busy        0079 14732   167     00:00:01 
+any=10@100[toronto] Idle/Nak    -    -       171     00:00:00 -
+any=10@100[toronto] Idle/Nak    -    -       172     00:00:00 -
+any=10@100[toronto] Idle/Nak    -    -       173     00:00:00 -
+any=10@100[vaio]    Busy        0074 4107    174     00:00:03 
+any=10@100[vaio]    Busy        0071 4100    175     00:00:03 
+any=10@100[vaio]    Busy        0072 4101    176     00:00:03 
+any=10@100[vaio]    Busy        0073 4106    177     00:00:03 
	

    List Cpus Heading

    Heading

    Description

    CPUSPEC[HOST] The cpu specification supplied by the user during submit.
    If hostgroups are specified (i.e. +any), the
    group is broken out into individual cpus, with the hostname
    of each processor shown in [brackets].
    STATE The state the cpu's 'task reservation' is in. In the following,
    'cpuserver' means the remote host serving the cpu.
    
        Idle        --  task idle, waiting for cpuserver
        Idle/Nak    --  job is still trying to reach the cpuserver
        Avail       --  cpuserver wants us to run a frame
        Start       --  found frame to start, reply to cpuserver
        Run         --  cpuserver telling us frame started ok
        Busy        --  frame is busy, reply to cpuserver
        CpuPass2    --  replies to cpuserver, remains in state
        JobPass     --  job told cpuserver we can't use its cpu (see NOTES for why)
        Unpass      --  remote reverting to idle state (cosmetic)

    FRAME The frame that is being rendered.
    PID The process id on HOST for the running render script. This is also
    the 'process group leader'.
    JOBTID The 'job task id' for this processor. Only useful as an
    argument commands like 'rush -rc'.
    ELAPSED The wall clock elapsed time in hours:minutes:seconds for the
    rendering frame.
    NOTES Various notes regarding the state of the job. These notes can
    sometimes be stale; the 'STATE' information is a more accurate
    representation of the state of the cpu.

  rush -lcf  
rush -lcf [-t timeout] [jobid..]

List Cpus Full.

A 'full' report for all of the current [specific] job's cpus.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -lf  
rush -lf [-t timeout] [jobid..]

List Frames.

Lists Frames for current [specific] job. Lists a job's frames to see what state they're in.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

% rush -lf                    # List Frames for the current job
STAT FRAME TRY HOSTNAME       PID     START          ELAPSED  NOTES
Run  5000  1   vaio           18092   02/26,02:15:40 00:00:21 
Run  5001  1   vaio           18093   02/26,02:15:40 00:00:21 
Run  5002  1   vaio           18100   02/26,02:15:50 00:00:11 
Que  5003  0   -              0       00/00,00:00:00 00:00:00 
Que  5004  0   -              0       00/00,00:00:00 00:00:00 
Que  5005  0   -              0       00/00,00:00:00 00:00:00 
Que  5006  0   -              0       00/00,00:00:00 00:00:00 
Que  5007  0   -              0       00/00,00:00:00 00:00:00 
Que  5008  0   -              0       00/00,00:00:00 00:00:00 
	

The columns contain the following information:

    List Cpus Heading

    Heading

    Description

    Stat The status of the frame. Can be any of:

    • Que - The frame is queued, ready to be run by an available cpu
    • Run - The frame is busy running
    • Done - The frame completed successfully
    • Fail - The frame failed to run successfully (exit code was 1)
    • Hold - The frame will not run until changed manually to Que

    The following are transitional states for frames, which you should only
    see for a short time:

    • Strt - Frame just started to run, no process id yet
    • Die - The frame is being killed, but hasn't received
      an exit code back yet

    Frame The frame number, padded 4 digits.
    Try The number of times this frame has been tried. Includes
    requeues due to exit(2), but does not include priority bumps.
    Hostname The name of the host that is running (or that ran) the frame.
    PID The process id of the render script. This is also the process group
    leader for the frame's process hierarchy. (In windows, the top level
    'Job Object')
    Start The date/time the render began running.
    Elapsed The wall clock elapsed time in hours:minutes:seconds.
    Notes Free form notes text, set by the user's invocation of rush -notes,
    e.g. 'rush -notes 0012:"License Error"'

  rush -lff  
rush -lff [-t timeout] [jobid..]

List Frames Full.

A 'full' report of all frames' state information.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -lfi  
rush -lfi [-t timeout] [jobid..]

List Frame Information.

Lists Frame info for current [specific] job, including average render times, ETAs, percent Busy/Done/Fail, etc. Example:

% rush -lfi vaio.139              # List Frame Info for job 'vaio.139'
                              Average    Total
Jobid        State Total Perc Elapsed    Elapsed    Average ETA
------------ ----- ----- ---- ---------- ---------- ------------------------
rotwang.3    Que   2     %4   -          -          -
rotwang.3    Run   2     %4   -          -          -
rotwang.3    Done  46    %92  00:00:11   00:08:36   Thu Feb 12 17:47:16 2004
rotwang.3    Fail  0     %0   -          -          -
rotwang.3    Hold  0     %0   -          -          -
rotwang.3    ---
rotwang.3    Render Elapsed: 00:02:25
rotwang.3      Render Start: 02/12,17:44:45
rotwang.3        Render End: 02/12,17:47:10
	

  • Jobid
    The jobid for this job. (If multiple jobs are being listed, this column shows which info goes with which job)

  • State
    The various states the frames can be in, and to the right, the accumulated data for all the frames in each of those states.

  • Total
    The total number of frames in each 'State'.

  • Perc
    The percentage of frames in each 'State'.

  • Average Elapsed
    The average render times. This is the sum of all the Done frames' render times divided by the total Done frames. (Currently only 'Done' state frames are shown)

  • Total Elapsed
    The total elapsed time for the frames; a sum of all the 'Elapsed' times in the 'Frames' report. (Currently only 'Done' state frames are shown)

  • Average ETA
    This is the estimated time of completion for the job, based on the average render time for frames rendered successfully so far vs. the wall clock time spent running the job. e.g.:

      oldest_start   = "The oldest start time for all Done frames"
      recent_end     = "Most recent end time for all Done frames"
      time_spent     = ( recent_end - oldest_start )
      time_per_frame = time_spent / total_done_frames
      time_to_go     = time_per_frame * ( total_que + total_hold )
      	
    This time will vary widely, since completion time is dependent on the cpu availability, which is always changing.

  • Render Elapsed
    The elapsed time for rendering, ie. how long the TD had to wait for the job to finish rendering. The difference between the 'Render Start' and the 'Render End' times.

  • Render Start
    The 'Start Time' for the first frame.

  • Render End
    The 'End Time' for the last frame that finished rendering.

Warning: the ETA is meant for ballpark estimates only, and is not meant to be taken literally.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -lhg  
rush -lhg [+hostgroup..]

(new in 102.31)

List Host Groups.

Shows all [specific] hostgroups your sysadmin has configured in rush.

If no arguments are specified, all hostgroups are listed. Otherwise, arguments can be the names of hostgroups to limit the report to display.

Both hostgroups and hostnames are sorted alphabetically.

% rush -lhg            # Lists all hostgroups when no arguments specified
  +any: erie
	ontario
	rotwang
	superior
	tahoe
	vaio

+dante: erie
	ontario
	superior

+intel: rotwang
	tahoe
	vaio

 +irix: erie
	ontario
	superior

+linux: rotwang
	tahoe
	vaio

% rush -lhg +dante     # Lists only the '+dante' hostgroup
+dante: erie
	ontario
	superior
	

  rush -lhc  
rush -lhc [criteria]

(new in 102.31)

List Host Criteria.
Shows all [specific] host criteria your sysadmin has configured in rush.

If no arguments are specified, all host criteria are listed. Otherwise, arguments can be the criteria names to limit the report to display.

Both criteria and host names are sorted alphabetically.

% rush -lhc            # Lists all criteria when no arguments specified
intel: vaio
       rotwang

 irix: erie
       ontario
       superior

linux: rotwang
       tahoe
       vaio

% rush -lhc linux      # Lists only hosts matching 'linux' criteria
linux: rotwang
       tahoe
       vaio
	

  rush -lj  
rush -lj [-t timeout] [remotehost]

List Jobs.

Shows all jobs running on the local [remote] host.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

[you@erie] % rush -lj                  # List Jobs on local workstation
STATUS   JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED REMARKS
-------- -------------- ------------------------- -------- ----- ----- ---- ---------- --------
Run      erie.167       100/340/master            tamu       %18    %0    8   01:15:28
Run      erie.169       100/410/master            tamu       %75    %0   20   01:11:21
Run      erie.170       100/390/master            tamu       %10   %34    4   01:10:44
Run      erie.171       100/360/master            tamu       %10   %34    4   01:02:43

[you@erie] % rush -lj tahoe            # List Jobs on remote host tahoe
STATUS   JOBID          TITLE                     OWNER    %DONE %FAIL BUSY    ELAPSED REMARKS
-------- -------------- ------------------------- -------- ----- ----- ---- ---------- --------
Run      tahoe.5        100/540/master            izzie      %25    %0    4   00:30:01
Run      tahoe.6        100/540/comp              izzie      %25    %0    4   00:30:02
Run      tahoe.10       100/640/master            izzie      %10    %0    4   00:22:48
Pause    tahoe.11       100/640/comp              izzie      %12    %0    4   00:22:49 Job paused.
	

The columns contain the following information:

    Job Report Heading

    Heading

    Description

    Status The status of the job.

    Here's a state diagram showing the various states the job status transitions through:


    Click on the image to see a larger version.

    • Wait - The job is waiting for another job or time to finish (WaitFor, -waitfor)
    • Pause - The job has been paused with -pause
    • Run - The job is available to run frames
    • Done - The job is done, all the frames rendered successfully
    • Fail - The job is done, some frames failed

    The following are transitional states for the job, which you should only
    see for a short time:

    • End - the job was ended with -end
    • Dump - the job was dumped with -dump
    • Done - all frames Done, about to run DoneCommand, -donecommand
    • DoneCmd - the configured DoneCommand is running
    • Remove - about to be removed from the system

    Jobid The job's jobid.
    Title The job's title (Title, -title).
    Owner The user that submitted the job.
    %Done The percentage of frames that are in the Done state.
    %Fail The percentage of frames that are in the Fail state.
    Busy The number of frames currently in the Run state.
    Elapsed The elapsed time in HH:MM:SS since the job started running.
    Remarks This field may contain messages like 'Job paused.' or
    'Wait for other jobs'.

  rush -ljf  
rush -ljf [-t timeout] [remotehost]

List Jobs Full.

Lists all jobs running on the local [remote] host, showing 'full information' in a multi-line format.

This report prints out more information than people usually need. This command is more intended to either be parsed by external scripts needing access to all information rush can provide, or for debugging.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.


[erco@tahoe] % rush -ljf
      Jobid: tahoe.9
      State: Done
   ActState: Run
      Title: HONDA
DefPriority: 1
     LogDir: //server/jobs/HONDA/SC4A/rushlogs
   LogFlags: 
 ImgCommand: xv //server/jobs/HONDA/SC4A/imgs/sc4a-%04d.tif
        Ram: 10
       Nice: 10
    MaxTime: 00:10:00
 FrameFlags: 
    Command: //server/jobs/HONDA/SC4A/render-vehicle 640 480
 ChkCommand: 
DoneCommand: 
   AutoDump: off
       User: erco (1000/100)
   DoneMail: -
  StartDate: Mon May 14 23:54:53 2001
    EndDate: -
    Elapsed: 11:22:34
     Frames: 50
       Cpus: superior=2@100 +farm=20@10
   Notes[0]: -
   Criteria: 
	

  rush -licpause  
rush -licpause [remotehost]

Pause Job Into A License Pause State.

Causes the job to pause for the number of seconds specified by the submit command 'LicPauseSecs' or by the most recent 'rush -licpausesecs' command.

Normally 'rush -licpause' is executed from within a render script, when it detects the renderer failed with a license error, causing the job to pause for typically a minute or two, then unpauses itself again.

For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors.

'rush -licpause' will be ignored if the job has already been paused with 'rush -pause'.


   :   (your render script)
   :
   ## YOUR RENDER COMMANDS HERE
   render /job/MYSHOW/MYSHOT/ribs/foo.rib
   set $err = $status

   if ( $err == $PRMAN_LIC_ERROR ) then

       # LICENSE ERROR OCCURRED -- PAUSE JOB, REQUEUE FRAME
       rush -licpause
       rush -notes ${RUSH_FRAME}:'License Error'
       exit 2

   endif
   :
	

  rush -licpausesecs  
rush -licpausesecs secs [jobid..]

Set Pause Time for License Errors

Sets the amount of time a job remains in the 'Pause' state when 'rush -licpause' is invoked from a render script.

For info on how to use 'rush -licpause' mechanism, see the 'How To' documentation on Render License Errors.

  rush -log  
rush -log [-tail] <framerange> [jobid..]

View Frame Error Logs.

Prints the frame logs for the specified frame range.

The '-tail' option can be specified to continuously output appended data as the log file grows.

Using the job's 'logdir' value, prints the logs found in that directory for the frames you specify, e.g.:

% rush -log 1-10,2 20   # view logs for frms #1-10 on 2s, and frm #20

  rush -logdir  
rush -logdir pathname [jobid..]

Log Directory.

Changes the job's log directory, where frame log files are written to.

This does not affect commands that are already running or have run. i.e., old logs are not moved to the new location, or renamed on the fly.

When changing the logdir with -logdir, no checks are made to see if 'path' is a valid directory.

(New in rush 102.31p and up)
'%s' can be used to insert the current jobid into the logdir path. Note: when the logdir is changed from the command line (rush -logdir) and has an embedded '%s', the directory is not created for you. Auto-creation of the logdir only occurs during job submission to avoid the "catch-22" scenario where the caller can't create the directory with a jobid in it if the jobid doesn't exist yet.

	rush -logdir //server/jobs/logs/%s    # eg. "//server/jobs/logs/tahoe.34"
        

See LogDir for more information.

  rush -logflags  
rush -logflags <-|keeplast|keepall> [jobid..]

Log Flags.

Changes the log flags for the job. Examples:

rush -logflags - # Logs are overwritten (Default)
rush -logflags keepall   # Keep all logs; concatenate old to *.old (e.g. 0012.old)
rush -logflags keeplast    # Like 'keepall', but only keeps last log (don't concatenate)

See LogFlags for more information.

  rush -maxcpus  
rush -maxcpus <#cpus> [jobid..]

Sets the maximum number of cpus a job can use.

Puts a cap on the entire job. If maxcpus is set to 0, there is no limit.

    MaxCpus
    rush -maxcpus 0 Disabled -- no limit
    rush -maxcpus 15 Limit job to no more than 15 cpus busy rendering

See MaxCpus for more information.

  rush -maxtime  
rush -maxtime <hh:mm:ss> [jobid..]

Sets maximum time for rendering frames.

Value is in HH:MM:SS (hours:minutes:seconds).

The system will automatically kill frames that exceed the specified maxtime.

You can control what happens to the frame when maxtime expires using MaxTimeState.

See the MaxTime submit command for more info.

  rush -maxtimestate  
rush -maxtimestate <que|fail|done|hold> [jobid..]

(New in rush 102.40 and up)

What happens when MaxTime expires.

By default, rush requeues (Que) a frame when its MaxTime timer expires. You can change this to any of:

    MaxTimeState Values
    rush -maxtimestate que Frames become "Que" when timer expires
    rush -maxtimestate fail Frames become "Fail" when timer expires
    rush -maxtimestate done Frames become "Done" when timer expires
    rush -maxtimestate hold Frames become "Hold" when timer expires

  rush -nice  
rush -nice <niceval> [jobid..]

Set Job's Niceness.

Sets the nice value for the job; 0 runs as normal user process, higher values (10, 20) increase niceness to others. See Nice submit command for more info.

  rush -notes  
rush -notes <framerange>:'notes..' [jobid..]

Frame Notes.

Sets Notes for frame(s) on current [specific] job. These notes will show up in the rightmost column of 'rush -lf' reports. Examples:

rush -notes 155:"license error"   # set notes for frame 155 to 'license error'
rush -notes 200-250:"redo" # set notes for frames 200 thru 250 to "redo"

  rush -offline  
rush -offline [-msg 'text'] [remotehost|+hostgroup..]

Offline My Machine.

Offlines the local [remote] daemon.

Lets frames that are already rendering complete before taking the processors offline. Once offline, no new frames will be started.

'rush -offline' is similar to 'rush -getoff', differing only in that -offline allows any currently rendering frames to complete before taking the processors offline. 'rush -getoff' will bump any running renders, freeing up the processors right away. The bumped frames will be requeued.

(New in rush 102.40 and up) An optional remark can be specified via "-msg 'text'" which will show up in 'rush -lac' reports, eg:

% rush -offline -msg 'Bad Memory' ontario
ontario: Offline

% rush -lac
HOST       OWNER    JOBID          TITLE                     FRM  PRI   PID        ELAPSED REMARKS
---------- -------- -------------- ------------------------- ---- ----- ------- ---------- --------
ontario    -        -              -                         -    -     -          Offline Bad Memory
:

Examples of using 'rush -offline':

rush -offline   # offline all cpus on local host
rush -offline tahoe # offline all of tahoe's cpus
rush -offline +farm   # offline the entire render farm
rush -offline -msg 'Failing drive' tahoe   # offline 'huron' with the remark 'Failing drive' (New in 102.40)

To return processors to the online state, use rush -online.

  rush -online  
rush -online [-msg 'text'] [remotehost|+hostgroup..]

Online My Machine.

Onlines the local [remote] daemon. Frames will be allowed to start running on the host.

(New in rush 102.40 and up) The '-msg' flag is not recommended for use with -online, due to the message's persistence in 'rush -lac' reports when online, but is supported for symmetry with the -getoff/-offline commands.

Examples of using 'rush -online':

rush -online   # online all cpus on local host
rush -online tahoe # online all of tahoe's cpus
rush -online +farm   # online the entire render farm

To take hosts offline, use either rush -offline or rush -getoff.

  rush -pause  
rush -pause [jobid..]

Pause Job

Pauses the current [specific] job. Busy frames will be allowed to finish, no new frames will be started.

To pause a job and kill the running frames, use:

    rush -pause ; rush -que run

To continue a job from pause, use rush -cont.

  rush -ping  
rush -ping [-t timeout] [remotehost|+hostgroup..]

Ping Rush Daemon.

(Administrative) Pings the local [remote] daemon to see if it's running, and what it's doing.

Use '-t timeout' to control the amount of time rush waits to make a connection to each remote before it gives up. The default is 40 seconds.

  rush -priority  
rush -priority [jobid..]

Change Default Job Priority.

Changes the default priority for the job, which affects all 'cpus' specifications that did not indicate priority values (and therefore used the default priority).

Example-- If a job submitted the following 'cpus' specifications:


    priority 10
    :
    :
    cpus erie=2@100    
    cpus tahoe=4
    cpus +any=10
    

..this would create a job with 2 cpus on erie at a priority of 100, and all other cpus at a priority of 10.

Changing this job later with 'rush -priority 20' would cause all cpus (except erie, which specified a priority) to change their priorities to 20.

See the Priority submit command for setting the initial default priority.
See Priority Description for a description of how priority values work.

  rush -push  
rush -push [-s secs] [-c tries] <file[,file..]> [host|+hostgroup]

Push File(s) To Remotes.

(Administrative) Pushes rush configuration files (such as rush.conf, hosts, etc) to remote servers. Used to distribute rush config files quickly.

  • -s secs - Specifies the number of seconds to wait for replies from remotes. Default is 5.

  • -c tries - Number of retries made if no response in the number of seconds specified (-s secs). Default is only 1 try is made.

Files to be 'pushed' must be specified by name. Only files in the rush etc directory can be 'pushed'. Specify filenames only, not full paths. More than one file can be specified, if separated by commas.

"file" may only contain the characters "A-Z", "a-z", "0-9", "-", "_", and a single ".". This prevents abusive use of -push, since only files in $RUSH_DIR/etc are affected.

Only root and the rush adminuser can use this command.

The sysadmin can enable/disable 'rush -push' operations via a setting in the rush.conf file, using the AllowPush flag.

% rush -push hosts,rush.conf # Push hosts and rush.conf file to all hosts
% rush -push rush.conf,license.dat     # Push rush.conf and license.dat to all rush hosts
% rush -push rush.conf +linux     # Push rush.conf to only hosts in the +linux hostgroup
% rush -push rush.conf tahoe huron     # Push rush.conf to hosts 'tahoe' and 'huron' only
% rush -push rush.conf,hosts,license.dat tahoe huron     # Push rush.conf, hosts, and license.dat to hosts
# 'tahoe' and 'huron' only

  rush -que  
rush -que <framerange|framestate..> [jobid..]

Re-Queue Frames

Changes frame(s) into the Que state for current [specific] job. If frames are running, they are killed and enter the 'Que' state on completion.

The frames to affect can either be specified as a frame range (i.e. 1-100) or as a frame state, for which all frames matching that state will be changed. Examples:

% rush -que 1-100 # Que frames 1 through 100
% rush -que fail # Que all frames currently Fail
% rush -que fail done    # Que all frames currently Fail or Done

  rush -ram  
rush -ram <ramval> [jobid..]

Set Job's Ram Use.

Sets the Ram use (in MB) for the job. See Ram submit command for more info.

  rush -ramlist  
rush -ramlist [remotehost..]

View the ram usage of rush jobs for localhost [remotehost] machine.

Shows a report of all running rush processes on the machine, including how much ram each has requested to reserve, and a running total for the available ram left over.

% rush -ramlist tahoe
STATE   JOBID/TITLE                    PRI    RAMUSE NOTES
Busy    neverland.57,grkenttest        1         800     
Busy    neverland.58,pflare            1         200     
					      ------
    Total ram on tahoe: 1024  
Available ram on tahoe: 24    
	

  rush -rc  
rush -rc <cpuspec|tidspec|hostname> [jobid..]

Remove Cpu

Removes Cpu from cpu list of current [specific] job. If the cpus being removed are busy running frames, the frames are killed and re-submitted elsewhere.

Cpus can be removed in one of several ways:

  • Individually by the JOBTID number, e.g. rush -rc .32
  • By hostname, e.g. rush -rc tahoe
  • By a particular cpu specification as shown in the 'rush -lc' report, e.g., rush -rc tahoe=4@200

To remove a cpu via 'JOBTID' values (e.g., 'rush -rc .334') you must precede each value with a period. When you delete by JOBTID, you are deleting single cpus from the 'rush -lc' report. Note: this report includes the JOBTID values so you can see which values to delete. If the cpu you delete is part of a larger specification, (e.g., tahoe=4@12), then the cpu count for the spec will be modified and the cpu count in that spec will be decremented as well (e.g., tahoe=3@12)

If you remove a hostname (e.g., 'rush -rc tahoe') then all cpu specifications that have that host name (e.g., tahoe=3@100) will be removed. Also, any hostgroups that expand to include that host will have that host removed from the expansion (e.g., +any=3@100, which includes tahoe).

 % rush -lc
CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
howland=2@900        JobPass     -    -       171     900   00:00:00 No more frames
howland=2@900        JobPass     -    -       172     900   00:00:00 No more frames
+any=1@1[howland]    JobPass     -    -       147     1     00:00:01 No more frames
+any=1@1[howland]    JobPass     -    -       148     1     00:00:00 No more frames
+any=1@1[nt-1]       JobPass     -    -       149     1     00:00:31 This is a 'neverhost'
+any=1@1[nt-1]       JobPass     -    -       150     1     00:00:31 This is a 'neverhost'
+any=1@1[rotwang]    JobPass     -    -       151     1     00:00:01 No more frames
+any=1@1[rotwang]    JobPass     -    -       152     1     00:00:01 No more frames

% rush -rc howland
nt-1.15 'howland=2@900': Removed
nt-1.15 '+any=1@1': 2 task(s) deleted

% rush -lc
CPUSPEC[HOST]        STATE       FRM  PID     JOBTID  PRI   ELAPSED  NOTES
+any=1@1[nt-1]       JobPass     -    -       149     1     00:00:37 This is a 'neverhost'
+any=1@1[nt-1]       JobPass     -    -       150     1     00:00:37 This is a 'neverhost'
+any=1@1[rotwang]    JobPass     -    -       151     1     00:00:08 No more frames
+any=1@1[rotwang]    JobPass     -    -       152     1     00:00:08 No more frames
	

If you remove a cpu specification (e.g., 'rush -rc +any=3@100'), it must match character-for-character the entry shown in the 'rush -lc' report for the job:

% rush -ac tahoe@100 # Add a cpu.
% rush -rc tahoe@100 # Now try to remove it
'tahoe@100' no such cpu specification # FAILED: need to use spec shown in 'rush -lc'
 
% rush -lc # Look at 'rush -lc' report
CPUSPEC      STATE  FRM   PID    ELAPSED .. 
tahoe=1@100  Run   0002  26747  00:00:11 ..    
# More complete specification in report.
 
% rush -rc tahoe=1@100 # Remove using spec shown in report
'tahoe=1@100' removed. # It works

  rush -reserve  
rush -reserve [-server <host>] <cpuspec> [ramval]

Reserve Cpus.

Reserves cpus by creating a 'dummy' job that holds the cpus. The job's title will be RESERVE. Reservation jobs are like any other-- to remove a reservation, dump the job. To change the reservation, you can use IRUSH or the usual rush commands like -ac/-rc to modify the job.

'cpuspec' indicates which processors are reserved, and at what priority. It is an error not to include a priority value in the cpuspec.

    (New in rush 102.40f and up)
    The optional '-server <host>' can be specified, to make 'host' the job server for the reserve job, instead of the default which is to have the job server be on the same host as the cpus being reserved.

A reservation job with a priority of '500' will prevent jobs with priorities less than 500, but yields to jobs higher than 500k. Including the 'k' flag in your reservation will also bump any renderings that are running on your machine that are lower than 500 priority. See Priority Description for more info on how rush priorities work.

As an example of reserving cpus on your workstation, let's assume you sit at a dual proc workstation called 'tahoe':

  • To reserve both processors of your dual proc workstation at a priority of '500', use:

    % rush -reserve tahoe=2@500k     # Reserve both cpus at 500k priority

  • To submit a job that actually renders on your own workstation, just submit with a cpu specification that requests your workstation at a priority higher than your reservation, with the 'kill' flag, e.g.:

    rush -reserve tahoe=1@510k     # Request one cpu on tahoe

  • To get both cpus, just request both as you normally would:

    rush -reserve tahoe=2@510k      # Request both cpus on tahoe

  • To remove the reservation for your machine, simply dump the reservation job.

  • To prevent all jobs from rendering on your workstation-- even your own-- make your reservation at a priority of 999. Or, just use 'rush -offline' and/or 'rush -getoff'.

The optional 'ram value' allows one to reserve an amount of ram per processor. This is useful on multiproc machines where you want to prevent large jobs from rendering on the few processors left unreserved. Increasing the ram value you reserve will decrease the size of jobs rush will allow to run on the unused processors.

If 'cpuspec' doesn't include the number of processors, one processor is assumed. If 'ramval' is unspecified, a value of '1' is assumed.

rush -reserve tahoe@998 # Reserve 1 cpu on tahoe@998
rush -reserve tahoe=2@998 128    # Reserve 2 cpus @998, 128MB of ram to each
rush -reserve -server univac tahoe=2@998 128    # Same as above, but jobserver will be 'univac'
rush -reserve tahoe=2@500 128 # Reserve 2 cpus @500, 128MB of ram to each

  rush -reorder  
rush -reorder <framerange..> [jobid..]

Reorder Frames.

Reorders frames in the frame list to the new order specified. <framerange> specifies the new order of the frames in the frame list. Several frame ranges can be specified. Unspecified frames are simply left alone.

Changing the order of the framelist affects the order frames are rendered in, since frames are issued from the top of the list, down.

Example-- If you have a job with 10 frames in the frame list that are in normal 1 thru 10 order, you can use 'rush -reorder' to get different orderings, such as these:

rush -reorder 10-1 # becomes 10 9 8 7 6 5 4 3 2 1
rush -reorder 1-10,2 2-10,2      # becomes 1 3 5 7 9   2 4 6 8 10

  rush -rf  
rush -rf <framerange..> [jobid..]

Remove Frames.

Removes frames from the frame list of the current [specific] job.

  rush -rn  
rush -rn <hostname|+hostgroup..> [jobid..]

Remove Nevercpus.

Removes Nevercpus for current [specific] job.

  rush -rotate  
rush -rotate [remotehost|+hostgroup..]

Rotate Daemon Log File

(Administrative) Rotates daemon logs on local [remote] host. Can be used in crontab scripts to rotate logs to parse output.

Logs can be automatically rotated with the LogRotateHour command in the rush.conf file.

Only root and the rush adminuser can use this command.

  rush -status  
rush -status [-s secs] [-c count] [remotehost|+hostgroup..]

Status Of Host.

(Administrative) 'rush -status' quickly reports the status of jobs and cpus on the local [remote] hosts.

Commonly used as 'rush -status' to get the status of the local host, or 'rush -status +any' for the status of all hosts.

Output is intended for other programs to parse and regenerate.

-s secs optionally sets the amount of time to wait for responses from remotes, default is 3.

-c count optionally specifies the number of times to continuously update the report, default is 1. -c 0 creates a continuously (indefinite) updating report.

The output contains several records of information for each host, one record per line. Host records start with an 'h' record, and terminate with a line of '--'.

7 different types of data records are possible. Data record types are defined by the first character in each record line, and can be one of:

    h - hostname header. Leads off records for the specified host.
    d - daemon information. Info about the running daemon.
    j - job information. One line per job.
    p - processor status. One line per processor. 
    C - Cpu status
    M - Memory status
    S - Swap status
    X - Error message (eg. "X PdhGetFormatted[usr0]: query not validated")
    
Each record has its own fields:

    h <hostname>
    d <sequence-id> <daemon> <version> <PID=pid> <online state> <jobs> <busy procs> <total procs>
    j <owner> <jobid> <job title> <job state> <elapsed> <percent done> <percent fail> <# frms busy> <"job remarks">
    p <owner> <jobid> <job title> <frame> <priority> <pid> <elapsed>
    C <ncpus> <usr%> <sys%> [<usr%> <sys%> ..]
    M <total mb> <used%>
    S <total mb> <used%>
    

  rush -submit  
rush -submit [remotehost]

Submit Job

Submits a job to be managed by the local [remote] job server. Optional [remotehost] can be specified, to make the remotehost the job server.

  rush -tasklist  
rush -tasklist [remotehost]

Task List.

Lists scheduled tasks on local [remote] cpu server. This is the scheduler's sorted list of tasks wanting to run on the host in question.

Tasks are sorted in the order they will be executed; the topmost task is the next one to be rendered.

Tasks that are busy are sorted to the bottom. Tasks that were denied by either the job, or by cpu criteria, are also sorted to the bottom.

Tasks are sorted into 'groups' of the same priority, and will round robin.

For a 'full' version of a tasklist report, use 'rush -tasklistfull'.

  rush -title  
rush -title <text> [jobid..]

Job Title

Sets title for current [specific] job. For more information, see Title.

  rush -trs  
rush -trs

Template Render Script

Prints the template render csh script on stdout. This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor.


% rush -trs > render_me 
% chmod +x render_me 
% vi render_me 
	

You will at least need to setup the render commands (the '### YOUR RENDER COMMAND(S) HERE' section) at minimum.

Caveat: the template render script can be modified and customized by the sysadmin via $RUSH_DIR/etc/templates.

  rush -try  
rush -try <newcount> <framerange..> [jobid..]

Set Try Count

Changes the 'Try' count for the frames specified.

A 'try count' is maintained for each frame in a job's frame list. This value is shown in the 'Try' column of 'rush -lf' reports, and is passed to render scripts via the $RUSH_TRY environment variable.

% rush -lf                     # List Frames to see try counts
STAT FRAME TRY HOSTNAME       PID     START          ELAPSED  NOTES
Done 0001  4   superior       4144    10/04,03:29:10 00:00:18 
Done 0002  4   tahoe          4160    10/04,03:29:30 00:00:16 
Done 0003  4   placid         4163    10/04,03:29:47 00:00:16 
Done 0004  6   huron          4168    10/04,03:30:04 00:00:16 
Done 0005  4   finger         4171    10/04,03:30:21 00:00:16 

% rush -try 0 1-5              # Reset Try count to zero
0001: tries was 4, now 0
0002: tries was 4, now 0
0003: tries was 4, now 0
0004: tries was 6, now 0
0005: tries was 4, now 0

% rush -lf                     # List Frames to see changes
STAT FRAME TRY HOSTNAME       PID     START          ELAPSED  NOTES
Done 0001  0   superior       4144    10/04,03:29:10 00:00:18 
Done 0002  0   tahoe          4160    10/04,03:29:30 00:00:16 
Done 0003  0   placid         4163    10/04,03:29:47 00:00:16 
Done 0004  0   huron          4168    10/04,03:30:04 00:00:16 
Done 0005  0   finger         4171    10/04,03:30:21 00:00:16
	

  rush -tss  
rush -trs

Template Submit Script.

Prints a template submit script on stdout. This can be redirected to a file, to quickly create a working script, which should then be customized using a text editor.


% rush -tss > submit_me 
% chmod +x submit_me 
% vi submit_me 
	

You will, for instance, need to customize the 'command', 'title', 'cpus' and 'ram' values at minimum.

Caveat: the template submit script can be modified and customized by the sysadmin via $RUSH_DIR/etc/templates.

  rush -uping  
rush -uping [-c count] [-s secs] [remotehost..]

UDP Ping.

(Administrative) UDP Pings the local [remote] daemon. Useful for checking for UDP packet dropping. If [remotehost] is unspecified, the local host is assumed.

Options:

  • -c <count>

    Sets the number of udp transmissions sent to the remote, using the number of seconds (-s secs) between each transmission. If -c is unspecified, the default is 1.

    (New in rush 102.40f and up) Zero can be specified for <count> to send udp transmissions indefinitely, or until you hit ^C.

  • -s <secs>

    Sets the number of seconds between transmissions. If -s is unspecified, the default is 5.


% rush -uping -c 5 -s 1 tahoe
>> UDP Packet Burst Response Test
>> Hit ^C when done.

       tahoe: uping 1 236020307 (0.0026 secs roundtrip)
       tahoe: uping 2 237020120 (0.0022 secs roundtrip)
       tahoe: uping 3 238020080 (0.0022 secs roundtrip)
       tahoe: uping 4 239020077 (0.0023 secs roundtrip)
       tahoe: uping 5 240019993 (0.0022 secs roundtrip)

	

  rush -waitfor  
rush -waitfor [-|waitjobid|timeval,[..,..]] [jobid]

Wait For Other Jobs or Times

Sets the 'waitfor' jobids or times for the current [specified] job. See the submit script docs on WaitFor for more info.

(New in 102.40) 'timeval' specifies a time to wait for. 'timeval' can be any of:

    WaitFor Time Values
    +8h 8 hours from now
    +60m 60 minutes from now
    +08:30:00 8 hours 30 minutes from now
    17:00 at 5pm
    5:30p at 5:30pm
    5:30pm at 5:30pm
    17:45,07/22/2002 at 5:45pm on 07/22/2002
    12:30p,07/22/2002 at 12:30pm on 07/22/2002

The 'WaitFor' time value will show up in 'rush -ljf' reports in two places; once in the 'WaitForTime:' field as an absolute time in MM/DD/YY,HH:MM:SS format, the other in the 'WaitFor:' field as a relative time in +HH:MM:SS format if the time has not yet expired.

For a tutorial on how to have one job wait for another, see Chaining Jobs.

Read the examples carefully to avoid confusing the [jobid] with the [waitjobids]. Remember to specify commas to separate the [waitjobids] instead of spaces. If only one [waitjobid] is being specified, add a trailing comma.

CAVEATS

  • Using 'rush -waitfor' clears all previous waitfor jobids and times

  • Setting 'waitfor' to multiple jobids AND times means ALL conditions must be met before job will start.

  • Specifying 'waitfor -' will disable the job from waiting for other jobs and times.

  • The jobs being waited for must be served by the same machine as the job doing the waiting. (i.e. the waitfor jobids and the job doing waiting must all be on the same jobserver)

  • Note the strange syntax where commas are mandatory when specifying the first argument after '-waitfor'. This prevents the parser from being confused as to which is the 'waitjobid' and which is the jobid to affect.

To differentiate between the new value of 'waitfor' and the jobid(s) to be affected, the new value of 'waitfor' must be a comma-separated list of jobids or times with no spaces. Examples:

rush -waitfor - # Disable all waitfor's
rush -waitfor tahoe.37,tahoe.38, # Job waits for 'tahoe.37' and 'tahoe.38' to complete ***
rush tahoe.40 -waitfor +8h, # tahoe.40 won't start running until 8 hours from now
rush tahoe.40 -waitfor tahoe.37,tahoe.38, # tahoe.40 now waits for 'tahoe.37' and 'tahoe.38' to complete ***
rush tahoe.40 -waitfor tahoe.37, # tahoe.40 now waits for 'tahoe.37' to complete ***
# Trailing comma on tahoe.37 is important!
rush tahoe.40 -waitfor 5:30p,tahoe.37,tahoe.38, # tahoe.40 won't start until 5:30pm and
# until 'tahoe.37' and 'tahoe.38' complete ***

*** depends on setting of 'rush -waitforstate'.

See also, rush -dependon, rush -waitforstate

  rush -waitforstate  
rush -waitforstate [done|donefail|fail|dump] [jobid]

Controls circumstances that cause WaitFor to trigger.

See the submit script docs on WaitForState for more info.

Examples:

rush -waitforstate dump # Causes 'waitfor' to wait on other job(s) to Dump themselves (Default)
rush -waitforstate fail # Causes 'waitfor' to wait on other job(s) to be Fail
rush -waitforstate donefail # Causes 'waitfor' to wait on other job(s) to either be Fail or Done
rush -waitforstate done # Causes 'waitfor' to wait on other job(s) to be Done