From: Greg Ercolano <erco@(email surpressed)>
Subject: 102.42d beta3
   Date: Mon, 02 Jul 2012 14:53:04 -0400
Msg# 2250
View Complete Thread (1 article) | All Threads
Last Next
All 102.42d beta testers: here's a list of items that will be fixed in beta3, which will be going out later today.
(Below that, a list of the 102.42dbeta2 modifications that you probably have already.)

BETA3 FIXES
    o Fixed problem with -fmt parsing that caused the first argument past
      the format string to be ignored. eg:
     
                rush -lj -fmt "Jobid Status" tahoe spirit erie
                                             -----
                                               |
                                               'tahoe' argument was being ignored

    o Sped up jobid lookups for 'rush -lj `cat jobids`' took ~30 seconds
      to list 4000 jobs (whereas 'rush -lj hostname' was instant).
     
      Replaced linear lookups using IsJobid() with JOBID_GetID() and doing simple
      integer lookups.

    o Fixed problem with 'rush -lj' not clearing away hostname on the tty if report is small, eg:
         rush -lj -fmt Remarks
      ..would print the hostname and a CRLF, then the report was supposed to clear
      away the hostname. But if the report is small (such as in the above, a '-' response would leave
      most of the hostname behind). Small changes to handle doing a cleol.

    o IMPORTANT: Added code to prevent checkpoint writes WHILE daemon is loaded checkpoints.
      It was possible to end up loosing jobs if one restarts the daemon while
      the daemon is loading a checkpoint.  (BRENT/LUMA)

    o Added 'WaitForDependency:' to -ljf report (FUELVFX/PAWEL)

    o Fixed duplicate entries for 'Dependency' in -ljf reports
      Jobids were showing up twice because submits now assign jobid first,
      causing two registrations to happen.(FUELVFX/PAWEL)

    o Mac: switched irush's menubar to use the Mac's system menu bar.
      Internals: Switched MyMenuBar from using Fl_Menu_Bar -> Fl_Sys_Menu_Bar

Internals:

    o Updated FLTK 1.3.x-svn to r9562
         > Fl_Tree mods for irush2 wip
         > Enabled FLTK_ABI_FEATURES



BETA2 FIXES

> 'rush -cp' can now increase or decrease the priority of all cpus
  rush -cp '*' @+5   -- increase priority by +5
  rush -cp '*' @-5   -- decrease priority by -5
  rush -cp +any=50@100 @+10  -- increase priority from 100 to 110

  Irush makes use of this feature in the 'Jobs' and 'All Jobs' report,
  so that one can easily raise/lower the priority of several jobs at once:



> 'rush -cp' now tries to keep the CPUSPEC column of 'rush -lc' synchronized with priority mods
  If all the tasks of a cpu specification are changed to the same priority value,
  then the cpuspec is updated. Example:

    $ rush -lc
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  JOBID          NOTES
    tahoe=2@100            Idle/Nak    -    -       2497    100   00:00:00 tahoe.8        -
    tahoe=2@100            Idle/Nak    -    -       2498    100   00:00:00 tahoe.8        -

    $ rush -cp tahoe=2@100 @200           *** CHANGE PRIORITY FROM @100 TO @200 ***
    tahoe.8 tahoe=2@200: priority was tahoe=2@100, now tahoe=2@200

    $ rush -lc                                                    *** CPUSPEC FIELD UPDATES ***
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  JOBID          NOTES
    tahoe=2@200            Idle/Nak    -    -       2497    200   00:00:00 tahoe.8        -
    tahoe=2@200            Idle/Nak    -    -       2498    200   00:00:00 tahoe.8        -

    $ rush -cp .2497 @300k                          *** CHANGE PRIORITY OF ONLY ONE OF THE TWO TASKS **
    tahoe.8 tahoe=2@200: priority was @200, now @300k

    $ rush -lc                                      *** CPUSPEC FIELD WILL *NOT* UPDATE ***
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  JOBID          NOTES
    tahoe=2@200            Idle/Nak    -    -       2497    300k  00:00:00 tahoe.8        -
    tahoe=2@200            Idle/Nak    -    -       2498    200   00:00:00 tahoe.8        -

    $ rush -cp .2498 @300k                          *** CHANGE PRIORITY SO THAT *BOTH* TASKS NOW AGREE **
    tahoe.8 tahoe=2@300k: priority was @200, now @300k

    $ rush -lc                                      *** CPUSPEC FIELD NOW UPDATES ***
    CPUSPEC[HOST]          STATE       FRM  PID     JOBTID  PRI   ELAPSED  JOBID          NOTES
    tahoe=2@300k           Idle/Nak    -    -       2497    300k  00:00:00 tahoe.8        -
    tahoe=2@300k           Idle/Nak    -    -       2498    300k  00:00:00 tahoe.8        -


> Added new taskcleanup algorithm
  Purges stale jobs out of the tasklists more frequently, lighter load.
  Rush.conf now configured to run the new cleanup every 3 hours:
      taskcleanuphours 0 3 6 9 12 15 18 21

  New behavior can be disabled via rush.conf with:
      use_taskcleanup_opt 0
      taskcleanuphours    5

> 'rush -laj' now supports large reports (solves "Message too long" errors)
      The following flags were added to the rush.conf file to manage the internals for this change:
      use_udptransactors      0       # disables use of new udp transactors (reverts to old behavior)
      udptransactor_fragsize  512     # max size of udp transactor fragmenting (-lac,-laj,-status,etc)
      udptransactor_ttl       5       # #secs udp transactors remain on server before expiring
      udptransactor_choke     0       # #usecs to usleep between sending /each/ frag sent by server

> Added 'processor affinity' support for linux.
   So now both windows and linux can have 'affinity=x' in the hosts file.
   (Still no affinity support for OSX: Apple's operating system does not support it)

> 'rush -push' now uses file compression to allow larger files to be pushed quickly
  The following rush.conf option specifies the file size at which point
  compression is used. (9000 bytes is the default):
    udpcompress 9000
  Compression can be disabled with:
    udpcompress 0

> Added ()'s to cpu specifications, eg. +any(1-4)=5@100 
  ..which lets one specify only hosts 1 through 4 in the +any group.
  Forward ported from 102.42a8b (ROMEY/ZOIC)
  This breaks the network protocol, so bump from 102.42c -> 102.42d


> Added 'rush -ljc' (List Job Counters).

  Every rush job now maintains 'change counters' which are incremented
  whenever an aspect of the job has changed:
	 'frame counter' -- incremented whenever the job's frames 
                            are added, removed, or change state
  This feature allows 'watcher scripts' to /easily/ see if some aspect
  of the job has changed, so that it can know if the job's info needs
  to be reloaded. This helps prevent a watcher script from constantly
  having to reload all info on every job to detect if anything changed.

> Root can submit jobs IF forceuid/forcegid configured AND uidrange/gidrange includes roots uid/gid

> 'rush -af' allows multiple frame ranges to be specified. eg:
  rush -af 1-10 20-45

> 'rush -catlog': added 'mail.log' and 'jobs-checkpoint' as 
  optional filenames to view.

> rush.conf: 
  Added the following optional commands:

     > Added 'cpuacct.dbasedir':      

         Usage: cpuacct.dbasedir [path] [dirmask] [filemask]
       Example: cpuacct.dbasedir "/server/somewhere" 0755 0644

       Example would cause cpu.acct logs to be written as:

	   /server/somewhere/[YYYY]/[MM]/[DD]/[HOSTNAME]-cpu.acct
	                     ---------------------------
			     |
			     Automatically generated dirs
			     and filename

	This feature will be used by web scripts that provide
	network wide utilization reports (WARNER), eg:

> Added 'net.resvauth', which allows disabling of forcing client connections to use port numbers below 1024. (affects unix only). Prevents problems with numerous client connections causing 'no more reserved ports' errors. Usage: net.resvauth [0|1] Example: net.resvauth 0 # disables use of reserved ports (disables default) > Added 'version=###' prefix, to allow conditional code execution based on the rush version#. > Added 'usercmd' to 'permit' section, to allow per-user control of the "rush -submit" 'user [uid] [gid] [username]' submit command. > Added the following submit commands: commandflags <flags> -- adjusts attributes of job's command jobstartcommandflags <flags> -- adjusts attributes of job's jobstartcommand jobrestartcommandflags <flags> -- adjusts attributes of job's jobrestartcommand jobdonecommandflags <flags> -- adjusts attributes of job's jobdonecommand jobdumpcommandflags <flags> -- adjusts attributes of job's jobdumpcommand ..and equivalent command line flags: -commandflags <flags> -jobstartcommandflags <flags> -jobrestartcommandflags <flags> -jobdonecommandflags <flags> -jobdumpcommandflags <flags> These flags control how commands execute: quote -- Handle quoting expandvars -- Expand environment variables of the form ${VARNAME} at runtime escape -- Handle escape character (\) useshell -- Use a native shell to run the command (unix:use bash, windows: use DOS) nolog -- (jobdonecommand only) disables creating jobdonecommand.log done -- (jobdonecommand only) runs command only if all frames are "Done" fail -- (jobdonecommand only) runs command only if any frames are "Fail" and rest are "Done" donefail -- (jobdonecommand only) runs command only if all frames are "Done" or "Fail" always -- (jobdonecommand only) always runs command when job is done (default) off -- turns off all the above flags So for instance: rush -submit <<EOF .. command python //server/path/myscript.py -scene //server/path/${RUSH_JOBID} commandflags useshell,expandvars .. EOF ..tells rush to use a bash or DOS shell to run the above command, and to expand the "${RUSH_JOBID}" at render time. o Added -fmt flag to 'rush -lj' and 'rush -laj' This allows jobs reports to be customized from the command line, eg:
See 'rush -lj -fmt help' for the large list of all the possible field names. These fields can also be used in the rush.conf file for these new options: # "rush -lj" rush.lj.format "-Status<STATUS>:-8 Jobid<JOBID>:-16 Title<TITLE>:-25 Owner<OWNER>:-14 %Done<%DONE>:5 %Fail<%FAIL>:5 FramesBusy<BUSY>:4 Elapsed<ELAPSED>:10 Remarks<REMARKS>" # "rush -laj" rush.laj.format "-Status<STATUS>:-8 Jobid<JOBID>:-16 Title<TITLE>:-25 Owner<OWNER>:-14 %Done<%DONE>:5 %Fail<%FAIL>:5 FramesBusy<BUSY>:4 Elapsed<ELAPSED>:10 Remarks<REMARKS>" # "rush -lj -x" + "rush -laj -x" rush.lj_x.format "-Status<Status>:-8 Jobid<Jobid>:-16 Title<Title>:-25 Owner<Owner>:-14 %Done<%Done>:5 %Fail<%Fail>:5 %Hold<%Hold>:5 Frames<#Frms>:5 FramesBusy<Busy>:5 Priority[Vals]<Priority>:-8 Ram:5 Progress<Progress>:-20 RenderElapsed<Rendertime>:12 Elapsed<Age>:12 Remarks<Remarks>" # "rush -lac" rush.lac.format "Hostname:-15 Owner:-14 Jobid:-16 Title:-15 Frame:-7 Priority<Pri>:-5 Pid:-7 Elapsed:10 Remarks" # "rush -lc" rush.lc.format "CpuSpec[Host]:-22 State:-11 Frame:-7 Pid:-7 JobTid:-7 Priority:-8 Elapsed:-8 Jobid:-14 Notes" # "rush -lf" rush.lf.format "Status:-6 Frame:-7 Try:-3 Hostname:-15 Pid:-7 Jobid:-16 Start:-14 Elapsed:-8 Notes" o Added 'rush -jobdata' and 'jobdata' submit command This allows generic info to be added to a job, similar but separate from 'job notes'. o Removed unused (and undocumented) -checkpoint command ("-jobs-checkpoint" is the only ever advertised command line flag) o Removed unused (and undocumented) jobchkcommand o input: > Added Edit -> Preferences -> Help Messages to allow disabling of new submit form tooltips. [IRUSH FEATURES: WIP] > Cleaner headings in reports > Column resizing is now done in the header instead of the middle of the report > Added more right-click menu features > IRUSH_FRAMES and IRUSH_JOBS environment variables help "Hotkey" scripts determine which frames have been selected in Frames reports, and whhich jobs have been selected in Jobs and All Jobs reports. [INPUT FEATURES: WIP] o SUBMIT SCRIPTS o > Added FixDriveMaps() to .common.pl All the new submit scripts run this function in their MAIN_Render() section to remap drives. o Added $G::rushlogdir option to .common.pl so that admin can force all logs to be written to a single directory. (default is off) *** TODO: DO THIS WITH ALL BIG SUBMIT SCRIPTS *** See the logdir section of submit-afterfx.pl o submit-afterfx: brought over ability to submit paths with spaces from mocean mods (03/22/2012) *** TODO: DO THIS WITH ALL BIG SUBMIT SCRIPTS *** See use of SpaceEscape() and SpaceUnescape(). Internals: o Updated FLTK 1.3.x-svn to r9545 > Has fixes for SGI build



Last Next