From: Greg Ercolano <erco@(email surpressed)>
Subject: [Q+A] Is there a way to gather accounting data from Rush?
   Date: Mon, 28 Nov 2005 12:48:21 -0800

> The producers of our show have asked me to collect accounting data
> for some of our shots, and thought Rush would be the place to get
> this information easily. Does rush keep track of all the renders it
> runs in an accounting file somewhere?
>
> Also, we'd like to generate some reports showing the usage of our
> render farm, to justify the purchase of new machines, or to split
> our shifts so we can get the most out of our render farm.
>
> Is there a way to do this, and are there any example scripts
> that can get us started so we can generate our own 'producer friendly'
> reports?

    You mean reports that look like this:
    http://www.seriss.com/rush-current/gifs/rush-utilization.gif

    ;)

    Yes, just about all the data you're interested is being collected
    as we speak into your cpu.acct file..

The cpu.acct File
-----------------
    Rush accumulates data on each machine for every frame it renders
    in the rush/var/cpu.acct file.

    This file is simple ascii data in white space delimited fields
    (like all the rush reports), and is one line of data per rendered frame
    that can easily be parsed by your own scripts (perl/python/java/awk..)
    to generate web pages, Excel data, simple ascii reports, or loaded
    into your own SQL databases for later processing.

    The 'per line' data includes the start time, job title, user,
    jobid, number of seconds the frame ran, and the exit code (0/1/2)
    indicating success/failure/retry.

    The cpu.acct file format is ascii, with one line of data per
    rendered frame, and is documented here:
    http://www.seriss.com/rush-current/rush/rush-cpu-acct.html

    This should have all the info you need to total up render times for
    each user/project, and be able to determine farm utilization.

Example "Farm Utilization Script"
---------------------------------
    Here's an example script that shows how to write a perl script
    to parse the data from the cpu.acct file.

    This script expects cpu.acct file(s) on stdin, and will generate
    a 'utilization report' in the form of a graph showing the percent
    utilization for each machine, basically how much time each machine
    spends rendering each day:

REPORT: http://seriss.com/rush-current/examples/cpu-actualization/cpu-actualization.out
SCRIPT: http://seriss.com/rush-current/examples/cpu-actualization/cpu-actualization-pl.txt

    You can download the script as 'cpu-actualization.pl', and invoke it as:

    UNIX: perl cpu-actualization.pl -asciigraph /usr/local/rush/var/cpu.acct
 WINDOWS: perl cpu-actualization.pl -asciigraph c:/rush/var/cpu.acct

    If you're using the new version of rush (102.42 and up), you can use
    the new 'rush -catlog' command to collect data from ALL the machines,
    and pump that data through the script to get a picture of the whole farm, eg:

         rush -catlog cpu.acct +any > all-cpu.acct
         perl cpu-actualization.pl -asciigraph all-cpu.acct

    ..or even as one line via a pipe:

         rush -catlog cpu.acct +any | perl /var/tmp/cpu-actualization.pl -asciigraph -

    In the comments at the top of the 'pl' script, you can see the command line
    usage that outputs either ascii graphs (default), or 'tab delimited' data
    (-spreadsheet) loadable by Excel/OpenOffice to generate nice 3d graphs, eg:

    SPREADSHEET IMAGE: http://www.seriss.com/rush-current/gifs/rush-utilization.gif
              COMMAND: rush -catlog cpu.acct +any | perl /var/tmp/cpu-actualization.pl -spreadsheet - > out.csv

Example Day Report
------------------
    If you configure your jobs with a 'jobdumpcommand' that sweeps the 'jobinfo'
    files (from your job's log directory) into a central location, you can
    generate things like "day reports" for jobs:

	http://seriss.com/rush-current/examples/dayreport/dayreport.out
	http://seriss.com/rush-current/examples/dayreport/dayreport-pl.txt

    Some producers find this useful to show the status of shots.
    The job titles can be coded (by edict to the animators, or via custom
    submit scripts) to indicate final renders vs. tests, so producers can
    determine which shots finaled, and which shots are still in progress.

New Rush 102.42+ Commands
-------------------------
    With the new version of rush (102.42 and up), there are a few new rush
    command line tools that let you more easily manage the cpu.acct file
    on all machines:

    rush -catlog
    ------------
    Lets you *collect* the cpu.acct file from all machines easily, eg:

        rush -catlog cpu.acct +any > all-cpu.acct

    Docs:
    http://www.seriss.com/rush-current/rush/rush-command-line-options.html#-catlog

    rush -rotate
    ------------
    Lets you *rotate* the cpu.acct log on each machine, eg:

        rush -rotate cpu.acct +any

    Docs:
    http://www.seriss.com/rush-current/rush/rush-command-line-options.html#-rotate

    The other way to rotate that log is via crontab.

    By default, the cpu.acct file never rotates out, so it probably has data
    going back to the day you installed rush.