From: Greg Ercolano <erco@(email surpressed)>
Subject: Re: How do I submit jobs with a staggered start?
   Date: Sun, 23 Oct 2005 12:18:04 -0700
Msg# 1065
View Complete Thread (4 articles) | All Threads
Last Next
Bryan Blevins wrote:
[posted to rush.general]

Hi all!

Currently, when we submit a job with rush. All the nodes pick up the job at the same time, and kill my little ole server. All the nodes are fighting for data all the way through the render. What I've been doing is to manually online the nodes one at a time, waiting about a minute between each. It works great for my server, and the renders go smooth, but is such a pain to do.

Is there a way in Rush to have a slight pause (say 30-60secs) between the que frames so that my server can dish out the data to the nodes more evenly?

	You can add some stagger code to your submit script so that it
	sleeps a little before starting the render.

	I'd recommend making the amount of sleep time a function of the
	frame number, with a cut off above a certain frame number, though
	just as easily it could be some random value.

	For instance, if you want the first 20 frames to have increasingly
	longer sleep times, you could add these lines to the -render section
	of the script:

----
if ( $ENV{RUSH_FRAME} < 50 )
    { sleep($ENV{RUSH_FRAME} * 2); }
----

	..which will cause frame 1 to sleep 2 seconds, frame 2 to sleep 4 secs,
	frame 10 to sleep 20 seconds, etc.

	Or, you can have frames sleep in blocks:

----
if ( $ENV{RUSH_FRAME} < 50 )
    { sleep( int( $ENV{RUSH_FRAME} / 10 ) * 30 ); }
----

	..where frames 1-9 won't sleep at all, 10-19 will sleep 30 secs,
	20-29 will sleep 60 seconds, etc.

	Or, a possibly better approach is to embed an 'rsh' command in the script
	that polls the server for its current load, and if the load is above some value,
	sleep some amount of time and check again. (below)

	Note: the following code is not tested, but should give you an idea
	on how to approach it. Basically you want to parse out the first load value
	from 'uptime', and if it's above some number you find to be the max load
	you want your server to be, this will cause the render to sleep until the
	load returns to normal.

----
# WAIT FOR SERVER'S LOAD TO GO BELOW SOME MAXIMUM
while ( 1 )
{
    my $max_load = 3;                   # change to taste

    # GET CURRENT LOAD FROM SERVER
    my $server_load = `rsh myserver uptime`;
    if ( $server_load =~ /load averages: (\d+)/ )
    {
        $server_load = $1;
        print "--- SERVER LOAD IS $server_load (MAX=$max_load): ";

        # IF LOAD VALUE LOW ENOUGH, START RENDER
        if ( $server_load <= $max_load )
            { print "OK\n"; last; }
    }
    else
        { print "--- ERROR GETTING SERVER'S LOAD: '$server_load': "; }

    # WAIT AND TRY AGAIN
    print "sleep 20\n";
    sleep(20);
}
----

--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)

Last Next