Bryan Blevins wrote:
[posted to rush.general]
Hi all!
Currently, when we submit a job with rush. All the nodes pick up the job at
the same time, and kill my little ole server. All the nodes are fighting
for data all the way through the render. What I've been doing is to
manually online the nodes one at a time, waiting about a minute between
each. It works great for my server, and the renders go smooth, but is such
a pain to do.
Is there a way in Rush to have a slight pause (say 30-60secs) between the
que frames so that my server can dish out the data to the nodes more evenly?
You can add some stagger code to your submit script so that it
sleeps a little before starting the render.
I'd recommend making the amount of sleep time a function of the
frame number, with a cut off above a certain frame number, though
just as easily it could be some random value.
For instance, if you want the first 20 frames to have increasingly
longer sleep times, you could add these lines to the -render section
of the script:
----
if ( $ENV{RUSH_FRAME} < 50 )
{ sleep($ENV{RUSH_FRAME} * 2); }
----
..which will cause frame 1 to sleep 2 seconds, frame 2 to sleep 4 secs,
frame 10 to sleep 20 seconds, etc.
Or, you can have frames sleep in blocks:
----
if ( $ENV{RUSH_FRAME} < 50 )
{ sleep( int( $ENV{RUSH_FRAME} / 10 ) * 30 ); }
----
..where frames 1-9 won't sleep at all, 10-19 will sleep 30 secs,
20-29 will sleep 60 seconds, etc.
Or, a possibly better approach is to embed an 'rsh' command in the script
that polls the server for its current load, and if the load is above some value,
sleep some amount of time and check again. (below)
Note: the following code is not tested, but should give you an idea
on how to approach it. Basically you want to parse out the first load value
from 'uptime', and if it's above some number you find to be the max load
you want your server to be, this will cause the render to sleep until the
load returns to normal.
----
# WAIT FOR SERVER'S LOAD TO GO BELOW SOME MAXIMUM
while ( 1 )
{
my $max_load = 3; # change to taste
# GET CURRENT LOAD FROM SERVER
my $server_load = `rsh myserver uptime`;
if ( $server_load =~ /load averages: (\d+)/ )
{
$server_load = $1;
print "--- SERVER LOAD IS $server_load (MAX=$max_load): ";
# IF LOAD VALUE LOW ENOUGH, START RENDER
if ( $server_load <= $max_load )
{ print "OK\n"; last; }
}
else
{ print "--- ERROR GETTING SERVER'S LOAD: '$server_load': "; }
# WAIT AND TRY AGAIN
print "sleep 20\n";
sleep(20);
}
----
--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)
|