I need to find a way to re-submit/re-queue failed frames when submit
from batches.
I might be missing something.
Let's say after rendering a batch of 20-29, frame 25 fails,
and you have a way to detect this by grepping the logs.
So you'll know for sure that 20-24 are OK, and 25-29 need to be
re-rendered.
If you just exit the script with exit(2), rush will try to re-render
the entire batch of 20-29.
Are you saying that you want it to:
a) Assume the problem was intermittent (ie. maya crashed due to a bug,
and will probably render 25-29 just fine if we re-invoke maya to just
render that frame range?
..or..
b) Assume the problem was caused by the user (a bad scene file)
and we should just append frames 25-29 to a text file somewhere
that we can later use to submit a job to fix these frames
after the user has intervened to fix the scenefile.
If a) then yes, just re-invoke maya with the new frame range,
and the user won't even have to know there was a failure.
If b) then I would think you could do any of a number of things:
o Have the render script create 0000.fix frames in the image dir
that a later 'fix job' could look for, and just render those frames
o Append the bad frames to a .txt file
o Have a 'jobdonecommand' that looks for bad frames, and submits
a 'fix job' in the pause state, and emails the user to fix the
problem, then unpause the fix job to run the fix frames..
It's easy to re-gueue the whole batch but if only one frame has failed
it's re-rendering all the good frames again.
If you have the logic in the render script to know which frames
need to be re-rendered, and are confident that just re-rendering those
is all that's needed to get them to render OK, then just re-invoke maya
with just the fix range.
Or, if you just want to tell the user which frames are bad, you can
use 'rush -notes $ENV{RUSH_FRAME}:"BAD FRAMES: $bad_start - $bad_end"'
so that the "Frames" report shows a message telling which frames were bad.
(this is safe to do when errors occur. Using 'rush -notes' isn't recommended
if run on EVERY frame.. that's too much load to the job server if there are
100's of render nodes. But logging (uncommon) error conditions is OK..)
--
Greg Ercolano, erco@(email surpressed)
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Cel: (Tel# suppressed)
Fax: (Tel# suppressed)
|