>
> On Dec 15, 2009, at 9:22 PM, Greg Ercolano wrote:
>
> [posted to rush.general]
>
> Mr. Daniel Browne wrote:
>> I'm writing a basic submit script to perform a unix cp copy =
>> operation on the farm.
>
> It sounds like you're submitting a job that copies frames,
> but want it to be done with one command, as opposed to one
> copy command per frame..?
Correct
>
> Since copying is pure I/O, I would think the best place to
> run a large frame copy command is only on the file server,
> to avoid network in/network out, so that the copy operation
> is entirely local.
Unfortunately the BlueArc architecture currently doesn't allow you do to operations like that within the NAS head. In fact, most of the head's shell commands do not support recursive operations.
>
> For this reason I'm not sure I understand the question;
> normally one wants to run things through the render queue
> to harness the power of parallel cpu use, where each machine
> works on a frame or batch of frames. Either that, or you want
> to throw a single operation at some available machine, and have
> it do the entire operation as a 'single frame rush job'.
>
We had thought since we perform other operations on frames to and from volumes on the NAS head that we could do the same with copy operations. I admit, it sounds a bit zany. At the very least I do have a script that would allow me to hand off the copy operation to a single farm machine as opposed to splitting it into pieces. Theoretically this would be faster as the head is executing these operations in parallel with the help of its cache memory.
>
> If you're trying to do shell substitution, I'm not sure, do you
> mean wild card expansions, like:
>
> cp /some/directory/foo.[0-9]*.tif /some/other/directory/
>
>> The alternative of course is to do a cp =
>> operation for each frame, but I was hoping to avoid the additional =
>> overhead.
>
> I wouldn't think the cp command would be much overhead,
> but I might be missing something.
>
> If you're trying to distribute the cp commands to a bunch
> of machines to parallelize the I/O, I'd think you'd want
> to do them either as single frames or in batches.
>
> Or better yet, batch several commands on the file server
> itself, eg:
>
> cp foo.0*.tif /some/other/dir & # copies 1000 frames (0000 - 0999) in bg
> cp foo.1*.tif /some/other/dir & # copies 1000 frames (1000 - 1999) in bg
> cp foo.2*.tif /some/other/dir & # etc..
>
> ..which if the box has multiple procs and the network is idle,
> should go really fast.
>
> If you can follow up with more specifics, I can probably
> help you narrow down a specific technique.
Yes, this is what I wanted to go for. The problem obviously is that you can't wildcard a frame range as a whole; you have to wildcard each digit with its potential range of values. Tedious, but possible. I was just hoping you might have a library routine already in place to do this.
I think I know what to do now; thanks Greg. Happy holidays.
-Dan
>
> --
> Greg Ercolano, erco@(email surpressed)
> Seriss Corporation
> Rush Render Queue, http://seriss.com/rush/
> Tel: 626-576-0010x23
> Fax: 626-576-0020
> Cel: 310-266-8906
>
>
> ----------
> Dan "Doc" Browne
> System Administrator
>
> Evil Eye Pictures
> d.list at evileyepictures.com
> Office: (415) 777-0666
>
|