From: Greg Ercolano <erco@(email surpressed)>
Subject: Re: Batch Copy Script
   Date: Wed, 16 Dec 2009 14:27:09 -0500
Msg# 1920
View Complete Thread (4 articles) | All Threads
Last Next
Mr. Daniel Browne wrote:
>>> The alternative of course is to do a cp operation for each frame,
>>> but I was hoping to avoid the additional overhead.
>> 
>> 	I wouldn't think the cp command would be much overhead,
>> 	but I might be missing something.
>>
>> 	If you're trying to distribute the cp commands to a bunch
>> 	of machines to parallelize the I/O, I'd think you'd want
>> 	to do them either as single frames or in batches.
>> 
>> 	Or better yet, batch several commands on the file server
>> 	itself, eg:
>> 
>> 		cp foo.0*.tif /some/other/dir &		# copies 1000 = frames (0000 - 0999) in bg
>> 		cp foo.1*.tif /some/other/dir &		# copies 1000 = frames (1000 - 1999) in bg
>> 		cp foo.2*.tif /some/other/dir &		# etc..
>>
>> 	..which if the box has multiple procs and the network is idle,
>> 	should go really fast.

> Yes, this is what I wanted to go for. The problem obviously is that you
> can't wildcard a frame range as a whole; you have to wildcard each digit
> with its potential range of values. Tedious, but possible. I was just
> hoping you might have a library routine already in place to do this.

	If you're using perl, you can use perl's built in copy command
	and make a loop, eg:

use File::Copy;
[..]
    for ( $t=$sfrm; $t<$efrm; $t++ ) {
        my $src = sprintf("/some/src/path/foo.%04d.tif", $t);
        my $dst = sprintf("/some/dst/path/foo.%04d.tif", $t);
        copy($src,$dst);
    }

	..however you might find the operating system's own copy command
	to be faster (these might be optimized to take advantage of threading)
	in which case:

    for ( $t=$sfrm; $t<$efrm; $t++ ) {
        my $src = sprintf("/some/src/path/foo.%04d.tif", $t);
        my $dst = sprintf("/some/dst/path/foo.%04d.tif", $t);
        system("cp $src $dst");
    }

	..and you could probably get 3 or 4 of those going at a time
	with use of fork()/wait() or threads. (I'd suggest throttling
	this to limit starting too many at a time, and using wait(),
	you can start the next one as soon as one gets done, so that
	there are always 3 or 4 going at a time, until you work your
	way through the entire loop)

	For instance, I have a script that I wrote a while ago:
	http://seriss.com/people/erco/unixtools/prun

	..the blurb description:

*********************************************************************************
Runs unix commands in parallel

Supply a list of commands to run, one per line, and it runs them all in parallel.
You can limit the number of parallel processes with the -M flag. Even keeps track
of all the stdout/stderr logs, serializing them for easy reading. Hit ^C to stop
all processes.

Great for multiple rsh, rcp or rdists to run on a whole network quickly.
Very small, simple program no one has time to write, but everyone always needs
when managing multiple hosts.
*********************************************************************************

	So it takes a file of one liner unix commands, and runs them in
	parallel with a throttle, backgrounding several at a time,
	each time one command gets done it moves further onto the next
	in the list (like rush's frame queue) keeping the throttle maxed.

	It was handy for running commands many copy commands in parallel
	and keeping the output syncrhonized. Might be helpful
	with or without Rush..!

	I hope it still works.. it looks like I wrote it last century,
	as it still uses pre-perl5 syntax.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Last Next