From: Greg Ercolano <erco@(email surpressed)>
Subject: Re: Batch Copy Script
   Date: Wed, 16 Dec 2009 00:22:30 -0500
Msg# 1918
View Complete Thread (4 articles) | All Threads
Last Next
Mr. Daniel Browne wrote:
> 	I'm writing a basic submit script to perform a unix cp copy =
> operation on the farm.

	It sounds like you're submitting a job that copies frames,
	but want it to be done with one command, as opposed to one
	copy command per frame..?

	Since copying is pure I/O, I would think the best place to
	run a large frame copy command is only on the file server,
	to avoid network in/network out, so that the copy operation
	is entirely local.

	For this reason I'm not sure I understand the question;
	normally one wants to run things through the render queue
	to harness the power of parallel cpu use, where each machine
	works on a frame or batch of frames. Either that, or you want
	to throw a single operation at some available machine, and have
	it do the entire operation as a 'single frame rush job'.

> Is there a routine already available in Rush to =
> perform shell substitution of the digits in a frame range, or would I =
> have to do that myself?

	Hmm, I might need a few more details.

	If you're making a submit script to do a copy, the scripting
	language you're using should work well for doing shell substitution.

	Do not try to submit to rush a simple 'cp' command, but rather,
	have the script submit itself, so that the script runs the cp command
	on each machine, so that you'll have all the facilities of the
	scripting language you're working with at your finger tips.
	(ie. shell wildcard expansion, sed/perl/awk access, etc)

	For some simple examples of writing scripts that
	"submit themselves", see:
	http://www.seriss.com/rush-current/rush/rush-submit.html

	For digits in a frame range, there are two variables rush supplies
	at execution time on all the machines; RUSH_FRAME and RUSH_PADFRAME,
	the latter being 0000 format, and the former having no padding at all.

	If you need some different padding, you can use printf() to change
	RUSH_FRAME into whatever padding you want, eg:

		perl -e 'printf("%06d", $ENV{RUSH_FRAME});'

	If you're trying to do shell substitution, I'm not sure, do you
	mean wild card expansions, like:

		cp /some/directory/foo.[0-9]*.tif /some/other/directory/

	..or do you really mean substitution, like sed/perl regex
	to turn e.g. foo.0001.tif into foo.0002.tif?

> The alternative of course is to do a cp =
> operation for each frame, but I was hoping to avoid the additional =
> overhead.

	I wouldn't think the cp command would be much overhead,
	but I might be missing something.

	If you're trying to distribute the cp commands to a bunch
	of machines to parallelize the I/O, I'd think you'd want
	to do them either as single frames or in batches.

	Or better yet, batch several commands on the file server
	itself, eg:

		cp foo.0*.tif /some/other/dir &		# copies 1000 frames (0000 - 0999) in bg
		cp foo.1*.tif /some/other/dir &		# copies 1000 frames (1000 - 1999) in bg
		cp foo.2*.tif /some/other/dir &		# etc..

	..which if the box has multiple procs and the network is idle,
	should go really fast.

	If you can follow up with more specifics, I can probably
	help you narrow down a specific technique.

-- 
Greg Ercolano, erco@(email surpressed)
Seriss Corporation
Rush Render Queue, http://seriss.com/rush/
Tel: (Tel# suppressed)
Fax: (Tel# suppressed)
Cel: (Tel# suppressed)

Last Next