CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Parallel unix system/backticks commands

 



sousuffer
New User

Jan 16, 2014, 9:00 AM

Post #1 of 2 (741 views)
Parallel unix system/backticks commands Can't Post

Hello all,

I'm new to the forums, so I apologize if this is simpler than I think. I have an array of files stores in the variable @files. For each file, I would like to run a unix-based command line program called FastQC, which generates an output file. I have 5 nodes with 12 cores per node, so I would like to be able to utilize this power to run 50 instances of the FastQC program simultaneously as part of my perl pipeline.

Ideally, I would also like to grab the exit code or output for each instance, since piping the output to a file would add an extra scanning step to ensure that all processes completed successfully - but if I have to do this I will.

The below code runs the processes sequentially, producing the result and an ".out" file with the status of the run:

for my $file (@files)
{
my $fastqc_script = "$fastqc_folder/fastqc
+ $input_folder/$file --outdir=$fastqc_output_folder >
+ $fastqc_output_folder/$file.out";
`$fastqc_script`;
}

The goal would be primarily to start the second run immediately after starting the first one, etc. The secondary goal would be to collect the exit codes in an array as they complete. Any assistance would be greatly appreciated. Thank you very much in advance for your help.


FishMonger
Veteran / Moderator

Jan 16, 2014, 9:29 AM

Post #2 of 2 (739 views)
Re: [sousuffer] Parallel unix system/backticks commands [In reply to] Can't Post

Take a look at Parallel::ForkManager - A simple parallel processing fork manager
http://search.cpan.org/~szabgab/Parallel-ForkManager-1.06/lib/Parallel/ForkManager.pm

Before you go that route, you should profile your script to see where it's spending most of its time.
Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
http://search.cpan.org/~timb/Devel-NYTProf-5.06/lib/Devel/NYTProf.pm

How long does it currently take for the script to process the files? Is it only 50 files, or do you have hundreds/thousands of file that you want to batch out 50 at a time?

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives