CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
Search Posts SEARCH
Who's Online WHO'S
Log in LOG

Home: Perl Programming Help: Intermediate:
Parallel unix system/backticks commands


New User

Jan 16, 2014, 9:00 AM

Post #1 of 2 (2756 views)
Parallel unix system/backticks commands Can't Post

Hello all,

I'm new to the forums, so I apologize if this is simpler than I think. I have an array of files stores in the variable @files. For each file, I would like to run a unix-based command line program called FastQC, which generates an output file. I have 5 nodes with 12 cores per node, so I would like to be able to utilize this power to run 50 instances of the FastQC program simultaneously as part of my perl pipeline.

Ideally, I would also like to grab the exit code or output for each instance, since piping the output to a file would add an extra scanning step to ensure that all processes completed successfully - but if I have to do this I will.

The below code runs the processes sequentially, producing the result and an ".out" file with the status of the run:

for my $file (@files)
my $fastqc_script = "$fastqc_folder/fastqc
+ $input_folder/$file --outdir=$fastqc_output_folder >
+ $fastqc_output_folder/$file.out";

The goal would be primarily to start the second run immediately after starting the first one, etc. The secondary goal would be to collect the exit codes in an array as they complete. Any assistance would be greatly appreciated. Thank you very much in advance for your help.

Veteran / Moderator

Jan 16, 2014, 9:29 AM

Post #2 of 2 (2754 views)
Re: [sousuffer] Parallel unix system/backticks commands [In reply to] Can't Post

Take a look at Parallel::ForkManager - A simple parallel processing fork manager

Before you go that route, you should profile your script to see where it's spending most of its time.
Devel::NYTProf - Powerful fast feature-rich Perl source code profiler

How long does it currently take for the script to process the files? Is it only 50 files, or do you have hundreds/thousands of file that you want to batch out 50 at a time?


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives