CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Basic question about improving speed of a script

 



sessmurda
Novice

Dec 15, 2010, 12:33 PM

Post #1 of 3 (702 views)
Basic question about improving speed of a script Can't Post

So I need to loop through a few files and compare specific columns. The most basic way of doing this is through nested foreach loops, and my current script is running that fine, it just will take a long time. I was wondering if there were any quicker substitutes for a nested foreach statement, as later I was going to do this again, but with 3 files to build a new one. The main issue is that each of these files is >2million lines, so any suggestion on improving the speed would be greatly appreciated.

Thanks


Zhris
Enthusiast

Dec 17, 2010, 11:11 AM

Post #2 of 3 (686 views)
Re: [sessmurda] Basic question about improving speed of a script [In reply to] Can't Post

Hi,

I've always found that when I must handle huge amounts of data, using the module Tie::File dramatically speeds things up for me.

Chris


sessmurda
Novice

Dec 17, 2010, 1:20 PM

Post #3 of 3 (682 views)
Re: [Zhris] Basic question about improving speed of a script [In reply to] Can't Post

Thanks, ended up using hashes for this which was so much faster (<60sec)


Code
use warnings; 
use Data::Dumper;
open (FILE1, "$ARGV[0]");
open (FILE2, "$ARGV[1]");

my %data;
my %data2;
while (my $line1 = <FILE1>)
{
chomp $line1;
my @fields1 = split(/\t/, $line1);
$data{$fields1[0]} = [ $fields1[1] ];
}
while (my $line2 = <FILE2>)
{
chomp $line2;

my @fields2 = split(/\t/, $line2);
unless (exists $data{$fields2[0]})
{
$data2{$fields2[0]} = [ $fields2[1] ];
}
}
foreach my $ID (keys %data2)
{
my ($info) = @{$data2{$ID}};
print "$ID\t$info\n";
}


 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives