Home: Perl Programming Help: Intermediate:
Basic question about improving speed of a script



sessmurda
Novice

Dec 15, 2010, 12:33 PM


Views: 1046
Basic question about improving speed of a script
So I need to loop through a few files and compare specific columns. The most basic way of doing this is through nested foreach loops, and my current script is running that fine, it just will take a long time. I was wondering if there were any quicker substitutes for a nested foreach statement, as later I was going to do this again, but with 3 files to build a new one. The main issue is that each of these files is >2million lines, so any suggestion on improving the speed would be greatly appreciated.

Thanks


Zhris
Enthusiast

Dec 17, 2010, 11:11 AM


Views: 1030
Re: [sessmurda] Basic question about improving speed of a script
Hi,

I've always found that when I must handle huge amounts of data, using the module Tie::File dramatically speeds things up for me.

Chris


sessmurda
Novice

Dec 17, 2010, 1:20 PM


Views: 1026
Re: [Zhris] Basic question about improving speed of a script
Thanks, ended up using hashes for this which was so much faster (<60sec)

Code
use warnings; 
use Data::Dumper; 
open (FILE1, "$ARGV[0]"); 
open (FILE2, "$ARGV[1]"); 
 
my %data; 
my %data2; 
while (my $line1 = <FILE1>) 
{ 
    chomp $line1; 
    my @fields1 = split(/\t/, $line1); 
    $data{$fields1[0]} = [ $fields1[1] ]; 
} 
while (my $line2 = <FILE2>) 
{ 
    chomp $line2; 
 
    my @fields2 = split(/\t/, $line2); 
    unless (exists $data{$fields2[0]}) 
{ 
    $data2{$fields2[0]} = [ $fields2[1] ]; 
} 
    } 
foreach my $ID (keys %data2) 
{ 
    my ($info) = @{$data2{$ID}}; 
    print "$ID\t$info\n"; 
}