CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Memory Leak

 



adussaq
Novice

Sep 30, 2009, 1:57 PM

Post #1 of 3 (479 views)
Memory Leak Can't Post

I am working with a file of about 15 GB (I have 98 GB of RAM) taking the lines into a hash and checking to see if they are repeated so:


Code
if ( defined ( $masterHash{$line} ) ) 
{
$masterHash{$line}++;
}
else
{
$masterHash{$line} = 1;
$count++;
}


This continues on printing out the "$masterHash" periodically based on predefined parameters ie what percent of the data in the following fashion:

Code
my $hereFile = $littleFile . "/" . "${FNUM}-$tmpFile"; 
$hereFile = "$hereFile"."$orgCount"."organisms_"."$percent"."percent_"."$count"."unique_motifs.txt";
push( @files, $hereFile );
open( HERE, ">$hereFile" )or warn ;

print HERE "#$orgCount organsisms\t$percent percent of total unique motifs\t$count total unique motifs\n";
foreach my $key ( keys %masterHash )
{
print HERE "$key\t$masterHash{$key}\n";
}
close( HERE );

When doing this the memory fills up really fast to the point that I crash my system, I figure I have a memory leak, but I am not quite sure how perl stores the memory nor am I sure how to get around this. I have tried using the module to autoflush the buffer, but that only slowed the program down and did not seem to free up any memory, any ideas?


FishMonger
Veteran / Moderator

Sep 30, 2009, 4:16 PM

Post #2 of 3 (466 views)
Re: [adussaq] Memory Leak [In reply to] Can't Post

Here are a couple modules that will help you to troubleshoot your script.

Devel::ObjectTracker - Track object life to detect memory leaks
http://search.cpan.org/~johnc/Devel-ObjectTracker-0.4/ObjectTracker.pm

Devel::Monitor - Monitor your variables/objects for memory leaks
http://search.cpan.org/~phcote/Devel-Monitor-0.9.0.7/lib/Devel/Monitor.pm

Devel::Size - Perl extension for finding the memory usage of Perl variables
http://search.cpan.org/~tels/Devel-Size-0.71/lib/Devel/Size.pm

The code you posted does have a couple issues, but it's too small of a sample to say if it's part of the problem you describe.


adussaq
Novice

Sep 30, 2009, 4:24 PM

Post #3 of 3 (462 views)
Re: [FishMonger] Memory Leak [In reply to] Can't Post

Thank you I will give that a try, if you have time can you tell me what the "issues" are with my code, I have been programming perl for a while, but I need to get better and I have a feeling that these issues arise in all of my code.....

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives