CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
Memory usage increasing

 



robertico
Novice

Mar 29, 2013, 9:02 AM

Post #1 of 3 (364 views)
Memory usage increasing Can't Post

I'm running this script and memory usage is increasing while it's processing the file.

The file is approx. 10 Gb.

I've added 'undef' but it doesn't matter.

Every chunk is just 1 Mb, so why is memory usage increasing ? (memory usage is approximately equal to bytes read)



Code
#!/usr/bin/perl 

use Try::Tiny;
use strict;
use warnings;
use File::Basename;
use IO::Handle;

my $substr= 'messageid';
my $infile = 'D:\MyFile.bin';
my $file = basename( $infile );
open (INFILE, $infile) or die "Can't open $infile: $!";
binmode INFILE;

my $outfile = "+>>" ."D:\\" .$file .".txt";
open (OUTFILE, $outfile);
OUTFILE->autoflush(1);

my $data="";
my $n=0;
my $strStart=0;
my $strEnd=0;

while (($n = read INFILE, $data, 1048576, $offset) !=0) {
if ($data =~ /messageid/) {
$strStart = $-[0];
$strEnd = $+[0];
print OUTFILE $offset ."\t" .$strStart ."\t" .$strEnd ."\n";
$offset = $strEnd + 1;
undef $data;
}
else{
$offset = $offset + $n -9;
undef $data;
}
}

close (INFILE);
close (OUTFILE);



(This post was edited by robertico on Mar 29, 2013, 1:10 PM)


FishMonger
Veteran / Moderator

Mar 30, 2013, 9:59 AM

Post #2 of 3 (345 views)
Re: [robertico] Memory usage increasing [In reply to] Can't Post

You should run a few tests using Devel::Size to find out how much memory each var is using. I suspect that it's not the one you think.
http://search.cpan.org/~nwclark/Devel-Size-0.78/lib/Devel/Size.pm

The code you posted won't compile, so I must assume that it's an untested trimmed up version of your real script.


Quote
memory usage is approximately equal to bytes read

By that I assume you mean the accumulated total of the bytes read and not the value of $n. If that's the case, then I would suspect the problem to be with $offset. I have not run any tests, but it's possible that the read function is reserving memory equal to (or more than) the offset value.

Instead of using the offset option of the read function, I'd probably use the seek and tell functions to reset the file pointer.


robertico
Novice

Mar 31, 2013, 9:25 PM

Post #3 of 3 (329 views)
Re: [FishMonger] Memory usage increasing [In reply to] Can't Post


Quote
Instead of using the offset option of the read function, I'd probably use the seek and tell functions to reset the file pointer.


That's it. Thank you very much.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives