CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Memory issue while opening and reading a file

 



koti688
Novice

Apr 3, 2009, 5:06 AM

Post #1 of 19 (3377 views)
Memory issue while opening and reading a file Can't Post

Hi Monks,
i am facing memory issue while opening and reading the contents of a file.

My Script is below.


Code
my $sigDataFile = getValueFromPath($event ,"data/file"); 
open(TEST,"<$sigDataFile") || die("This file will not open!");
my @records = ();
@records = <TEST>;
chomp @records;
close(TEST);
my @keys;
my @values;
foreach my $rec(@records) {
my ($key,$value)=split(/\=/,$rec);
push @keys,$key;
push @values,$value;
}


My $sigdata file is a flat text file about 3 Gb which has a format of key=value on each line , like below.

<key>=<value>
<key>=<value>
<key>=<value>
<key>=<value>
<key>=<value>
........
........
........
<key>=<value>
<key>=<value>
<key>=<value>
<key>=<value>
<key>=<value>

While i run my script on command prompt. it says like "Out of Memory".Can you please tell me , is there any fault in my script while opening and reading the contents of the file.

My System details are like
Pentium(R) 4 CPU 2.26 GHz
1.00GB of RAM.

Thanks in advance
You can reply Your suggetions on koti688@gmail.com


Tumata
User


Apr 3, 2009, 5:18 AM

Post #2 of 19 (3375 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

Are you joking? Wink

You shouldn't read whole file (especially if you know that it is very big). You run out of free memory. Instead reading the whole file it would be better read it line by line (or even more better by some number of lines).


koti688
Novice

Apr 3, 2009, 5:23 AM

Post #3 of 19 (3373 views)
Re: [Tumata] Memory issue while opening and reading a file [In reply to] Can't Post

Hi tumata,
can you please tell me how to read it line by line after opening the file.

Can you please explain a little bit in the code.


koti688
Novice

Apr 3, 2009, 5:25 AM

Post #4 of 19 (3371 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

can i dod this way



Code
 
my (@keys, @values);
open(TEST,"<$sigDataFile") || die("This file will not open!");
while(<TEST>) {
my ($key,$value)=split /=/;
push @keys,$key;
push @values,$value;
}
close(TEST);

print @keys;
print @values;



Tumata
User


Apr 3, 2009, 5:33 AM

Post #5 of 19 (3369 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

Ok. After you opened file for reading do something like this:

Code
#!/usr/bin/perl -w 

my $filename = '/tmp/text.txt';

open FH, "<$filename" or die ($!);

while ( <FH> ) {
print;
}

close FH;


It will display whole your file line by line.


Tumata
User


Apr 3, 2009, 5:34 AM

Post #6 of 19 (3367 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

Yes! Great! But why don't you use hash?


FishMonger
Veteran / Moderator

Apr 3, 2009, 6:47 AM

Post #7 of 19 (3363 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post


In Reply To
can i dod this way



Code
 
my (@keys, @values);
open(TEST,"<$sigDataFile") || die("This file will not open!");
while(<TEST>) {
my ($key,$value)=split /=/;
push @keys,$key;
push @values,$value;
}
close(TEST);

print @keys;
print @values;


That sill puts your entire file into memory, but now you're using 2 arrays instead of 3. The only advantage is that you're not duplicating the data.

What do you need to do with the data?


(This post was edited by FishMonger on Apr 3, 2009, 6:48 AM)


koti688
Novice

Apr 3, 2009, 6:55 AM

Post #8 of 19 (3360 views)
Re: [Tumata] Memory issue while opening and reading a file [In reply to] Can't Post

hash to read the keys and values ??? how to do it .

i need to send the key and the value to my Back end . after reading the keys and values into respective arrays. like below.


Code
foreach my $key (@keys) { 
my $length = length($values[$idx]);
foreach my $FakeFEObject (@FakeFEObjects) {
my $out = $FakeFEObject->putData($key, $length, $values[$idx]);
if ($out == 0) {
push @failedKeys, $key;
}
}
$idx++;
}

above FakeFEObject is a object created to send the keys and values to the Back End.

Is there any other other solutions you trying to tell to send my Keys and respected values with this created object


koti688
Novice

Apr 3, 2009, 6:58 AM

Post #9 of 19 (3360 views)
Re: [FishMonger] Memory issue while opening and reading a file [In reply to] Can't Post

Hi Fishmonger
i need to send the key and the value to my Back end . after reading the keys and values into respective arrays. like below.


Code
	 
foreach my $key (@keys) {
my $length = length($values[$idx]);
foreach my $FakeFEObject (@FakeFEObjects) {
my $out = $FakeFEObject->putData($key, $length, $values[$idx]);
if ($out == 0) {
push @failedKeys, $key;
}
}
$idx++;
}

above FakeFEObject is a object created to send the keys and values to the Back End.

Is there any other other solutions you trying to tell to send my Keys and respected values with this created object .


FishMonger
Veteran / Moderator

Apr 3, 2009, 8:22 AM

Post #10 of 19 (3349 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

When opening a filehandle:
1) Use a lexical var for the filehandle
2) Use the 3 arg form of open
3) The die statement should include the filename and the reason it failed

Don't load the entire file into 1 more arrays that you later loop over. Instead, process each line 1 at a time passing it to $FakeFEObject->putData


Code
open my $TEST,'<', $sigDataFile or die "'$sigDataFile' file will not open $!"; 

my @failedKeys;
while( <$TEST> ) {
chomp;
my ($key,$value) = split /=/;
my $out = $FakeFEObject->putData($key, length($value), $value);

# you might want to use a hash or put the entire line into the array instead of just the key.
push @failedKeys, $key if $out == 0;
}



1arryb
User

Apr 3, 2009, 9:30 AM

Post #11 of 19 (3345 views)
Re: [FishMonger] Memory issue while opening and reading a file [In reply to] Can't Post

Hi Fish,

I can understand why you prefer lexically scoped filehandles, but why prefer the 3 arg open? Do they go together?

Larry


KevinR
Veteran


Apr 3, 2009, 9:42 AM

Post #12 of 19 (3343 views)
Re: [1arryb] Memory issue while opening and reading a file [In reply to] Can't Post


In Reply To
Hi Fish,

I can understand why you prefer lexically scoped filehandles, but why prefer the 3 arg open? Do they go together?

Larry


Its a security issue. It discussed in the perl open tutorial:

http://perldoc.perl.org/perlopentut.html

Lexical filehandles and the 3 argument style are not bound together except as a "best practices" rule.
-------------------------------------------------


(This post was edited by KevinR on Apr 3, 2009, 9:44 AM)


FishMonger
Veteran / Moderator

Apr 3, 2009, 9:57 AM

Post #13 of 19 (3340 views)
Re: [1arryb] Memory issue while opening and reading a file [In reply to] Can't Post

The main reason (and possibly the only reason) for using the 3 arg form is to have a clear separation between the mode and the expression (filename) which adds readability. The 3 arg form can be done with either lexical or bareword filehandles.


FishMonger
Veteran / Moderator

Apr 3, 2009, 10:04 AM

Post #14 of 19 (3339 views)
Re: [KevinR] Memory issue while opening and reading a file [In reply to] Can't Post

I forgot about the possible security issue.

It's a vague reference, but I think this is the section of the tutorial that Kevin is referring to.


Quote
There is also a 3-argument version of open, which lets you put the special redirection characters into their own argument:

open( INFO, ">", $datafile ) || die "Can't create $datafile: $!";

In this case, the filename to open is the actual string in $datafile , so you don't have to worry about $datafile containing characters that might influence the open mode, or whitespace at the beginning of the filename that would be absorbed in the 2-argument version. Also, any reduction of unnecessary string interpolation is a good thing.



KevinR
Veteran


Apr 3, 2009, 10:40 AM

Post #15 of 19 (3336 views)
Re: [FishMonger] Memory issue while opening and reading a file [In reply to] Can't Post

Yes, thats exactly what I was referring too. If you ever have to write code that is used in a high risk environment the three argument style is pretty much mandatory.
-------------------------------------------------


Tumata
User


Apr 3, 2009, 12:58 PM

Post #16 of 19 (3333 views)
Re: [FishMonger] Memory issue while opening and reading a file [In reply to] Can't Post

Why do you advise to use a lexical var for the filehandle?


KevinR
Veteran


Apr 3, 2009, 3:07 PM

Post #17 of 19 (3327 views)
Re: [Tumata] Memory issue while opening and reading a file [In reply to] Can't Post


In Reply To
Why do you advise to use a lexical var for the filehandle?


Some reasons I am aware of:

Filehandles are normally global in scope, but lexical filehandles are not, unless you declare them globally. Also "strict" will catch a typo in a filehandle name if you use a lexical filehandle but it can't for a bareword filehandle, and perl automatically closes a file when a lexical filehandle goes out of scope. You can also use the case of your choice with a lexical filehandle where as with a bareword you should stick with all uppercase to avoid collisions with reserved words.
-------------------------------------------------


FishMonger
Veteran / Moderator

Apr 4, 2009, 7:11 AM

Post #18 of 19 (3306 views)
Re: [koti688] Memory issue while opening and reading a file [In reply to] Can't Post

Cross post
http://www.experts-exchange.com/Programming/Languages/Scripting/Perl/Q_24291912.html


(This post was edited by FishMonger on Apr 4, 2009, 7:12 AM)


KevinR
Veteran


Apr 4, 2009, 10:30 AM

Post #19 of 19 (3296 views)
Re: [FishMonger] Memory issue while opening and reading a file [In reply to] Can't Post

Thanks Fish
-------------------------------------------------

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives