CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
putting 500K rows into an Array?

 



Alf
New User

Apr 18, 2011, 11:46 PM

Post #1 of 6 (1529 views)
putting 500K rows into an Array? Can't Post

Hi Guys,

I'm having problem trying to load a file into an array.

The file is more than 500,000 lines and I am trying to load it using the following:

open(FILE,"filename.txt");

binmode(FILE);

@FileData = <FILE>;

close(FILE)

This works fine for files up to 480K lines, no problem. But once the file is above 500K then it just stops

and never get to the next line of code in the perl script. I've been using this few lines for many years now

and this is the 1st time I c this happening. I put some debug lines in to just simply print a message in between

every line of code and found it reached the line @FileData = <FILE>;

and then that's it.

I am on ActivePerl v5.6.1 built MSWin32-x86-multi-thread (I know, it's old) and have 3.8Gb ram (with 1.9Gb free

at point of execution of the code). I didn't observe any major RAM drain during the runs.

Is there a Limit in Perl Array size? I can't ever remember reading a limit from any where.

Help please?



Thx a million


rovf
Veteran

Apr 19, 2011, 5:36 AM

Post #2 of 6 (1521 views)
Re: [Alf] putting 500K rows into an Array? [In reply to] Can't Post

I don't know about Perl 5.6, but at least in 5.10, there is no 500K limit on the array size. Whether this applies to your version is something you can easily test yourself from the command line:


perl -lwe 'my @a; for(my $i=0; $i<600000; ++$i) {$a[$i]=$i}; print "ended"'


Depending on the size of your lines, of course it might be a space problem, in particular since you are running it on Windows. Do you really have to slurp that huge file into memory? Maybe Tie::File would be an alternative.


Alf
New User

Apr 19, 2011, 3:59 PM

Post #3 of 6 (1505 views)
Re: [rovf] putting 500K rows into an Array? [In reply to] Can't Post

Hi rovf,

thx for the reply. I've subsequently put your line into a perl script file and it ran fine. And then I cut out those few lines from my main program and into a separate clean perl script and that ran fine too. So it's not the array size that's causing the problem. It's something else. I'll continue my investigation. Thanks for your help.

regards.


miller
User

Apr 19, 2011, 5:00 PM

Post #4 of 6 (1504 views)
Re: [Alf] putting 500K rows into an Array? [In reply to] Can't Post

I'm guessing that the problem is not the size of your array, but of the size of your file. Unless your perl is compiled with uselargefiles=define, then it won't be able to handle files over 2gigs in size.

Just do a perl -V and see if you have that capability. If you don't then you'll either have to upgrade your perl or use split to split the file into managable bites for your version of perl.

- Miller


Alf
New User

Apr 19, 2011, 10:06 PM

Post #5 of 6 (1496 views)
Re: [miller] putting 500K rows into an Array? [In reply to] Can't Post

Thx. I've done a perl -V and got uselargefiles=undef
can I change that to define?

Thx


miller
User

Apr 19, 2011, 10:10 PM

Post #6 of 6 (1493 views)
Re: [Alf] putting 500K rows into an Array? [In reply to] Can't Post

Yes, but you'll have to recompile perl, and I don't know if the version you're using supports it.

I suggest that you download and install strawberry perl instead. I find it more user friendly and installing modules is a snap with cpan.

http://strawberryperl.com/

- Miller

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives