Perl Guru Forums: Perl Programming Help: Intermediate: putting 500K rows into an Array?

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
putting 500K rows into an Array?

Alf
New User

Apr 18, 2011, 11:46 PM

Post #1 of 6 (1247 views)

putting 500K rows into an Array?	Can't Post

Hi Guys,

I'm having problem trying to load a file into an array.

The file is more than 500,000 lines and I am trying to load it using the following:

open(FILE,"filename.txt");

binmode(FILE);

@FileData = <FILE>;

close(FILE)

This works fine for files up to 480K lines, no problem. But once the file is above 500K then it just stops

and never get to the next line of code in the perl script. I've been using this few lines for many years now

and this is the 1st time I c this happening. I put some debug lines in to just simply print a message in between

every line of code and found it reached the line @FileData = <FILE>;

and then that's it.

I am on ActivePerl v5.6.1 built MSWin32-x86-multi-thread (I know, it's old) and have 3.8Gb ram (with 1.9Gb free

at point of execution of the code). I didn't observe any major RAM drain during the runs.

Is there a Limit in Perl Array size? I can't ever remember reading a limit from any where.

Help please?

Thx a million

rovf
Veteran

Apr 19, 2011, 5:36 AM

Post #2 of 6 (1239 views)

Re: [Alf] putting 500K rows into an Array? [In reply to]	Can't Post

I don't know about Perl 5.6, but at least in 5.10, there is no 500K limit on the array size. Whether this applies to your version is something you can easily test yourself from the command line:

perl -lwe 'my @a; for(my $i=0; $i<600000; ++$i) {$a[$i]=$i}; print "ended"'

Depending on the size of your lines, of course it might be a space problem, in particular since you are running it on Windows. Do you really have to slurp that huge file into memory? Maybe Tie::File would be an alternative.

Alf
New User

Apr 19, 2011, 3:59 PM

Post #3 of 6 (1223 views)

Re: [rovf] putting 500K rows into an Array? [In reply to]	Can't Post

Hi rovf,

thx for the reply. I've subsequently put your line into a perl script file and it ran fine. And then I cut out those few lines from my main program and into a separate clean perl script and that ran fine too. So it's not the array size that's causing the problem. It's something else. I'll continue my investigation. Thanks for your help.

regards.

miller
User

Apr 19, 2011, 5:00 PM

Post #4 of 6 (1222 views)

Re: [Alf] putting 500K rows into an Array? [In reply to]	Can't Post

I'm guessing that the problem is not the size of your array, but of the size of your file. Unless your perl is compiled with uselargefiles=define, then it won't be able to handle files over 2gigs in size.

Just do a perl -V and see if you have that capability. If you don't then you'll either have to upgrade your perl or use split to split the file into managable bites for your version of perl.

- Miller

Alf
New User

Apr 19, 2011, 10:06 PM

Post #5 of 6 (1214 views)

Re: [miller] putting 500K rows into an Array? [In reply to]	Can't Post

Thx. I've done a perl -V and got uselargefiles=undef
can I change that to define?

Thx

miller
User

Apr 19, 2011, 10:10 PM

Post #6 of 6 (1211 views)

Re: [Alf] putting 500K rows into an Array? [In reply to]	Can't Post

Yes, but you'll have to recompile perl, and I don't know if the version you're using supports it.

I suggest that you download and install strawberry perl instead. I find it more user friendly and installing modules is a snap with cpan.

http://strawberryperl.com/

- Miller

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives