CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
Perl is skipping instruction...

 



ampentium2
Novice

Apr 1, 2009, 10:56 PM

Post #1 of 21 (2319 views)
Perl is skipping instruction... Can't Post

Hi, sorry if such topic is already available... search through but didn't found any.

anyway, i'm trying to load a long list (500k lines of text) of text file into an array. Since this will take some time, i would like to tell the user that the list is loading and hopefully they can wait. so, i make a script that looks like this...


Code
print "The system is loading"; 

open (F,"test2.txt") || die "Cant";
@d=<F>;

print "The system is complete loading";


the problem here, the perl executed the open() instruction first despite of the print instruction being at first. any idea why is this happen and is there anyway to do this?


KevinR
Veteran


Apr 1, 2009, 11:21 PM

Post #2 of 21 (2316 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

I am really not sure what you are trying to do, but.....


Code
print "The system is loading, please wait\n"; 
sleep(1);

open (F,"test2.txt") or die "Cant open file $!";
@d=<F>;
close F;

print "The system is complete loading";

-------------------------------------------------


ampentium2
Novice

Apr 1, 2009, 11:29 PM

Post #3 of 21 (2314 views)
Re: [KevinR] Perl is skipping instruction... [In reply to] Can't Post

tried, but to no avail the open() instruction is still doing it's best to be the first...

what i am trying to do here is, before the list is loaded into the array, i would like to prompt the user to wait... and then load the file....

the flow would be much like this.

Start > Tell the user the load is in progress > load the list into array > tell the user it's done

but what happen is

Start > load the list into array > tell the user the load is in progress > tell the user it's done

it didn't print when the print is instructed, instead wait till open() is done first...


KevinR
Veteran


Apr 1, 2009, 11:32 PM

Post #4 of 21 (2313 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

Try this:


Code
$| = 1; 
print "The system is loading, please wait\n";

open (F,"test2.txt") or die "Cant open file $!";
@d=<F>;
close F;

print "The system is complete loading";

-------------------------------------------------


ampentium2
Novice

Apr 1, 2009, 11:35 PM

Post #5 of 21 (2311 views)
Re: [KevinR] Perl is skipping instruction... [In reply to] Can't Post

it works! thanks...

what actually happen?


Tumata
User


Apr 2, 2009, 12:20 AM

Post #6 of 21 (2305 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

Your problem was caused by output buffering. The aim of such buffering is to increase perfomance as write is quite expensive operation. But setting $| to 1 disables output buffering.


FishMonger
Veteran / Moderator

Apr 2, 2009, 5:29 AM

Post #7 of 21 (2300 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

Slurping a large file into an array is almost always (99.5% of the time) the wrong approach. It's better to process a file in chunks (either by line or paragraph).


ampentium2
Novice

Apr 2, 2009, 9:28 AM

Post #8 of 21 (2294 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post

I see...

since it's not in paragraphed format, what should i do? multiple array perhaps? the only thing i could think of is that... i knew it shouldn't be slurped into one array, especially now with the buffer is disabled...

still new in data processing... especially big one...


FishMonger
Veteran / Moderator

Apr 2, 2009, 9:33 AM

Post #9 of 21 (2292 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

Post your script and include:

1) what it is supposed to do

2) what is it doing that it shouldn't

3) what is it not doing that it should

4) any/all warnings and error that it generates

Is this a class homework assignment?


ampentium2
Novice

Apr 2, 2009, 9:48 AM

Post #10 of 21 (2290 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post

i'm afraid i couldnt... it's a job assigned to me... plus, the script is not in my laptop, but in my computer at the office. and i'm at home... so, i'll see what i can do when i'm at the office later...

but so far, the only problem is what i've said in the start of the thread... and it's solved by that $| which i still think a bad idea to leave that the way it is... yeah, it'll work since i'm only processing little chunk of data... wait till there's numerous data i need to catch... other than that, everything's fine... it does what it should, and it doesn't do what it shouldn't...

what it should do is, it fetch a log from a file... and read it through... after that, i would fetch important things from the log and interpret it so a report can be easily understand...

what i did for now is, open the file, load the log into an array... scan it line by line using scalar ($array[$x]) and by using string matching, find the data that is similar to the keyword used...


KevinR
Veteran


Apr 2, 2009, 10:31 AM

Post #11 of 21 (2287 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

I think if you reset $| to 0 it turns on buffering again:

$|=0;

maybe Fish knows for sure
-------------------------------------------------


FishMonger
Veteran / Moderator

Apr 2, 2009, 10:40 AM

Post #12 of 21 (2285 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

There is no problem with using $| to dynamically disable/enable buffering. It is often done.

The real problem is with loading the entire file into an array just so you can loop through that array; that's very inefficient because you're looping ofver your data twice, once to load the array and again to process the array. That's not a big issue as long as the file is small, but as you work with larger files, you'll see a major drop in efficiency.

If you really want to access it as an array, especially if it's a large file, then do it efficiently by using Tie::File.
http://search.cpan.org/~mjd/Tie-File-0.96/lib/Tie/File.pm


FishMonger
Veteran / Moderator

Apr 2, 2009, 10:42 AM

Post #13 of 21 (2284 views)
Re: [KevinR] Perl is skipping instruction... [In reply to] Can't Post

$| is evaluated in booleon context, so yes resetting it to 0 enables buffering.


KevinR
Veteran


Apr 2, 2009, 2:15 PM

Post #14 of 21 (2279 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post


In Reply To
There is no problem with using $| to dynamically disable/enable buffering. It is often done.

The real problem is with loading the entire file into an array just so you can loop through that array; that's very inefficient because you're looping ofver your data twice, once to load the array and again to process the array. That's not a big issue as long as the file is small, but as you work with larger files, you'll see a major drop in efficiency.

If you really want to access it as an array, especially if it's a large file, then do it efficiently by using Tie::File.
http://search.cpan.org/~mjd/Tie-File-0.96/lib/Tie/File.pm


My experience with Tie::File is that it certainly saves on memory but can be very slow in some circumstances. There is also a bug report on CPAN that version .96 does not work with perl 5.10 but there is a patch to download.
-------------------------------------------------


FishMonger
Veteran / Moderator

Apr 2, 2009, 3:31 PM

Post #15 of 21 (2278 views)
Re: [KevinR] Perl is skipping instruction... [In reply to] Can't Post

Tie::File can be slow if you're modifying each row and forgot to enable defer writing, which means that it saves the file to disk each and every time you modify an element. Enabling defer will tell it to wait on the disk write until you issue a flush. Now, that also means that it will use more memory resources. How much more? I'm not sure; I guess it depends on the size of the file and the extent of the edits.


ampentium2
Novice

Apr 5, 2009, 8:41 PM

Post #16 of 21 (2245 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post

thanks all. Tie::File is an external module right? it might not be a good idea when this script is going to be used by lot of people after this. then whoever need to run this script need to install the module on the first time they run the script on their machine... and this script will not stay at one machine only.

i'll stick with the array at the moment until i find another better solution.


FishMonger
Veteran / Moderator

Apr 5, 2009, 9:00 PM

Post #17 of 21 (2244 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

Tie::File is a core module and comes with the base install of Perl. So, there should be no reason for you to worry.


ampentium2
Novice

Apr 5, 2009, 9:38 PM

Post #18 of 21 (2236 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post

Ouh... I read somewhere it says something like being an external module... guess, i might get my info wrong then...


FishMonger
Veteran / Moderator

Apr 5, 2009, 9:49 PM

Post #19 of 21 (2235 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

It is an external module, but it's a core module that is included with your perl installation.


ampentium2
Novice

Apr 5, 2009, 11:55 PM

Post #20 of 21 (2229 views)
Re: [FishMonger] Perl is skipping instruction... [In reply to] Can't Post

i'd try the Tie::File... what happen is, instead of spiking 50% of my cpu, my cpu is under constant stress of 80%-90% for core#1 and 20%-30% for core#2 as long as the script is scanning...

i would say, they're not efficient at all if it is in this case... just like what kevin said, it turns out to take longer time to complete...


FishMonger
Veteran / Moderator

Apr 6, 2009, 7:59 AM

Post #21 of 21 (2220 views)
Re: [ampentium2] Perl is skipping instruction... [In reply to] Can't Post

The advantage of Tie::File shows up when you're working with a huge file that can't fit into RAM and needs to be accessed as an array. The file you're working is not that big. In this case, you're better off using a while loop on the filehandle instead of slurping.

Here's a benchmark test on a 32MB 584747 line file that shows the while loop to be almost 500% faster than slurping.

First, the contents of the test script.

Code
[root@fc4dev ~]# cat test.pl 
#!/usr/bin/perl

use strict;
use warnings;
use Benchmark qw(cmpthese) ;

my $file = 'data.txt';

cmpthese( -10, {
'while_file' => \&while_file,
'slurp_file'=> \&slurp_file
});


sub while_file {
my $cnt;
open my $FH, $file or die $!;
while ( <$FH> ) { $cnt++ }
close $FH;
}

sub slurp_file {
my $cnt;
open my $FH, $file or die $!;
my @array = <$FH>;
$cnt++ for @array;
close $FH;
}


Now the output of the benchmark

Code
[root@fc4dev ~]# ./test.pl 
s/iter slurp_file while_file
slurp_file 4.79 -- -82%
while_file 0.838 471% --


 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives