Perl Guru Forums: Perl Programming Help: Beginner: I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Beginner:
I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

digioleg54
User

May 9, 2017, 9:19 AM

Post #1 of 14 (4465 views)

I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?	Can't Post

I have huge file 10 000 lines. The first is a header, then actual data, from which we need 6 fields. The file is pipe delimiter.
Is there any way to split it in n one action?
In shell we can do

Code

    $cmd=("awk -F'|' '{print \$($i_secid+1) \"\|\" \$($i_trdr+1) \"\|\" \$($i_encd+1) \"\|\" \$($i_market_price+1) \"\|\" \$($i_last_prcdt+1) \"\|\" \$($i_secnm+1)}' $if|sort -u");

Or I have to open file, read line by line and split each line?

Thanks

BillKSmith
Veteran

May 9, 2017, 11:29 AM

Post #2 of 14 (4452 views)

Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

I think you are looking for the Command Switches '-a' and '-F'.
Good Luck,
Bill

Laurent_R
Veteran / Moderator

May 9, 2017, 11:12 PM

Post #3 of 14 (4446 views)

Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

Bill has shown the options you're probably looking for (you'll also need the -e option of the command line), but I want to make just a couple of side comments.

First, 10,000 lines is really not huge. I'm dealing daily with files that have tens and even hundreds of millions of lines, sometimes even more.

Then, if your concern is the size of the file and performance, the fact that your shell/awk command line does not appear to loop explicitly on the file does not mean that it does something different under the hood. The awk part of your command does loop on every line of the file. Any program doing what you need has somehow to loop over the lines. So the fact that you don't see the loop does not mean that there is no loop and that it will be faster.

digioleg54
User

May 10, 2017, 5:58 AM

Post #4 of 14 (4442 views)

Re: [Laurent_R] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

So, your opinion, I have to open the file, read line by line in the loop, get the fields, I need, do something, then go to read another line?

Thanks

BillKSmith
Veteran

May 10, 2017, 7:38 AM

Post #5 of 14 (4433 views)

Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

Please carefully read my previous reference! Study the example for -a. It uses -n and -e. That implies that you should also read about -n. Laurent already told you about -e. The text of that section refers your to -F (same as I did).

Either explain why this document does not apply to your problem or show us sample code using these features that does not do what you expect.
Good Luck,
Bill

digioleg54
User

May 10, 2017, 10:13 AM

Post #6 of 14 (4429 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

No, there is definitely loop, I just didn't put it

Code

 my @lns=`$cmd`; 
    foreach (@lns) { 
        chop; 
        if (/ TBA/) { 
            $ln_tp=$_; 
            ($secid,$trdr,$encd,$market_price,$last_prcdt,$secnm)=split(/\|/,$ln_tp); 
            ($market_price_new,$last_prcdt_new,$lookup_i,$ln_o)=("","","","na"); 
            if ($market_price=~/100\.[0]+$/ || $last_prcdt lt $old_prcdt) { 
		# print STDERR "STDERR:>>$secid,$trdr,$encd,$market_price,$last_prcdt\n"; 
                if (($tp_sth->execute($secid,$trdr,$encd,$market_price,$last_prcdt)) && ($ln_o=$tp_sth->fetchrow_array())) { 
                    if ($ln_o !~ /na/) { 
                        ($market_price_new,$lookup_i)=split(/\|/,$ln_o); 
                        ($market_price_new,$last_prcdt_new)=split(/\:/,$market_price_new); 
                        # $last_prcdt_new=$last_prcdt_new.'*'; 
                        $ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm); 
                    } 
                    else { 
                        ($market_price_new,$last_prcdt_new)=($market_price,$last_prcdt); 
                        # ($last_prcdt_new=~s/\s/0/g); 
                        # $last_prcdt_new=$last_prcdt_new.'*'; 
                        $lookup_i='prc_trdr=""'; 
                        $ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm); 
                    } 
                    $_r_trade_price->{$secid}=join(":",$market_price_new,$last_prcdt_new,$lookup_i); 
                } 
            } 
            # print STDERR "$ln_tp|$lookup_i\n"; 
            $rec_cnt++; 
        } 
    }

thanks

digioleg54
User

May 10, 2017, 10:56 AM

Post #7 of 14 (4424 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

I.e. you want to say that line

Code

my @lns= perl -ane print pop ($if) "\n" 
foreach (@lns){ 
.... 
}

give me correct result? But I don't nee each value. I need six of them. How can I show the delimiter and each of 6 fields?

digioleg54
User

May 10, 2017, 11:33 AM

Post #8 of 14 (4422 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

Mr. Smith. You gave me command line. I do it in Perl script. What should I put instead

Code

perl

in front of command line?

BillKSmith
Veteran

May 10, 2017, 9:14 PM

Post #9 of 14 (4402 views)

Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

No, I did not give you a command line. I gave you a reference to the documentation on "command switches". Read it again! It tells you where you can put them in a script.

Look at the example that I referred to before. Do you see the loop now?

The new code you have posted is so much different from that example that I cannot comment on what is wrong. How can I help you to learn perl if you are unwilling or unable to read the applicable documents?
Good Luck,
Bill

digioleg54
User

May 11, 2017, 4:18 AM

Post #10 of 14 (4395 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

I read everything. There are only lines,which start from Perl -e .....
Where did you see the loop?

BillKSmith
Veteran

May 11, 2017, 6:47 AM

Post #11 of 14 (4390 views)

Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

Quote

1. perl -ane 'print pop(@F), "\n";'

is equivalent to
1. while (<>) {
2. @F = split(' ');
3. print pop(@F), "\n";
4. }

Emphasis added
Good Luck,
Bill

digioleg54
User

May 11, 2017, 7:17 AM

Post #12 of 14 (4389 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

That is exactly what I asked before, do I need loop or not

BillKSmith
Veteran

May 11, 2017, 8:49 AM

Post #13 of 14 (4384 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

The best answer I can give is that you clearly need a loop over your data. You can code it explicitly or you can set the -n flag and let perl do it for you. Likewise with the split. You must split each line on pipe characters. You can code that yourself or you can set the -a (and -F) flag and let perl do it for you. Note: -F only works if you also use -a. and -a only works if you use -n.

The switches do not give any additional capability. They are intended to make the routine part easier. As long as you do see that the example is a template for your code, Do not use switches! Write your own I/O, loop, and split.
Good Luck,
Bill

digioleg54
User

May 11, 2017, 1:37 PM

Post #14 of 14 (4371 views)

Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to]	Can't Post

Thank you

Search for (options)

Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives