CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

 



digioleg54
User

May 9, 2017, 9:19 AM

Post #1 of 14 (3533 views)
I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? Can't Post

I have huge file 10 000 lines. The first is a header, then actual data, from which we need 6 fields. The file is pipe delimiter.
Is there any way to split it in n one action?
In shell we can do

Code
    $cmd=("awk -F'|' '{print \$($i_secid+1) \"\|\" \$($i_trdr+1) \"\|\" \$($i_encd+1) \"\|\" \$($i_market_price+1) \"\|\" \$($i_last_prcdt+1) \"\|\" \$($i_secnm+1)}' $if|sort -u");


Or I have to open file, read line by line and split each line?

Thanks


BillKSmith
Veteran

May 9, 2017, 11:29 AM

Post #2 of 14 (3520 views)
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

I think you are looking for the Command Switches '-a' and '-F'.
Good Luck,
Bill


Laurent_R
Veteran / Moderator

May 9, 2017, 11:12 PM

Post #3 of 14 (3514 views)
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

Bill has shown the options you're probably looking for (you'll also need the -e option of the command line), but I want to make just a couple of side comments.

First, 10,000 lines is really not huge. I'm dealing daily with files that have tens and even hundreds of millions of lines, sometimes even more.

Then, if your concern is the size of the file and performance, the fact that your shell/awk command line does not appear to loop explicitly on the file does not mean that it does something different under the hood. The awk part of your command does loop on every line of the file. Any program doing what you need has somehow to loop over the lines. So the fact that you don't see the loop does not mean that there is no loop and that it will be faster.


digioleg54
User

May 10, 2017, 5:58 AM

Post #4 of 14 (3510 views)
Re: [Laurent_R] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

So, your opinion, I have to open the file, read line by line in the loop, get the fields, I need, do something, then go to read another line?

Thanks


BillKSmith
Veteran

May 10, 2017, 7:38 AM

Post #5 of 14 (3501 views)
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

Please carefully read my previous reference! Study the example for -a. It uses -n and -e. That implies that you should also read about -n. Laurent already told you about -e. The text of that section refers your to -F (same as I did).

Either explain why this document does not apply to your problem or show us sample code using these features that does not do what you expect.
Good Luck,
Bill


digioleg54
User

May 10, 2017, 10:13 AM

Post #6 of 14 (3497 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

No, there is definitely loop, I just didn't put it

Code
 my @lns=`$cmd`; 
foreach (@lns) {
chop;
if (/ TBA/) {
$ln_tp=$_;
($secid,$trdr,$encd,$market_price,$last_prcdt,$secnm)=split(/\|/,$ln_tp);
($market_price_new,$last_prcdt_new,$lookup_i,$ln_o)=("","","","na");
if ($market_price=~/100\.[0]+$/ || $last_prcdt lt $old_prcdt) {
# print STDERR "STDERR:>>$secid,$trdr,$encd,$market_price,$last_prcdt\n";
if (($tp_sth->execute($secid,$trdr,$encd,$market_price,$last_prcdt)) && ($ln_o=$tp_sth->fetchrow_array())) {
if ($ln_o !~ /na/) {
($market_price_new,$lookup_i)=split(/\|/,$ln_o);
($market_price_new,$last_prcdt_new)=split(/\:/,$market_price_new);
# $last_prcdt_new=$last_prcdt_new.'*';
$ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm);
}
else {
($market_price_new,$last_prcdt_new)=($market_price,$last_prcdt);
# ($last_prcdt_new=~s/\s/0/g);
# $last_prcdt_new=$last_prcdt_new.'*';
$lookup_i='prc_trdr=""';
$ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm);
}
$_r_trade_price->{$secid}=join(":",$market_price_new,$last_prcdt_new,$lookup_i);
}
}
# print STDERR "$ln_tp|$lookup_i\n";
$rec_cnt++;
}
}


thanks


digioleg54
User

May 10, 2017, 10:56 AM

Post #7 of 14 (3492 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

I.e. you want to say that line

Code
my @lns= perl -ane print pop ($if) "\n" 
foreach (@lns){
....
}

give me correct result? But I don't nee each value. I need six of them. How can I show the delimiter and each of 6 fields?


digioleg54
User

May 10, 2017, 11:33 AM

Post #8 of 14 (3490 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

Mr. Smith. You gave me command line. I do it in Perl script. What should I put instead

Code
perl

in front of command line?


BillKSmith
Veteran

May 10, 2017, 9:14 PM

Post #9 of 14 (3470 views)
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

No, I did not give you a command line. I gave you a reference to the documentation on "command switches". Read it again! It tells you where you can put them in a script.

Look at the example that I referred to before. Do you see the loop now?

The new code you have posted is so much different from that example that I cannot comment on what is wrong. How can I help you to learn perl if you are unwilling or unable to read the applicable documents?
Good Luck,
Bill


digioleg54
User

May 11, 2017, 4:18 AM

Post #10 of 14 (3463 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

I read everything. There are only lines,which start from Perl -e .....
Where did you see the loop?


BillKSmith
Veteran

May 11, 2017, 6:47 AM

Post #11 of 14 (3458 views)
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post


Quote
1. perl -ane 'print pop(@F), "\n";'


is equivalent to
1. while (<>) {
2. @F = split(' ');
3. print pop(@F), "\n";
4. }

Emphasis added
Good Luck,
Bill


digioleg54
User

May 11, 2017, 7:17 AM

Post #12 of 14 (3457 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

That is exactly what I asked before, do I need loop or not


BillKSmith
Veteran

May 11, 2017, 8:49 AM

Post #13 of 14 (3452 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

The best answer I can give is that you clearly need a loop over your data. You can code it explicitly or you can set the -n flag and let perl do it for you. Likewise with the split. You must split each line on pipe characters. You can code that yourself or you can set the -a (and -F) flag and let perl do it for you. Note: -F only works if you also use -a. and -a only works if you use -n.

The switches do not give any additional capability. They are intended to make the routine part easier. As long as you do see that the example is a template for your code, Do not use switches! Write your own I/O, loop, and split.
Good Luck,
Bill


digioleg54
User

May 11, 2017, 1:37 PM

Post #14 of 14 (3439 views)
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line? [In reply to] Can't Post

Thank you

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives