Home: Perl Programming Help: Beginner:
I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?



digioleg54
User

May 9, 2017, 9:19 AM


Views: 6322
I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

I have huge file 10 000 lines. The first is a header, then actual data, from which we need 6 fields. The file is pipe delimiter.
Is there any way to split it in n one action?
In shell we can do

Code
    $cmd=("awk -F'|' '{print \$($i_secid+1) \"\|\" \$($i_trdr+1) \"\|\" \$($i_encd+1) \"\|\" \$($i_market_price+1) \"\|\" \$($i_last_prcdt+1) \"\|\" \$($i_secnm+1)}' $if|sort -u");


Or I have to open file, read line by line and split each line?

Thanks


BillKSmith
Veteran

May 9, 2017, 11:29 AM


Views: 6309
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

I think you are looking for the Command Switches '-a' and '-F'.
Good Luck,
Bill


Laurent_R
Veteran / Moderator

May 9, 2017, 11:12 PM


Views: 6303
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

Bill has shown the options you're probably looking for (you'll also need the -e option of the command line), but I want to make just a couple of side comments.

First, 10,000 lines is really not huge. I'm dealing daily with files that have tens and even hundreds of millions of lines, sometimes even more.

Then, if your concern is the size of the file and performance, the fact that your shell/awk command line does not appear to loop explicitly on the file does not mean that it does something different under the hood. The awk part of your command does loop on every line of the file. Any program doing what you need has somehow to loop over the lines. So the fact that you don't see the loop does not mean that there is no loop and that it will be faster.


digioleg54
User

May 10, 2017, 5:58 AM


Views: 6299
Re: [Laurent_R] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

So, your opinion, I have to open the file, read line by line in the loop, get the fields, I need, do something, then go to read another line?

Thanks


BillKSmith
Veteran

May 10, 2017, 7:38 AM


Views: 6290
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

Please carefully read my previous reference! Study the example for -a. It uses -n and -e. That implies that you should also read about -n. Laurent already told you about -e. The text of that section refers your to -F (same as I did).

Either explain why this document does not apply to your problem or show us sample code using these features that does not do what you expect.
Good Luck,
Bill


digioleg54
User

May 10, 2017, 10:13 AM


Views: 6286
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

No, there is definitely loop, I just didn't put it

Code
 my @lns=`$cmd`; 
foreach (@lns) {
chop;
if (/ TBA/) {
$ln_tp=$_;
($secid,$trdr,$encd,$market_price,$last_prcdt,$secnm)=split(/\|/,$ln_tp);
($market_price_new,$last_prcdt_new,$lookup_i,$ln_o)=("","","","na");
if ($market_price=~/100\.[0]+$/ || $last_prcdt lt $old_prcdt) {
# print STDERR "STDERR:>>$secid,$trdr,$encd,$market_price,$last_prcdt\n";
if (($tp_sth->execute($secid,$trdr,$encd,$market_price,$last_prcdt)) && ($ln_o=$tp_sth->fetchrow_array())) {
if ($ln_o !~ /na/) {
($market_price_new,$lookup_i)=split(/\|/,$ln_o);
($market_price_new,$last_prcdt_new)=split(/\:/,$market_price_new);
# $last_prcdt_new=$last_prcdt_new.'*';
$ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm);
}
else {
($market_price_new,$last_prcdt_new)=($market_price,$last_prcdt);
# ($last_prcdt_new=~s/\s/0/g);
# $last_prcdt_new=$last_prcdt_new.'*';
$lookup_i='prc_trdr=""';
$ln_tp=join("|",$secid,$trdr,$encd,$market_price_new,$last_prcdt_new,$secnm);
}
$_r_trade_price->{$secid}=join(":",$market_price_new,$last_prcdt_new,$lookup_i);
}
}
# print STDERR "$ln_tp|$lookup_i\n";
$rec_cnt++;
}
}


thanks


digioleg54
User

May 10, 2017, 10:56 AM


Views: 6281
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

I.e. you want to say that line

Code
my @lns= perl -ane print pop ($if) "\n" 
foreach (@lns){
....
}

give me correct result? But I don't nee each value. I need six of them. How can I show the delimiter and each of 6 fields?


digioleg54
User

May 10, 2017, 11:33 AM


Views: 6279
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

Mr. Smith. You gave me command line. I do it in Perl script. What should I put instead

Code
perl

in front of command line?


BillKSmith
Veteran

May 10, 2017, 9:14 PM


Views: 6259
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

No, I did not give you a command line. I gave you a reference to the documentation on "command switches". Read it again! It tells you where you can put them in a script.

Look at the example that I referred to before. Do you see the loop now?

The new code you have posted is so much different from that example that I cannot comment on what is wrong. How can I help you to learn perl if you are unwilling or unable to read the applicable documents?
Good Luck,
Bill


digioleg54
User

May 11, 2017, 4:18 AM


Views: 6252
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

I read everything. There are only lines,which start from Perl -e .....
Where did you see the loop?


BillKSmith
Veteran

May 11, 2017, 6:47 AM


Views: 6247
Re: [digioleg54] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?


Quote
1. perl -ane 'print pop(@F), "\n";'


is equivalent to
1. while (<>) {
2. @F = split(' ');
3. print pop(@F), "\n";
4. }

Emphasis added
Good Luck,
Bill


digioleg54
User

May 11, 2017, 7:17 AM


Views: 6246
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

That is exactly what I asked before, do I need loop or not


BillKSmith
Veteran

May 11, 2017, 8:49 AM


Views: 6241
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

The best answer I can give is that you clearly need a loop over your data. You can code it explicitly or you can set the -n flag and let perl do it for you. Likewise with the split. You must split each line on pipe characters. You can code that yourself or you can set the -a (and -F) flag and let perl do it for you. Note: -F only works if you also use -a. and -a only works if you use -n.

The switches do not give any additional capability. They are intended to make the routine part easier. As long as you do see that the example is a template for your code, Do not use switches! Write your own I/O, loop, and split.
Good Luck,
Bill


digioleg54
User

May 11, 2017, 1:37 PM


Views: 6228
Re: [BillKSmith] I have to split huge file in Perl. Can it be done in one action or need the loop and read line by line?

Thank you