CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Sorting

 



sohnaeo
Novice

Jul 16, 2006, 6:26 PM

Post #1 of 14 (3113 views)
Sorting Can't Post

Hello Guys,

I have comma separated CSV file. And there are 5 columns date,delta time, bits, bitsin, bitsout.

Task is

1) sort the file by date,

2) ignore the bits

3) compare the bitsin and bitsout and keep the highest one

4) sort the higest one,

5) print all above four steps comma separated.

I have done first 3 steps and i dont know how i can perform step 4 any body can help me



Raw Data File Contents

12/6/2006 00:02:23,301,1151.78735352,677.26245117,474.52490234
12/6/2006 00:07:22,299,1108.97656250,641.49835205,467.47827148
12/6/2006 00:12:22,300,1020.15997314,590.40002441,429.76000977
12/6/2006 00:17:23,301,981.26245117,562.01995850,419.24252319

OutPut File

1/6/2006 00:03:46 300,641.06665039,477.60000610,641.06665039
1/6/2006 00:08:46 300,563.78668213,425.60000610,563.78668213
1/6/2006 00:13:47 301,505.99334717,430.72424316,505.99334717

Wanting Output

1/6/2006 00:03:46 300,641.06665039,477.60000610,641.06665039 ,505.99334717
1/6/2006 00:08:46 300,563.78668213,425.60000610,563.78668213 ,563.78668213
1/6/2006 00:13:47 301,505.99334717,430.72424316,505.99334717 ,641.06665039



open(RAW,"raw_data") or die ("could not open the file $!");<br>

while(<RAW>) { <br>

chomp; <br>

next if /^(\s)*$/; <br>

my @this_record = split(/,/,$_);<br>

push(@records,\@this_record);<br>

}<br>

my @sorted = sort{$a->[0] <=> $b->[0]} @records; <br>

foreach $record (@sorted) { <br>

if($record->[3] >= $record->[4]) { <br>

$final = $record->[3];<br>

} <br>

else { <br>

$final = $record->[4]; <br>

} <br>

print "$record->[0] $record->[1],$record->[3],$record->[4],$final\n"; <br>

} <br>

close(RAW); <br>
<br>


jim_lad
Novice

Jul 18, 2006, 2:26 AM

Post #2 of 14 (3101 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

Can you not just use another a<=>b sorter on that data?


sohnaeo
Novice

Jul 18, 2006, 4:45 AM

Post #3 of 14 (3097 views)
Re: [jim_lad] Sorting [In reply to] Can't Post

Thanks for reply. If you check my code I want to sort $final .For this I need to push $final in array and then sort but it is in the While loop. If I

sort it outside the loop then how i can get the previous array values.

check the code again

open(RAW,"raw_data") or die ("could not open the file $!");

while(<RAW>) {

chomp;

next if /^(\s)*$/;

my @this_record = split(/,/,$_); //date,delta,bits,bitsin,bitsout

push(@records,\@this_record);

}

my @sorted = sort{$a->[0] <=> $b->[0]} @records; //sort date

foreach $record (@sorted) {

if($record->[3] >= $record->[4]) { //compare bitsin and bitsout and keep the highes one

$final = $record->[3];

}

else {

$final = $record->[4]; //compare bitsin and bitsout and keep the highes one

}

###Want to sort $final but it is in the loop

print "$record->[0] $record->[1],$record->[3],$record->[4],$final\n";

//print sorted date, delta, bitsin, bitsout, wanting to print sorted highest bitsin or bitsout
}

close(RAW);


(This post was edited by sohnaeo on Jul 18, 2006, 4:50 AM)


jim_lad
Novice

Jul 18, 2006, 6:30 AM

Post #4 of 14 (3093 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

Do the sorter as a subroutinue and call it?


KevinR
Veteran


Jul 18, 2006, 2:11 PM

Post #5 of 14 (3090 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

first, this is not going to sort the lines by date:

my @sorted = sort{$a->[0] <=> $b->[0]} @records;

you might be fooled because the data is already in order by date.
-------------------------------------------------


(This post was edited by KevinR on Jul 18, 2006, 2:17 PM)


KevinR
Veteran


Jul 18, 2006, 2:20 PM

Post #6 of 14 (3088 views)
Re: [KevinR] Sorting [In reply to] Can't Post

the dates in the raw data don;t correspond to the dates in the sorted data:

Raw Data File Contents

12/6/2006 00:02:23,301,1151.78735352,677.26245117,474.52490234
12/6/2006 00:07:22,299,1108.97656250,641.49835205,467.47827148
12/6/2006 00:12:22,300,1020.15997314,590.40002441,429.76000977
12/6/2006 00:17:23,301,981.26245117,562.01995850,419.24252319

OutPut File

1/6/2006 00:03:46 300,641.06665039,477.60000610,641.06665039
1/6/2006 00:08:46 300,563.78668213,425.60000610,563.78668213
1/6/2006 00:13:47 301,505.99334717,430.72424316,505.99334717


but I think that can be ignored for now as sorting the dates will be a matter of converting them into epoch time (or some other method) to sort them properly.
-------------------------------------------------


KevinR
Veteran


Jul 18, 2006, 2:42 PM

Post #7 of 14 (3087 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

I can't understand what step 3 and 4 are supposed to do:

3) compare the bitsin and bitsout and keep the highest one

4) sort the higest one,

can you explain it better?
-------------------------------------------------


sohnaeo
Novice

Jul 18, 2006, 5:02 PM

Post #8 of 14 (3085 views)
Re: [KevinR] Sorting [In reply to] Can't Post

Hello Kevin,

Thanks for reply. I am trying to calculate 95th percentile

To compute the 95th percentile value on a data set, sort the data set by value from higest to lowest, discard the highest 5% of the sorted samples, and the next highest sample becomes the 95th percentile value for the data set.

For example

12/6/2006 00:02:23,301,677.26245117,474.52490234

Date, POLL time means 301 seconds 5 minutes,bitsin, bitsout,

So for 95th percentile calculation you have to keep bitsin, bits out and the highest one between them.

So the output will be like

1/6/2006 00:03:46 ,300,641.067,477.600,641.067
1/6/2006 00:08:46 ,300,563.787,425.600,563.787
1/6/2006 00:13:47 ,301,505.993,430.724,505.993

Date,delta,bitsin,bitsout,highest one.

And if you check my code there is if clause which is comparing bitsin and bitsout and assign highest one to variable $final.

Now I want to sort $final. How can i do that.

So the ideal output should be like as follow

1/6/2006 00:03:46 ,300,641.067,477.600,641.067,505.993
1/6/2006 00:08:46 ,300,563.787,425.600,563.787,563.787
1/6/2006 00:13:47 ,301,505.993,430.724,505.993,641.067

Date,Delta,bitsin,bitsout,Highest one, Sorted highest one

One more thing is that my raw data file is not sorted already. I am soring by date.

If still unclear please let me know I can explain




(This post was edited by sohnaeo on Jul 18, 2006, 10:02 PM)


KevinR
Veteran


Jul 19, 2006, 8:48 AM

Post #9 of 14 (3064 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

let me address the date sorting issue.


Code
my @records = (); 
while(<DATA>) {
chomp;
next if (/^\s*$/);
my @this_record = split(/,/,$_);
push(@records,\@this_record);
}
my @sorted = sort{$a->[0] <=> $b->[0]} @records;
print "$_\n" for @sorted;

__DATA__
12/6/2006 00:02:23,301,1151.78735352,677.26245117,474.52490234
12/6/2006 00:17:23,301,981.26245117,562.01995850,419.24252319
12/6/2006 00:07:22,299,1108.97656250,641.49835205,467.47827148
12/6/2006 00:12:22,300,1020.15997314,590.40002441,429.76000977

the above code using the data prints:


Code
12/6/2006 00:02:23 301 1151.78735352 677.26245117 474.52490234 
12/6/2006 00:17:23 301 981.26245117 562.01995850 419.24252319
12/6/2006 00:07:22 299 1108.97656250 641.49835205 467.47827148
12/6/2006 00:12:22 300 1020.15997314 590.40002441 429.76000977


checking the error log:

..... Argument "12/6/2006 00:17:23" isn't numeric in numeric comparison (<=>) at .....

as for the rest of what you are trying to do I do not understand your explanation.
-------------------------------------------------


(This post was edited by KevinR on Jul 19, 2006, 8:52 AM)


sohnaeo
Novice

Jul 20, 2006, 3:40 AM

Post #10 of 14 (3055 views)
Re: [KevinR] Sorting [In reply to] Can't Post

Hello Kevin,

I have sorted out date sorting problem. lets forget about percentile. Suppose you have one file and this file contains following

1,11

2,12

3,1

4,13

5,14

6,15

7,16

8,6

9,17

10,5

Now I want to compare 1st field and 2nd field and then write highest value in 3rd column like as follow

1,11,11

2,12,12

3,1,3

4,13,13

5,14,14

6,15,15

7,16,16

8,6,8

9,17,17

10,5,10

And then sort the 3rd column and write output in 4th column as

1,11,11,3

2,12,12,8

3,1,3,10

4,13,13,11

5,14,14,12

6,15,15,13

7,16,16,14

8,6,8,15

9,17,17,16

10,5,10,17

Now I think it should make sense to you. how can you do that. I will be thankful.



Regards,

Farhan


KevinR
Veteran


Jul 20, 2006, 11:31 AM

Post #11 of 14 (3049 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post

one way:


Code
use strict; 
use Data::Dumper;
my @AoA = ();
while(<DATA>){
chomp;
my($var1,$var2)=split(/,/);
push @AoA,[$var1,$var2,($var1>$var2?$var1:$var2)];
}
my @sorted_fourth = sort{$a->[2] <=> $b->[2]} @AoA;
for my $i (0..$#sorted_fourth){
$AoA[$i][3]=$sorted_fourth[$i][2];
}
print Dumper(@AoA);
__DATA__
1,11
2,12
3,1
4,13
5,14
6,15
7,16
8,6
9,17
10,5

-------------------------------------------------


(This post was edited by KevinR on Jul 20, 2006, 11:37 AM)


sohnaeo
Novice

Jul 20, 2006, 10:29 PM

Post #12 of 14 (3039 views)
Re: [KevinR] Sorting [In reply to] Can't Post

Bundle of thanks, Can you help me little bit more. It is printing like

$VAR9 = [
9,
17,
17,
16
];
$VAR10 = [
10,
5,
10,
17
];

But I want to print like 10,5,10,17. BTW what does Dumper do?

And I could not understand this line

push @AoA,[$var1,$var2,($var1>$var2?$var1:$var2)];

Could you explain what is ($var1>$var2?$var1:$var2)

Thanks,

farhan


(This post was edited by sohnaeo on Jul 20, 2006, 10:30 PM)


KevinR
Veteran


Jul 21, 2006, 12:53 AM

Post #13 of 14 (3035 views)
Re: [sohnaeo] Sorting [In reply to] Can't Post


Quote
BTW what does Dumper do?


the Data::Dumper module (a core module) is a convenient way to print out complex data or any data for that matter. It's a good tool to use for checking data structures when writing or debugging code.


Quote
Could you explain what is ($var1>$var2?$var1:$var2)


this is the ternary operator. I used it to determine which value to put into the third column (the one that is greatest in value). It reads like this:

condition ? true part : false part

if the condition part is true the true part is the return value, otherwise the false part is the return value, it's essentially the same as:


Code
if ($var1 > $var2) { 
$var3 = $var1;
}
else {
$var3 = $var2;
}



Quote
But I want to print like 10,5,10,17.



Code
my @AoA = ();  
while(<DATA>){
chomp;
my($var1,$var2)=split(/,/);
push @AoA,[$var1,$var2,($var1>$var2?$var1:$var2)];
}
my @sorted_fourth = sort{$a->[2] <=> $b->[2]} @AoA;
for my $i (0..$#sorted_fourth){
$AoA[$i][3]=$sorted_fourth[$i][2];
}
for (@AoA) {
print join(',',@$_),"\n";
}
__DATA__
1,11
2,12
3,1
4,13
5,14
6,15
7,16
8,6
9,17
10,5

-------------------------------------------------


sohnaeo
Novice

Jul 21, 2006, 9:22 AM

Post #14 of 14 (3023 views)
Re: [KevinR] Sorting [In reply to] Can't Post

Hello Kevin,

Thanks a lot for this. Everything is clear to me now. It is great website and you are so helpful.



heaps of thanks again



Farhan

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives