CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
script to count per hour success/error from logs

 



jeffersno1
Novice

Nov 12, 2012, 1:14 PM

Post #1 of 10 (4004 views)
script to count per hour success/error from logs Can't Post

Hi All,

I need some help! i know when I've been beaten! I help support a pass booking system and I need to generate some stats on a daily basis showing pass booking attempts (success and failure) every hour.

I'm trying to write a script that does the following:
1 - Script will run from cron daily against yesterdays files and store results. For now i'm using a testdata file
2 - search for pass booking success and failures
3 - Print to screen how many pass success and errors per hour by pass type eg:

Expected results should look like:

Time, Pass Name, Success, Failures
2012-11-08_06:00:00,passA,1,0
2012-11-08_07:00:00,passA,0,1
2012-11-08_07:00:00,passB,1,0
2012-11-08_07:00:00,passC,1,0
2012-11-08_07:00:00,passF,0,3

example date

2012-11-08_06:13:39:094,BOOK,444771234567,,admin@sys_3501,passA,,success,1341
2012-11-08_07:00:22:967,BOOK,442345678999,,admin@sys_3744,passB,,success,591
2012-11-08_07:02:10:128,BOOK,444444444444,,admin@sys_3672,passC,,success,792
2012-11-08_07:02:10:128,BOOK,444466666777,,admin@sys_3672,passA,,internal_server_error,92
2012-11-08_07:02:11:128,BOOK,444444444455,,admin@sys_3672,passF,,internal_server_error,12
2012-11-08_07:02:12:128,BOOK,444466666666,,admin@sys_3673,passF,,internal_server_error,23
2012-11-08_07:02:13:128,BOOK,444477777777,,admin@sys_3675,passF,,internal_server_error,11

I've tried to do this by incrementing the hour time which I've pulled out from the linearray.

The script works but it doesn't print the first hours worth of data, and now I've realised I don't know how to get the errors calculated, should i be using hashes ? They've always confused me!!

Any advice is much appreciated.


Code
#!/usr/bin/perl -w 

use POSIX;

my $filename = "/home/sys/testdata";
open (LOGFILE, $filename ) or die ("Could not open log file.");

$success_time="";
#$connclosed_time=0;
#$timedout_time=0;

for $line (<LOGFILE>)
{
chomp($line);
if (length($line) > 0)
{
@lineArray=split(",",$line);
if ( $lineArray[1] =~ /BOOK/ )
{
$lineArray[0] =~ s/_/:/g;
($date,$hh,$mm,$ss,$ms)=split(":",$lineArray[0]);
$time="$hh";
#$time="$date:$hh";
$key = $lineArray[5];
if (! defined $resultsHash{ $key })
{
$resultsHash{ $key } = [0,0]; ### there can 2 outcomes success or failure
}
my $temp = $resultsHash{ $key };
if ( defined $lineArray[7] && $lineArray[7] =~ /success/ )
{
if ($success_time eq "")
{
print "Time Changed (FIRST) $time\n";
$success_time=$time;
}
if ( $success_time eq $time )
{
# Same time
@$temp[0]++; # Increase Success Counter for this key
}
else
{
# Time Changed
print "Time Changed from $success_time to $time\n";
# Need to reset all Key counters not just this one.
$total=0;
foreach (sort keys %resultsHash)
{
$reset=$resultsHash{$_};
print " $date\_$success_time:$mm,$_,@$reset[0]\n";
$total=$total+@$reset[0];
@$reset[0]=0;
}
print " Total - $total\n";
# Increase Success Counter for this key only
@$temp[0]++;
$success_time=$time;
}
}
}
}
}
close LOGFILE;



(This post was edited by jeffersno1 on Nov 13, 2012, 12:15 AM)


Laurent_R
Veteran / Moderator

Nov 12, 2012, 2:27 PM

Post #2 of 10 (4000 views)
Re: [jeffersno1] script to count per hour success/error from logs [In reply to] Can't Post

Can you possibly explain in more details the content of your data. At the time, I can only try to make wild guesses which may turn out to be plain wrong.

Please tell us the general context of what these lines are supposed to report, and then explain the fields (don't tell me the first field is a date, this much I've picked up, but what does this date mean), etc.

Also, you have two different file formats, what are they?

Finally, please explain more what you want to extract from these files, giving enough details so that we know what you want to add, and so on. An example of the expected result from your input would also help.

Just one quick point on your code, which I have not yet tried to understand (I am waiting for that to get an understanding of your data and your requirement), you should "use strict;". This will force you to declare your variables in the adequate scope, which may seem a bit tedious, but it helps so much finding errors and bugs that no Perl program containing more than 3 lines (I am sure some would argue here for no program having more than 1 line) should be written without the use of stricture.


jeffersno1
Novice

Nov 13, 2012, 12:38 AM

Post #3 of 10 (3992 views)
Re: [Laurent_R] script to count per hour success/error from logs [In reply to] Can't Post

Hi Laurent,

Thanks for the response and you're right, i basically copied and pasted from log files without making it clear. I'm hoping the amendments makes it a little clearer.

I'm still a novice when it comes to coding, trying to get there, i do use strict but not all the time. I will try and make an extra effort to include this in future. Sly

Thanks for the advice


FishMonger
Veteran / Moderator

Nov 13, 2012, 8:36 AM

Post #4 of 10 (3978 views)
Re: [jeffersno1] script to count per hour success/error from logs [In reply to] Can't Post

Use the warnings pragma instead of the -w switch.

ALWAYS use the strict pragma and declare your vars, which is done with the 'my' keyword.

Var names should give describe what kind of data they hold. Some or most of your vars fails to do so.

Use a lexical var for the filehandle instead of the bareword and use the 3 arg form of open and the die statement should include the reason it failed.

It's more efficient to use a while loop instead of the for loop, when looping over the file.

When you split the csv line, only extract/save the needed fields and drop the rest.

You should build up a HoHoH (Hash of Hashes of Hashes) that holds the data.


Code
#!/usr/bin/perl 

use strict;
use warnings;

my $logfile = 'data.log';
open my $log_fh, '<', $logfile or die "failed to open '$logfile' $!";

my %booking_stats;
while (my $line = <$log_fh>) {

chomp $line;
my ($datestamp, $booking, $pass_name, $status) = (split(/,/, $line))[0,1,5,7];
next if $booking ne 'BOOK';

my ($date) = $datestamp =~ /^([^:]+)/;
if ($status eq 'success') {
$booking_stats{"$date:00:00"}{$pass_name}{success}++;
}
else {
$booking_stats{"$date:00:00"}{$pass_name}{fail}++;
}
}
close log_fh;

print "Time,Pass Name,Success,Failures\n";

foreach my $date ( sort keys %booking_stats ) {
foreach my $name ( sort keys %{$booking_stats{$date}} ) {
$booking_stats{$date}{$name}{success} //= 0;
$booking_stats{$date}{$name}{fail} //= 0;

printf("%s,%s,%d,%d\n", $date,
$name,
$booking_stats{$date}{$name}{success},
$booking_stats{$date}{$name}{fail}
);
}
}



(This post was edited by FishMonger on Nov 13, 2012, 8:36 AM)


jeffersno1
Novice

Nov 13, 2012, 11:51 AM

Post #5 of 10 (3968 views)
Re: [FishMonger] script to count per hour success/error from logs [In reply to] Can't Post

Hi FishMonger,

Thanks for your advice, I will make sure I use strict from now on. if it makes good practice im all for it, even if it can be a pain in the a55 :)

I attempted your code on the sample file and its come back with the following error:

Can't modify division (/) in division (/) at ./rafile2.pl line 32, near "0;"
(Might be a runaway multi-line // string starting on line 31)
Execution of ./rafile2.pl aborted due to compilation errors.

Do you think i need to update my version of perl ? im running v5.8.8.

Thanks again the your help, dont know where id be without you guys :)


FishMonger
Veteran / Moderator

Nov 13, 2012, 12:12 PM

Post #6 of 10 (3966 views)
Re: [jeffersno1] script to count per hour success/error from logs [In reply to] Can't Post

5.8.8 was released in Jan 2006, so yes it's now out of date and should be upgraded. The current release is 5.16.1

//= was added starting around v5.10.x

To support your older version, you can change it to ||=


Code
        $booking_stats{$date}{$name}{success} ||= 0; 
$booking_stats{$date}{$name}{fail} ||= 0;



jeffersno1
Novice

Nov 25, 2012, 11:56 AM

Post #7 of 10 (3763 views)
Re: [FishMonger] script to count per hour success/error from logs [In reply to] Can't Post

Hi Fishmonger,

You've shown me a new way of scripting, i like the style!! Wink

Thanks for your help here, i know its a very old release, but I'm governed by what releases i can use (internal security). I have asked them to update the packages available.

I've tweaked the script to break down all the errors just added a few elsif statements and email results to another department, just one thing, i would like to add a total to the end, total up all the success and errors etc. I've tried adding another foreach loop using the key and value but i don't get a printout. Any suggestions?


Code
foreach my $name ( sort keys %booking_stats )


Many thanks


Laurent_R
Veteran / Moderator

Nov 25, 2012, 2:04 PM

Post #8 of 10 (3755 views)
Re: [jeffersno1] script to count per hour success/error from logs [In reply to] Can't Post


In Reply To
I've tried adding another foreach loop using the key and value but i don't get a printout. Any suggestions?


Code
foreach my $name ( sort keys %booking_stats )


Many thanks


Do you really expect that code to print out something? Where is the print statement? What you're doing here is to list the keys of your hash, sort them in "asciibetical" order and assign $name to them one after the other, and you don't do anything usefull with the result.

In addition, it does not take unto account the fact that %booking_stats is a compound data structure (hash of hashes of hashes, if I remember right). You'll need to dereference the whole shebang to get at the relevant data. I'll will not say anymore on this, because I have no idea where the data that you want to add and print out is stored.

The alternative, probably an easier and more efficient one, is to add up the values (successes, failures, etc.) in counters when you process them earlier in your program, so that, at the end, you can just print these counters without having to re-scan the whole data structure.


jeffersno1
Novice

Nov 25, 2012, 2:25 PM

Post #9 of 10 (3752 views)
Re: [Laurent_R] script to count per hour success/error from logs [In reply to] Can't Post

Hi Laurent,

I have tried several print statements but was just highlighting the top of the code I've tried. I didn't want to post every attempt at getting the results. But here are a few of the failed attempts


Code
foreach my $key ( sort keys %booking_stats ) { 
$test = $booking_stats{$name}{success};
print "TEST = $success : @$tester\n";
}

foreach my $key ( sort keys %booking_stats ) {
$test2 = $booking_stats {$_};
print "TEST2 : $test2 : $_ : @$test2\n";
}

They all fail, as you can see from the last one i try all sorts!! my query was that I'm already sorting the keys, it was just the print statement that i cant get my head around.
I don't understand how the

Code
{success}

parts are not variables, i don't know how they are referenced.

Thanks


Laurent_R
Veteran / Moderator

Nov 25, 2012, 11:18 PM

Post #10 of 10 (3726 views)
Re: [jeffersno1] script to count per hour success/error from logs [In reply to] Can't Post

Hi, yeah, now your code makes more sense to me (even though, of course, it does not work).

Why don't you change this part:


Code
    if ($status eq 'success') {  
$booking_stats{"$date:00:00"}{$pass_name}{success}++;
}
else {
$booking_stats{"$date:00:00"}{$pass_name}{fail}++;
}


to include a couple of total counters for successes and failures. Something like this:


Code
    if ($status eq 'success') {  
$booking_stats{"$date:00:00"}{$pass_name}{success}++;
$total_successes ++;
}
else {
$booking_stats{"$date:00:00"}{$pass_name}{fail}++;
$total_failures ++;
}


The, at the end of your procedure, just print the new counter variables.

Note that the new variables have to be declared outside of the limited lexical scope where they are used; probably just declare them near the top of your program.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives