CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
Lightbulb Efficient way to take count for multiple strings from a huge log file.

 



senthilkumar_ak
Novice

Aug 15, 2008, 1:35 AM

Post #1 of 4 (2159 views)
Lightbulb Efficient way to take count for multiple strings from a huge log file. Can't Post

Hi all,

I have a huge log [more than 1 GB] file generate from server everyday. I am checking the all exception from the file, send count for each every exception and transaction. The log contains 24 hour data and I need to take only for a time 5:00 am to 5:00 pm. I used a shell script to do it but its take huge time. I will explain here how I have done in the shell script.

I used the command SED for taking 5 to 5 from the log file first like the below

sed -n "/: 05:/,/: 18:/p" Trace.log > temp.log

And then taking count for each exception like this.

E1= `grep –ic “DataException” temp.log
E2= `grep –ic “DBException” temp.log
Totalexception = E1+ E2 | bc -l

Some time I want to search for two strings {means if the first string available in the line then search for the next string among the output}

where I used

count=`cat count_gcbs.log | grep -i “LimitsBusinessLogic” | grep -ic “DBUnexpectedException”`

and displaying the total count.

My problem is this script is taking huge time and I am greping the string again & again in the file and opening the file frequently. I know this will be done in perl very quick and easily. So please help me on this Is there any way is available to tune my script using perl.

log file snippet

Worker#1665jennings50CLI: 05:05:16 902 ChangeLimitsBusinessLogic.validateCorpLimits Entry
Worker#1665jennings50CLI: 05:05:16 902 ChangeLimitsBusinessLogic.getContactSummary Entry
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.getContactSummary Got contact from GOPM
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.getContactSummary Exit
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.doAuditInsert Entry
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.doAuditInsert Exit
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogicHelper.hasPassedSLACheck Entry
Worker#1665jennings50CLI: 05:05:16 985 ChangeLimitsBusinessLogicHelper.hasPassedSLACheck Exit

Regards,
senthil kumar ak


shawnhcorey
Enthusiast


Aug 15, 2008, 5:57 AM

Post #2 of 4 (2155 views)
Re: [senthilkumar_ak] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to] Can't Post


In Reply To
My problem is this script is taking huge time and I am greping the string again & again in the file and opening the file frequently. I know this will be done in perl very quick and easily. So please help me on this Is there any way is available to tune my script using perl.


1GB files are going to take a lot of time to process regardless of what software you use.

Note that this is untested; I don't have a log file to test it on.


Code
#!/usr/bin/perl 

use strict;
use warnings;

my %Count = ();

while( <> ){

if( /LimitsBusinessLogic/ && /DBUnexpectedException/ ){
$Count{DBUnexpectedException} ++;
}

# Add more if statements here

}

for my $id ( sort keys %Count ){
printf "%-30s %d\n", $id, $Count{$id};
}

__END__


__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".


senthilkumar_ak
Novice

Aug 18, 2008, 9:11 AM

Post #3 of 4 (2129 views)
Re: [shawnhcorey] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to] Can't Post

Thanks for the reply
but there is one more thing you left in your script. I want to search for only 5 -5 which i used sed command in the shell. how i can use that in the perl.


shawnhcorey
Enthusiast


Aug 18, 2008, 10:26 AM

Post #4 of 4 (2127 views)
Re: [senthilkumar_ak] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to] Can't Post


In Reply To
Thanks for the reply
but there is one more thing you left in your script. I want to search for only 5 -5 which i used sed command in the shell. how i can use that in the perl.


Add a state machine to track when to count.


Code
#!/usr/bin/perl  

use strict;
use warnings;

my %Count = ();
my $state = 0;

while( <> ){

if( /: 05:/ ){
$state = 1;
}
if( /: 18:/ ){
$state = 0;
}

if( $state == 1 ){

if( /LimitsBusinessLogic/ && /DBUnexpectedException/ ){
$Count{DBUnexpectedException} ++;
}

# Add more if statements here

}
}

for my $id ( sort keys %Count ){
printf "%-30s %d\n", $id, $Count{$id};
}

__END__


__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives