Perl Guru Forums: Perl Programming Help: Beginner: Lightbulb Efficient way to take count for multiple strings from a huge log file.

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Beginner:
Lightbulb Efficient way to take count for multiple strings from a huge log file.

senthilkumar_ak
Novice

Aug 15, 2008, 1:35 AM

Post #1 of 4 (1833 views)

Lightbulb Efficient way to take count for multiple strings from a huge log file.	Can't Post

Hi all,

I have a huge log [more than 1 GB] file generate from server everyday. I am checking the all exception from the file, send count for each every exception and transaction. The log contains 24 hour data and I need to take only for a time 5:00 am to 5:00 pm. I used a shell script to do it but its take huge time. I will explain here how I have done in the shell script.

I used the command SED for taking 5 to 5 from the log file first like the below

sed -n "/: 05:/,/: 18:/p" Trace.log > temp.log

And then taking count for each exception like this.

E1= `grep –ic “DataException” temp.log
E2= `grep –ic “DBException” temp.log
Totalexception = E1+ E2 | bc -l

Some time I want to search for two strings {means if the first string available in the line then search for the next string among the output}

where I used

count=`cat count_gcbs.log | grep -i “LimitsBusinessLogic” | grep -ic “DBUnexpectedException”`

and displaying the total count.

My problem is this script is taking huge time and I am greping the string again & again in the file and opening the file frequently. I know this will be done in perl very quick and easily. So please help me on this Is there any way is available to tune my script using perl.

log file snippet

Worker#1665jennings50CLI: 05:05:16 902 ChangeLimitsBusinessLogic.validateCorpLimits Entry
Worker#1665jennings50CLI: 05:05:16 902 ChangeLimitsBusinessLogic.getContactSummary Entry
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.getContactSummary Got contact from GOPM
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.getContactSummary Exit
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.doAuditInsert Entry
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogic.doAuditInsert Exit
Worker#1665jennings50CLI: 05:05:16 903 ChangeLimitsBusinessLogicHelper.hasPassedSLACheck Entry
Worker#1665jennings50CLI: 05:05:16 985 ChangeLimitsBusinessLogicHelper.hasPassedSLACheck Exit

Regards,
senthil kumar ak

shawnhcorey
Enthusiast

Aug 15, 2008, 5:57 AM

Post #2 of 4 (1829 views)

Re: [senthilkumar_ak] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to]	Can't Post

In Reply To

My problem is this script is taking huge time and I am greping the string again & again in the file and opening the file frequently. I know this will be done in perl very quick and easily. So please help me on this Is there any way is available to tune my script using perl.

1GB files are going to take a lot of time to process regardless of what software you use.

Note that this is untested; I don't have a log file to test it on.

Code

#!/usr/bin/perl 
 
use strict; 
use warnings; 
 
my %Count = (); 
 
while( <> ){ 
 
  if( /LimitsBusinessLogic/ && /DBUnexpectedException/ ){ 
    $Count{DBUnexpectedException} ++; 
  } 
 
  # Add more if statements here 
 
} 
 
for my $id ( sort keys %Count ){ 
  printf "%-30s %d\n", $id, $Count{$id}; 
} 
 
__END__

__END__

I love Perl; it's the only language where you can bless your thingy.

Perl documentation is available at perldoc.perl.org. The list of standard modules and pragmatics is available in perlmodlib.

Get Markup Help. Please note the markup tag of "code".

senthilkumar_ak
Novice

Aug 18, 2008, 9:11 AM

Post #3 of 4 (1803 views)

Re: [shawnhcorey] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to]	Can't Post

Thanks for the reply
but there is one more thing you left in your script. I want to search for only 5 -5 which i used sed command in the shell. how i can use that in the perl.

shawnhcorey
Enthusiast

Aug 18, 2008, 10:26 AM

Post #4 of 4 (1801 views)

Re: [senthilkumar_ak] Lightbulb Efficient way to take count for multiple strings from a huge log file. [In reply to]	Can't Post

In Reply To

Thanks for the reply
but there is one more thing you left in your script. I want to search for only 5 -5 which i used sed command in the shell. how i can use that in the perl.

Add a state machine to track when to count.

Code

#!/usr/bin/perl  
  
use strict;  
use warnings;  
  
my %Count = ();  
my $state = 0; 
 
while( <> ){  
 
  if( /: 05:/ ){ 
    $state = 1; 
  } 
  if( /: 18:/ ){ 
    $state = 0; 
  } 
  
  if( $state == 1 ){ 
 
    if( /LimitsBusinessLogic/ && /DBUnexpectedException/ ){  
      $Count{DBUnexpectedException} ++;  
    }  
  
    # Add more if statements here  
  
  } 
}  
  
for my $id ( sort keys %Count ){  
  printf "%-30s %d\n", $id, $Count{$id};  
}  
  
__END__

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives