CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Regular Expressions:
perl script help

 

First page Previous page 1 2 Next page Last page  View All


perlisgreat
Novice

May 28, 2009, 8:17 AM

Post #1 of 36 (16933 views)
perl script help Can't Post

Hi,

I am writing a perl script and need some in extracting some data. Basically it is a Log Analyzer and there are multiple log files from where the data comes.

i am done with the grep part of the userID with this code-

Code
 
#!/usr/bin/perl -w
open(VALUE, "< arcotwebfort_20May09_00_03_55.log");
@stock = <VALUE>;
@matches = (grep(/$username/, @stock));
print @matches;

it gave me this output--

Code
 


i got the sessionID's from this,they are-10088781,10088815. like this there are plenty of sessionID related to a single user.

i collected this information in an array @matches. now i want to take sessionID's from this array and search for the sessionID in the logfile which AuthMechanism it belongs to.
There are three-four AuthMechanism to which a user can belong.they are--
QnAModule::authenticate
UPAuthModule::authenticate
UPAuthModule::authenticate

On the basis of these mechanisms i need to know the user belongs to which mechanism. This part is very tricky for me and i am stuck here as i am not well wersed with Perl and learning it.
I will upload my logfile. please suggest me to proceed further.
If my questions are not clear to you please let me know i will try to explain them better.Please help me in proceeding further, i am also trying to write it myself.
What i want is what the given user is doing in the session id and the session id is related to which mechanism.

So let say my script takes username as arguememnt, i am able to print all the information that is related to that user with sessionID's. this sessionID is related to diffrent Authentication mechanism mentioned earlier.The user can belong to any or all mechanisms.my basic criteria is that the user belongs to which mechanism and he is doing what(data related to the username in mechanisms)
The output should be like this--
say i gave the input username as 01018603, i will be able to get the output as--
the example belongs to one AuthMechanism ie QnA

Quote
if [ sessionID_No_Got_from_the_username_list == QnA sessionID ]; then
print all the data inside that sessionID that relates to user with timestamp
print this also
Handle_NSPAdvice:: Handling NSPAdvice for mechanism [2] with time stamp.

Any advice on the script or help will be very greatful.
Thanks
NT


(This post was edited by namishtiwari on Jun 12, 2009, 3:54 AM)
Attachments: logfile.txt (247 KB)


1arryb
User

May 28, 2009, 8:28 AM

Post #2 of 36 (16931 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi namishtiwari,

I think we need a little more information:

1. How are you relating session ids to users? Is the user id listed in the logfile or do you have a separate list?

2. How are the session ids related to authentication methods? Again, is this information available in the logfile or do you have a separate list? I don't see anything like "QnAModule", etc. in the file you uploaded.

Cheers,

Larry


perlisgreat
Novice

May 29, 2009, 3:10 AM

Post #3 of 36 (16920 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Hi,

Thanks for paying attention to my problem. I am stuck at this place and not able to proceed.i cna only upload 250kb of file and my logfile is quite huge. Regarding your queries the answers are below--

1.. Yes userID is is listed in the logfile by name userName. When we search for the userName, it gives us details about the userName and which sessionID's it belongs.SessionID comes after tid value and leaving one more field after that,consisting of 8digits. each seesion id gives us the details about which mechanism it belongs to.
2. Each sessionID belongs to a mechanism,if we specifically search for the SessionID's it will show the details like this.

Code
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO:    pid 2172 tid 688: 0: 10081229: Message: ArAuthBinaryAuthRequestMgr::setSecurityInfo: security_info_offset [0] is invalid. Assuming security info not available. 
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 60: 10081229: Entering [MechQnA::handleAuthRequest]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 60: 10081229: Exiting [MechQnA::handleAuthRequest]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 17: 10081229: Using session identifier [1:10081229]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Entering QnAModule::openSession
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Exiting QnAModule::openSession
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 17: 10081229: ArAuthFrameworkImpl::doPreAuth::1:10081229:: Authentication mechanism returned [0] for AuthIdentity [01018603]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Entering QnAModule::authenticate
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: QnA Session Id :[1:10081229]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: QNA Step - ASK AUTH QUESTIONS
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: User DB Query Details:
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: userName :[01018603]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Group ID :1020
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: User ID :-1
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Number of rows fetched from DB : 1
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Entering QnAUserData::dump
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Exiting QnAUserData::dump
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Set of questions(bitmap) selected : 3
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Session tracker Id associated with QnA challenge(auth round questions) : [1:10081229]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Exiting QnAModule::authenticate
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 17: 10081229: ArAuthFrameworkImpl::doAuth::1:10081229:: Authentication mechanism returned [1] for AuthIdentity [01018603]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Handle_NSPAdvice:: Handling NSPAdvice for mechanism [2]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 17: 10081229: ArAuthFrameworkImpl::doPostAuth::1:10081229:: Authentication mechanism returned [1] for AuthIdentity [01018603]
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Entering QnAModule::closeSession
Tue May 19 22:29:12.634 2009 Morocco Standard Time INFO: pid 2172 tid 688: 160: 10081229: Exiting QnAModule::closeSession
Tue May 19 22:29:12.649 2009 Morocco Standard Time INFO: pid 2172 tid 688: 60: 10081229: Entering [MechQnA::handleAuthResMoreSteps]
Tue May 19 22:29:12.649 2009 Morocco Standard Time INFO: pid 2172 tid 688: 60: 10081229: Exiting [MechQnA::handleAuthResMoreSteps]
Tue May 19 22:29:12.649 2009 Morocco Standard Time INFO: pid 2172 tid 688: 60: 10081229: Sending Continue with challenge
Tue May 19 22:29:12.649 2009 Morocco Standard Time INFO: pid 2172 tid 688: 61: 10081229: Packet being sent

Thi sdetails will simply tell us which mechanis it belong as we can see the lines like "QnA Session Id :[1:10081229]" in the sessionID related data. This information is available in the logfile. QnAModule is there in the logfile uploaded.
I think now question will be much clear,if not still kindly tell me. and i will try to elaborate more.I am again uploading the logfile.txt file.

Thanks
NT
Attachments: logfile.txt (247 KB)


KevinR
Veteran


May 29, 2009, 8:37 AM

Post #4 of 36 (16907 views)
Re: [1arryb] perl script help [In reply to] Can't Post


In Reply To
Hi namishtiwari,

I think we need a little more information:

1. How are you relating session ids to users? Is the user id listed in the logfile or do you have a separate list?

2. How are the session ids related to authentication methods? Again, is this information available in the logfile or do you have a separate list? I don't see anything like "QnAModule", etc. in the file you uploaded.

Cheers,

Larry


You're being quoted on another forum Larry (without credit)

http://www.unix.com/shell-programming-scripting/110862-perl-script-help.html#post302320855
-------------------------------------------------


1arryb
User

May 29, 2009, 8:54 AM

Post #5 of 36 (16903 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi namishtiwari,

For parsing large files, you have to process them line-by-line, not read the whole thing into memory. Perhaps something like this:

Code
#!/usr/bin/perl 

use strict;
use warnings;
no warnings "uninitialized";

my $logFile = $ARGV[0];
die "usage: $0 <logFile>" unless $logFile;
die "Logfile $logFile doesn't exist" unless -f "$logFile";
open(my $log, "<", $logFile) or die "Can't open $logFile for reading.";
print "Processing file $logFile...\n";
my $authenticates = {};
my $n = 0;
while(my $line = <$log>) {
# Outer loop. Look for an interesting part of the log file.
$n++;
$line =~ tr/\r\n//d;
if ( $line =~ /Entering (\S+)::authenticate/ ) {
# Aha! Start of the interesting part.
my $authentication = $1;
while ( $line = <$log> ) {
# Inner loop: Find the UserName.
$n++;
$line =~ tr/\r\n//d;
if ( $line =~ /userName :[[](\S*)[]]/ ) {
my $userName = "$1" ? "$1" : "NULL"; # You have some null userNames in your log.
my $nSessions = $authenticates->{$authentication}->{$userName};
$nSessions = $nSessions ? $nSessions += 1 : 1; # Tabulate the number of sessions for this authMethod/userName.
$authenticates->{$authentication}->{"$userName"} = $nSessions;
last;
} elsif ( $line =~ /(?:Entering|Exiting) \S+::authenticate/ ) {
print "ERROR: Misparse at line $n of $logFile\n";
last;
}
}
}
}
close($log);

# Output global results:
print "AuthMethod\tUser\tSessions\n";
for my $authMethod (sort(keys(%$authenticates))) {
for my $user (sort(keys(%{$authenticates->{$authMethod}}))) {
print "$authMethod\t$user\t$authenticates->{$authMethod}->{$user}\n";
}
}


Running this code against your sample logfile.txt, produces the following:

Code
Processing file logfile.txt... 
AuthMethod User Sessions
QnAModule 01018603 3
QnAModule 01214225 2
QnAModule 01789302 1
QnAModule NULL 2


As you can see by your output, you have some data problems. This is a good example why parsing log files for data is a fairly weak interface: You may or may not get good or complete data.

Cheers,

Larry


(This post was edited by 1arryb on Jun 1, 2009, 1:00 PM)


perlisgreat
Novice

Jun 9, 2009, 12:33 AM

Post #6 of 36 (16824 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Hi 1arryb,

Thank You very much for paying attention to my problem and helping me for my problem. I was not in office due to some family casuality and could not reply.

Except for QnA Module authentication the user name is coming like this --

so it is clear that pattern is not common for all but they are diffrent.

In this script i want to pass user name as command line arguement and i want to pass multiple log files not just one. I want to print the timestamp also and what the user was doing in the authentication module. I am thankful for your kind help.
Any kind of help is greatly appericiated.

Thanks
NT

Quote



(This post was edited by namishtiwari on Jun 12, 2009, 3:53 AM)


1arryb
User

Jun 9, 2009, 7:25 AM

Post #7 of 36 (16818 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi namishtiwari,

Ok. What have you tried?

Larry


perlisgreat
Novice

Jun 10, 2009, 2:10 AM

Post #8 of 36 (16803 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Hi Larry,

I made little change in the script to get the user name for other authentication methods.
I added this line in the script of yours and now able to get the values. i added

Code
if ( $line =~ /userName :[[](\S*)[]]/ || $line =~ /UPAuth SUCCESS for user[[](\S*)[]]/ || $line =~ /OTP SUCCESS for user(()(\S*)())/ ) {  

inplace of

if ( $line =~ /userName :[[](\S*)[]]/)


i need suggestion in passing multiple log files not just one, as in my case there are multiple log files for the span of a day. is it possible to use metacharacters in the logfile name like-
if i enter logfile* it will pick all the files matching this.
You are an expert in this field so your sugegstion will be very valueable to me.how i can get the timestamp because in this case it seems very difficult to me.

Thanks
NT


1arryb
User

Jun 10, 2009, 7:35 AM

Post #9 of 36 (16794 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi namishitwari,

Easy-peasey. The perl glob() command is what you're looking for. You'll need to pass the log directory and log prefix to your script instead of a file name and add an outer loop to handle the multiple files. Perhaps something like:

Code
... 
my $logDir = $ARGV[0];
my $logPrefix = $ARGV[1]
die "usage: $0 <logDir> <logPrefix>" unless $logDir and $logPrefix;
die "Log dir $logDir doesn't exist" unless -d "$logDir";
for my $logFile ( glob("$logDir/${logPrefix}*") ) {
# Process $logFile as before.
...
}


Depending on whether you want per-file or summary statistics, you can keep the output code where it is or move it outside of the loop on $logFile.

UPDATE:

I didn't see the bit about timestamps. Getting the timestamp is easy.

Code
... 
# Note: The '(?:...)' regex means "group these regular expressions but don't remember the match".
if ( $line =~ /^(.*) INFO:.*(?:userName [:]|UPAuth SUCCESS for user|OTP SUCCESS for user)(?:\(|\[)(\S*)(?:\)|\])/ ) {
my $timestamp = $1;
my $userName = $2 ? $2 : "NULL";
...


The trick will be if you need to convert the timestamp to some other format. That can be a chore.

UPDATE2:

I'm also not sure what you intend to do with the timestamps. You'll have to change your output to report each session, instead of summarizing sessions per authMethod.

Cheers,

Larry


(This post was edited by 1arryb on Jun 10, 2009, 8:28 AM)


perlisgreat
Novice

Jun 11, 2009, 7:17 AM

Post #10 of 36 (16771 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Hi Larry,

Your suggestions and piece of code snippet are very helpful to me. I am getting a good idea about Perl now.
I have some more queries sir,

how can i make a comma seperated output summary.
i attached the sample log file.

I have to capture these data in the log file.

Arcotid
Time Stamp, Username, Success, Failure, Error Code, Error Message

In the log snippet the userID can be found in-


Code
Arcot Native Server: recvd AA_BIN_MSG_VER_CHG 
Cert Subject String....:[CN=01095848;O=MLS; UserID=01095848; CardName=ARCARD]

Success and failure are like this-

Code
ArcotID Auth SUCCESS (serial Number [e93b]) 
Auth fail due to signature verification.Serial Number [11872](This is error message also)

Code
 
error code is -
Authentication mechanism returned [3], here 3 is error code and for success error code is always 0.

Formatting and printing in csv format is the part i am unaware,so i require your help in this.I need to do this for all authentication methods so if i get help in one i will be able to do it for rest.

Thanks
NT


(This post was edited by namishtiwari on Jun 12, 2009, 2:17 AM)


1arryb
User

Jun 11, 2009, 7:29 AM

Post #11 of 36 (16767 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

hi namishtiwari,

Maybe I'm being obtuse, but to get comma separated output you...use commas?

Code
... 
print "$authMethod,$user,$authenticates->{$authMethod}->{$user}\n";
...


Ok, think about your problem. Your latest requirement is to output 1 record for each session. This is actually easier, since you no longer need to group statistics by AuthMethod. Furthermore, since logfile entries naturally occur in timestamp order, you don't even need to sort your output!

What this means is you can throw away the $authenticates hash altogether and print each output line as soon as you parse it.

Why don't you give it a try?

Cheers,

Larry


perlisgreat
Novice

Jun 12, 2009, 7:24 AM

Post #12 of 36 (16741 views)
Re: [1arryb] perl script help [In reply to] Can't Post


Code
#!/usr/bin/perl -w 
my $timestamp;
my $line = "Fri May 29 18:29:57.357 2009 Morocco Standard Time INFO: pid 3216 tid 1724: 170: 132192: apricocot Native Server: recvd AA_BIN_MSG_VER_CHG";
if ($line =~ /^(.*) INFO: .* recvd AA_BIN_MSG_VER_CHG/) {
$timestamp = $1;
print $timestamp;
}


It is not printing anything, regular expression in the if block is not correct, kindly help me in correcting it.

Thanks
NT


1arryb
User

Jun 14, 2009, 9:19 AM

Post #13 of 36 (16732 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi namishtiwari,

Your regex has an extra space in it.

Larry


KevinR
Veteran


Jun 14, 2009, 10:21 AM

Post #14 of 36 (16731 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Just an FYI Larry:

http://www.unix.com/shell-programming-scripting/112045-regex-problem.html
-------------------------------------------------


perlisgreat
Novice

Jun 15, 2009, 2:33 AM

Post #15 of 36 (16714 views)
Re: [1arryb] perl script help [In reply to] Can't Post


In Reply To
hi namishtiwari,

Maybe I'm being obtuse, but to get comma separated output you...use commas?

Code
... 
print "$authMethod,$user,$authenticates->{$authMethod}->{$user}\n";
...


Ok, think about your problem. Your latest requirement is to output 1 record for each session. This is actually easier, since you no longer need to group statistics by AuthMethod. Furthermore, since logfile entries naturally occur in timestamp order, you don't even need to sort your output!

What this means is you can throw away the $authenticates hash altogether and print each output line as soon as you parse it.

Why don't you give it a try?

Cheers,

Larry


Thanks Larry for your suggestions.
I tried this code --


Code
#!/usr/bin/perl 

print "Hello, World...\n";
my $logFile = $ARGV[0];
die "usage: $0 <logFile>" unless $logFile;
die "Logfile $logFile doesn't exist" unless -f "$logFile";
open(my $log, "<", $logFile) or die "Can't open $logFile for reading.";
print "Processing file $logFile...\n";
#my $authenticates = {};
my $n = 0;
my $Time_Stamp ;
my $User_Name;
my $Success;
my $Failure;
my $ErrorCode;
my $ErrorMsg;
while(my $line = <$log>) {
$n++;
$line =~ tr/\r\n//d;
if ($line =~ /^(.*)INFO:.*recvd AA_BIN_MSG_GET_WLT/) {
$Time_Stamp = $1;
while ( $line = <$log> ) {
if ($line =~ /recvd AA_BIN_MSG_VER_CHG$/) {
while ( $line = <$log>) {
if ($line =~ /Cert.*UserID=(\d+)/) {
$User_Name = $1 ? $1 : "NULL" ;
}
}
while ( $line = <$log> ) {
if ($line =~ /ArcotID Auth SUCCESS/) {
$Success = "Success";
}
}

while ( $line = <$log> ) {
if ($line =~ /Auth fail due to signature verification/) {
$Failure = "Failure";
}
}
while ( $line = <$log> ) {
if ($line =~ /Authentication mechanism returned \[(\d+)\]/) {
$ErrorCode = $1;
}
}
while ( $line = <$log>) {
if ($line =~ /(Auth fail due to signature verification.)/) {
$ErrorMsg = $1 ? $1 : "Null";
}
}
}print "$Time_Stamp , $User_Name , $Success , $Failure , $ErrorCode , $ErrorMsg \n";
next;
}
}


}

I am not getting the desired output.
I think there is some logic problem in my code,though i am getting all the values in variables but while printing nothing comes.

The snippet of the log file is like this--

Code
Native Server: recvd AA_BIN_MSG_GET_WLT 
Fri May 29 18:31:53.467 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 17: 132357: ArAuthFrameworkImpl::doAuth::11:132357:: Authentication mechanism returned [1] for AuthIdentity [;ARCARD]
Fri May 29 18:31:53.467 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 170: 132357: Handle_NSPAdvice:: Handling NSPAdvice for mechanism [4]
Fri May 29 18:31:53.467 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 17: 132357: ArAuthFrameworkImpl::doPostAuth::11:132357:: Authentication mechanism returned [1] for AuthIdentity [01302904]
Fri May 29 18:31:53.482 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 60: 132357: Sending Continue with challenge
Fri May 29 18:31:53.482 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 61: 132357: Packet being sent
Fri May 29 18:31:53.482 2009 Morocco Standard Time INFO: pid 3216 tid 2672: 60: 0: Packet Sent (MsgType:110, Size: 1564 Bytes).
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 60: 0: Packet Received (MsgType:109, Size: 1463 Bytes).
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 60: 0: Locale Name[en], Locale ID[1]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 0: 132358: Message: ArAuthBinaryAuthRequestMgr::setSecurityInfo: security_info_offset [0] is invalid. Assuming security info not available.
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 17: 132358: Using session identifier [11:132358]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 17: 132358: ArAuthFrameworkImpl::doPreAuth::11:132358:: Authentication mechanism returned [0] for AuthIdentity []
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Arcot Native Server: recvd AA_BIN_MSG_VER_CHG
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Cert Subject String....:[CN=01302904;O=MLS; UserID=0123456; CardName=ARCARD]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Cert issuer............:[]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Cert serial String.....:[e93b]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: ArcotId Certificate received with signature have ArcotExtension. Cert issuer=CN=
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: ArcotID Auth SUCCESS (serial Number [e93b])
For Fail the line is --
Auth fail due to signature verification.Serial Number [11872]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Session tracker Id associated with verify signed challenge[11:132351]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 17: 132358: ArAuthFrameworkImpl::doAuth::11:132351:: Authentication mechanism returned [0] for AuthIdentity []
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Handle_NSPAdvice:: Handling NSPAdvice for mechanism [4]
Fri May 29 18:31:53.498 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: Handle_NSPAdvice::NSP Action :[NSP_RESET]
Fri May 29 18:31:53.514 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 170: 132358: NSP Update Query is based on Serial Number
Fri May 29 18:31:53.514 2009 Morocco Standard Time INFO: pid 3216 tid 2888: 17: 132358: ArAuthFrameworkImpl::doPostAuth::11:132351:: Authentication mechanism returned [0] for AuthIdentity [01302904]



perlisgreat
Novice

Jun 16, 2009, 7:20 AM

Post #16 of 36 (16688 views)
Re: [1arryb] perl script help [In reply to] Can't Post


In Reply To
Hi namishitwari,

Easy-peasey. The perl glob() command is what you're looking for. You'll need to pass the log directory and log prefix to your script instead of a file name and add an outer loop to handle the multiple files. Perhaps something like:

Code
... 
my $logDir = $ARGV[0];
my $logPrefix = $ARGV[1]
die "usage: $0 <logDir> <logPrefix>" unless $logDir and $logPrefix;
die "Log dir $logDir doesn't exist" unless -d "$logDir";
for my $logFile ( glob("$logDir/${logPrefix}*") ) {
# Process $logFile as before.
...
}


Depending on whether you want per-file or summary statistics, you can keep the output code where it is or move it outside of the loop on $logFile.

UPDATE:

I didn't see the bit about timestamps. Getting the timestamp is easy.

Code
... 
# Note: The '(?:...)' regex means "group these regular expressions but don't remember the match".
if ( $line =~ /^(.*) INFO:.*(?:userName [:]|UPAuth SUCCESS for user|OTP SUCCESS for user)(?:\(|\[)(\S*)(?:\)|\])/ ) {
my $timestamp = $1;
my $userName = $2 ? $2 : "NULL";
...


The trick will be if you need to convert the timestamp to some other format. That can be a chore.

UPDATE2:

I'm also not sure what you intend to do with the timestamps. You'll have to change your output to report each session, instead of summarizing sessions per authMethod.

Cheers,

Larry


Hi Larry,

Here is the snippet of the code--


Code
#!/usr/bin/perl  

my $logDir = $ARGV[0];
my $logPrefix = $ARGV[1];
die "usage: $0 <logDir> <logPrefix>" unless $logDir and $logPrefix;
die "Log dir $logDir doesn't exist" unless -d "$logDir";
for my $logFile ( glob("$logDir/${logPrefix}*") )
{
open($log, "<", $logFile) or die "Can't open $logFile for reading.";
open(FP_OUT,">temp1") or die "cannot create file temp1 for writing";
print "Processing file $logFile...\n";
my $Time_Stamp ;
$line_number=0;
my $User_Name;
my $Success;
my $Failure;
my $ErrorCode;
my $ErrorMsg;

while( $line = <$log> )
{
$line_number++;
chomp($line);
if ($line =~ /^(.*)INFO:.*Entering OTPAuthModule::authenticate/)
{
$Time_Stamp = $1;
printf "$line_number: in Entering OTPAuthModule::authenticate\n";
printf FP_OUT "$Time_Stamp,";
die "some error in OTP_Search_For_UserID "if(OTP_Search_For_UserID() == -1);
die "some error in OTP_Search_For_Success_Or_failure "if(OTP_Search_For_Success_Or_failure() == -1);
}
}
}

sub OTP_Search_For_UserID
{
while ( $line = <$log> )
{
$line_number++;
chomp($line);
if ( $line =~ /OTP User DB Query is based on username\[(..*)\] and groupID\[(..*)\]/ )
{
printf "$line_number: in user id = $1 \n";
$User_Name = $1 ? $1 : "NULL" ;
printf FP_OUT "$User_Name,";
return 0;

}
elsif($line =~ /Prepared to Send OK/)
{
printf "$line_number : did not get user id got prepared to send ok \n";
return -1;
}
}
return -1;
}

sub OTP_Search_For_Success_Or_failure
{
while ( $line = <$log> )
{
$line_number++;
chomp($line);
if ($line =~ /OTP SUCCESS.*/) {
printf "$line_number: in sucess\n";
printf FP_OUT "Success,";
$ErrorMsg="null";
while ( $line = <$log> )
{
$line_number++;
if ($line =~ /ArAuthFrameworkImpl::doPostAuth.*Authentication mechanism returned \[(\d+)\]/)
{

printf FP_OUT "${1}, $ErrorMsg\n";
return 0;
}
elsif($line =~ /Prepared to Send OK/)
{
printf "$line_number : END\n";
return 0;
}
}
return -1;
}

elsif($line =~ /Message: OTP FAILED.*Reason : (.*)!/)
{
$ErrorMsg=$1;
printf "$line_number: Failure\n";
printf FP_OUT "Failure,";
while ( $line = <$log> )
{
$line_number++;
if ($line =~ /ArAuthFrameworkImpl::doPostAuth.*Authentication mechanism returned \[(\d+)\]/)

{
printf FP_OUT "${1}, $ErrorMsg\n";
return 0;
}
elsif($line =~ /Sending Invalid credential/)
{
printf "$line_number : END\n";
return 0;
}

}
return -1;;
}
}
}


i am running this file as-

C:\Perl Script>perl OTP_Authentication.pl ./arcotwebfort_29May09* >results.txt

but it gives me
usage: OTP_Authentication.pl <logDir> <logPrefix> at OTP_Authentication.pl line
5.

Can you please tell me how to use this.
Am i running the program fine. The log files exists in the same directory.

Thanks
NT


(This post was edited by namishtiwari on Jun 16, 2009, 7:24 AM)


1arryb
User

Jun 16, 2009, 7:32 AM

Post #17 of 36 (16686 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

namishtiwari,

Try:

Code
C:\Perl Script>perl OTP_Authentication.pl . arcotwebfort_29May09 >results.txt


If '.' doesn't work, use the full path of the directory containing the logfiles.

Cheers,

Larry


perlisgreat
Novice

Jun 16, 2009, 7:49 AM

Post #18 of 36 (16685 views)
Re: [1arryb] perl script help [In reply to] Can't Post

Hi Larry,

Thank you very very much. It did work,but in temp1 file it did not pasted the whold data for all the files, though in file results.txt everything is coming. The results.txt file shows that 3 files are processed and it collected the information which is printed in the code but in temp1 data for only 1 file is coming not for all.
Is there something wrong in the code or temp1 need to be handeled in a diffrent way.

Thanks
NT


1arryb
User

Jun 16, 2009, 7:55 AM

Post #19 of 36 (16682 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

namishtiwari,

At line 10 you're re-opening temp1 in overwrite mode for each file. Move the open/close statements outside the loop on logFile.

UPDATE: This would have been more apparent to both of us if you properly indented your code.

Cheers,

Larry


(This post was edited by 1arryb on Jun 16, 2009, 7:56 AM)


perlisgreat
Novice

Jun 16, 2009, 8:04 AM

Post #20 of 36 (16680 views)
Re: [1arryb] perl script help [In reply to] Can't Post


In Reply To
namishtiwari,

At line 10 you're re-opening temp1 in overwrite mode for each file. Move the open/close statements outside the loop on logFile.

UPDATE: This would have been more apparent to both of us if you properly indented your code.

Cheers,

Larry


Thank You for your valueable suggestions and your patience. I am really thankful to you as you made me to learn so many things. I changed the mode to append mode and it worked fine.
Still plenty of part is left, so will need your help in near future.

Thanks
NT


perlisgreat
Novice

Jun 16, 2009, 8:41 AM

Post #21 of 36 (16674 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi,

Here is my code snippet


Code
#!/usr/bin/perl 

my $logFile = $ARGV[0];
die "usage: $0 <logFile>" unless $logFile;
die "Logfile $logFile doesn't exist" unless -f "$logFile";
open($log, "<", $logFile) or die "Can't open $logFile for reading.";
print "Processing file $logFile...\n";
my $n = 0;
my $ArcotIDError_Count = 0;
my $QnAError_Count = 0;
my $UPError_Count = 0;
my $OTPError_Count = 0;
my $errorCode;
while( $line = <$log> )
{
$n++;
$line =~ tr/\r\n//d;
if ($line =~ /recvd AA_BIN_MSG_VER_CHG/)
{
while( $line = <$log> )
{
if($line =~ /Handling NSPAdvice for mechanism \[4\]/)
{
while ( $line = <$log> )
{
if ($line =~ /Authentication mechanism returned \[(\d+)\]/)
{
my $errorCode = $1;
print "$errorCode\n";
if ($errorCode != 0 && $errorCode != 1)
{
$ArcotIDError_Count++;
}
}
}
}
}
}
}
print "Total Number Of ArcotID Authentication ErrorCode returned is $ArcotIDError_Count\n";


This code does not give me correct output. Correct output according to my logfile is 142 but it gives me 352.
I want to count it in between these lines


Quote
starting point-
Apricot Native Server: recvd AA_BIN_MSG_VER_CHG
and
ending point-
(Prepared to Send OK) OR (Sending Invalid credential).

I am able to check the beginning statement but how i can check the ending statement, i mentioned .


FishMonger
Veteran / Moderator

Jun 16, 2009, 10:46 AM

Post #22 of 36 (16665 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

I fail to see any valid reason why you need to use the 3 nested while loops. Do you?


FishMonger
Veteran / Moderator

Jun 16, 2009, 1:26 PM

Post #23 of 36 (16659 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

If you strip out the needless nested loops, it becomes much easier to read/understand/maintain.


Code
#!/usr/bin/perl  

use strict;
use warnings;

my $logFile = $ARGV[0] or die "usage: $0 <logFile>";

open my $log, '<', $logFile or die "Can't open '$logFile' for reading. $!";
print "Processing file $logFile...\n";

my $ArcotIDError_Count;

# uncomment this line if you need to skip over the preceeding lines
#while(<$log>) { last if /Handling NSPAdvice for mechanism/ };

while( my $line = <$log> ) {

if ($line =~ /Authentication mechanism returned \[(\d+)\]/) {
my $error_code = $1;
if ($error_code > 1) {
$ArcotIDError_Count++;
printf("line:%5d \t error code:%3d\n", $. , $error_code);
}
}
}
print "Total Number Of ArcotID Authentication ErrorCode returned is $ArcotIDError_Count\n";


Using your attached logfile, this is the output I expected and received.


Code
[root@rkb-2 ~]# ./OTP_Authentication.pl arcotwebfort_29May09 
Processing file arcotwebfort_29May09...
line: 55 error code: 10
line: 57 error code: 10
line: 680 error code: 3
line: 685 error code: 3
line: 727 error code: 10
line: 729 error code: 10
line: 769 error code: 3
line: 784 error code: 3
line: 815 error code: 3
line: 820 error code: 3
line: 999 error code: 3
line: 1004 error code: 3
line: 1464 error code: 3
line: 1469 error code: 3
Total Number Of ArcotID Authentication ErrorCode returned is 14



perlisgreat
Novice

Jun 22, 2009, 5:26 AM

Post #24 of 36 (16600 views)
Re: perl script help [In reply to] Can't Post

Hi,

Here is my piece of code.


Code
#!/usr/bin/perl 

my $logFile = $ARGV[0];

die "usage: $0 <logFile>" unless $logFile;
die "Logfile $logFile doesn't exist" unless -f "$logFile";

my %no_of_questions;
my %timestamp;
open (FP,"<$logFile") or die "unable to open file $!" ;
@records=<FP>;
chomp(@records);

for($index=0;$index<@records;$index++)
{
if($records[$index] =~ /^(.*)INFO:.*Entering QnAModule::authenticate/)
{
#printf "in INFO:.*Entering QnAModule::authenticate\n";
$Time_Stamp = $1;
#die "Some error in search_for_username_and_no_of_questions" if (search_for_username_and_no_of_questions() == -1);
#die "Some error in end_block" if (end_block() == -1);
search_for_username_and_no_of_questions();
end_block();

}
#last;
}




sub search_for_username_and_no_of_questions
{
printf "inside search_for_username_and_no_of_questions\n";
for($index=0;$index<@records;$index++)
{
if ($records[$index] =~ /userName :\[(..*)\]/)
{
#printf "in user name identification \n";
$User_Name = $1;
$User_Name =~ s/\s+//g;
for(;$index<@records;$index++)
{
if ($records[$index] =~ /Set of questions\(bitmap\) selected : (\d+)/)
{
#printf "in set of questions\n";
$Bitmap_Number_For_Questions = $1;
$no_of_questions{$User_Name}=$Bitmap_Number_For_Questions;
$timestamp{$User_Name}=$Time_Stamp;
#printf "DETAIL: LINE NO: $index\n";
#printf "$Time_Stamp,$User_Name,$Bitmap_Number_For_Questions\n";
return 0;
}
}


}
last;
}
}

sub end_block
{
printf "Inside end_block\n";
for($index=0;$index<@records;$index++)
{

if($records[$index] =~ /QNA Step.*AUTH IN PROGRESS/)
{
printf "DETAIL: auth in progress\n";
for(;$index<@records;$index++)
{
if ($records[$index] =~ /ArAuthFrameworkImpl::doPostAuth.*Authentication mechanism returned \[(..*)\] for AuthIdentity \[(..*)\]/)
{
#printf "done one request \n";
#printf "$records[$index]\n";
my $return_code = $1;
my $temp_user = $2;
$temp_user =~ s/\s+//g;
#printf "user = $temp_user\n";
if(exists($no_of_questions{$temp_user}))
{
if ($return_code == 0)
{
printf "$temp_user,$no_of_questions{$temp_user} , $timestamp{$temp_user} , \"Success\" \n";
return 0;
}
elsif ($return_code > 1)
{
printf "$temp_user,$no_of_questions{$temp_user} , $timestamp{$temp_user} , \"Failure\" \n";
return 0;
}
for(;$index<@records;$index++)
{
if ( ($records[$index] =~ /Prepared to Send OK/) || ($records[$index] =~ /Sending Invalid credential/) )
{
#printf "end of file\n";
}
}
}
}
last;
}
}
}
#return -1;
}


here is the sample data-

Code
monday 12:55:66 INFO:Entering QnAModule::authenticate 
monday 12:55:66 userName :[prasanna]
monday 12:55:66 Set of questions(bitmap) selected : 13
monday 12:55:66 QNA Step.*AUTH IN PROGRESS
monday 12:55:66 ArAuthFrameworkImpl::doPostAuth.*Authentication mechanism returned [4] for AuthIdentity [prasanna]
monday 12:55:66 Prepared to Send OK
monday 12:55:66 Sending Invalid credential
monday 12:55:66 INFO:Entering QnAModule::authenticate
monday 12:55:66 userName :[prasanna]
monday 12:55:66 Set of questions(bitmap) selected : 13
monday 12:55:66 QNA Step.*AUTH IN PROGRESS
monday 12:55:66 ArAuthFrameworkImpl::doPostAuth.*Authentication mechanism returned [4] for AuthIdentity [prasanna]
monday 12:55:66 Prepared to Send OK
monday 12:55:66 Sending Invalid credential


when i ran the script on this data i got the actual result ie

Quote
inside search_for_username_and_no_of_questions
Inside end_block
DETAIL: auth in progress
DETAIL: auth in progress


it did not print any data. Kindly give some suggestions.
Thanks
NT

Quote


(This post was edited by namishtiwari on Jun 22, 2009, 8:26 AM)


perlisgreat
Novice

Jun 23, 2009, 7:05 AM

Post #25 of 36 (16562 views)
Re: [namishtiwari] perl script help [In reply to] Can't Post

Hi,

I was able to solve this problem and the code is working fine now. i am not using these statements in my code

Code
 
use strict;
use warnings;

but when i use i got so many warning message like this --

Code
 
Global symbol "$index" requires explicit package name at per_user_qna_detail.pl
line 67.


This error came for all the global symbol.
How i can get rid of this.


(This post was edited by namishtiwari on Jun 23, 2009, 7:08 AM)

First page Previous page 1 2 Next page Last page  View All
 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives