Home: Need a Custom or Prewritten Perl Program?: I need a program that...:
Extract specific data (user ID) from log file



vdthoughts
New User

Nov 5, 2013, 2:36 AM


Views: 14552
Extract specific data (user ID) from log file

I would like to extract the user ID's only from the log file. I have copied the sample code below. All the user ID's are inbetween the text "ID principal: " & ", ". Later i would like to use the data to be copied into an excel sheet to create a chart. Can anybody help me to write a perl script for the same. I have never done some complex task on perl prog till now.
---------------------------------------------------------
Sample log file code:

Tue Nov 5 09:00:00 2013
SI Floating Licenses: 14
ID principal: IDB2COB, ID principal: kdk4oe, ID principal: oij7nde, ID principal: qay8plm, ID principal: wsx9okn, ID principal: LNA6COB, ID principal: lak3kor, ID principal: deveas
SI Seat Licenses: 7
ID principal: ISN1COB, ID principal: kao7wbe, ID principal: kor4sds, ID principal: HSC4WE, ID principal: usr4cint, ID principal: SUU4COB, ID principal: birr
IM Floating Licenses: 8
ID principal: tram2we, ID principal: IDB2COB, ID principal: kao7wbe, ID principal: HSC4WE, ID principal: VBN1KOR, ID principal: adf3rtz, ID principal: MAG4KOR, ID principal: ASI3COB
IM Seat Licenses: 2
ID principal: kor4sds, ID principal: ribbs


Tue Nov 5 09:30:00 2013
SI Floating Licenses: 10
ID principal: IDB2COB, ID principal: kdk4oe, ID principal: oij7nde, ID principal: qay8plm, ID principal: wsx9okn, ID principal: LNA6COB, ID principal: lak3kor, ID principal: deveas
SI Seat Licenses: 7
ID principal: ISN1COB, ID principal: kao7wbe, ID principal: kor4sds, ID principal: HSC4WE, ID principal: usr4cint, ID principal: SUU4COB, ID principal: birr
IM Floating Licenses: 8
ID principal: tram2we, ID principal: IDB2COB, ID principal: kao7wbe, ID principal: HSC4WE, ID principal: VBN1KOR, ID principal: adf3rtz, ID principal: MAG4KOR, ID principal: ASI3COB
IM Seat Licenses: 2
ID principal: kor4sds, ID principal: ribbs


Tue Nov 5 10:00:00 2013
SI Floating Licenses: 7
ID principal: IDB2COB, ID principal: kdk4oe, ID principal: oij7nde, ID principal: qay8plm, ID principal: wsx9okn, ID principal: LNA6COB, ID principal: lak3kor, ID principal: deveas
SI Seat Licenses: 7
ID principal: ISN1COB, ID principal: kao7wbe, ID principal: kor4sds, ID principal: HSC4WE, ID principal: usr4cint, ID principal: SUU4COB, ID principal: birr
IM Floating Licenses: 8
ID principal: tram2we, ID principal: IDB2COB, ID principal: kao7wbe, ID principal: HSC4WE, ID principal: VBN1KOR, ID principal: adf3rtz, ID principal: MAG4KOR, ID principal: ASI3COB
IM Seat Licenses: 2
ID principal: kor4sds, ID principal: ribbs
-----------------------------------------------------------


FishMonger
Veteran / Moderator

Nov 5, 2013, 6:13 AM


Views: 14545
Re: [vdthoughts] Extract specific data (user ID) from log file

What have you tried?

What errors/warnings are you getting?

Please post the code you're testing.


BillKSmith
Veteran

Nov 5, 2013, 8:19 AM


Views: 14533
Re: [vdthoughts] Extract specific data (user ID) from log file

You can meet your minimum requirements without a "complex task."

Read data by logical record. ( set $/ = q() refer: perldoc perlvar)

For each record:

Use one regex to extract the date (You will probably need it for your table)

Use a second regex to extract all the ID's in that record

Write both to one record of a .csv file (Which Excel can read.)


Use Excel to do anything else you want.

You would then be ready to attempt to use a module such as Excel::Writer::XLSL to create the spreadsheet with Perl.
Good Luck,
Bill


vdthoughts
New User

Nov 5, 2013, 8:21 AM


Views: 14532
Re: [FishMonger] Extract specific data (user ID) from log file

Code I have written till now.
Since I am very new to Perl am very sorry for my coding standards.

#Extract specific data (user ID) from log file
[perl] #!/usr/bin/perl

$LOGFILE = "C:\\Users\\vut2kor\\Desktop\\lic_usage.log";
open(LOGFILE) or die("Could not open log file.");
foreach $line (<LOGFILE>)
{
@matches =~ /ID principal:\s+(\w+),/g;
}
close(LOGFILE);
[/perl]

Compilation error:
C:\Users\vut2kor>perl -wc C:\Users\vut2kor\Desktop\Extract.pl
Applying pattern match (m//) to @array will act on scalar(@array) at C:\Users\vut2kor\Desktop\Extract.pl line 9.
Name "main::line" used only once: possible typo at C:\Users\vut2kor\Desktop\Extract.pl line 7.
Name "main::matches" used only once: possible typo at C:\Users\vut2kor\Desktop\Extract.pl line 9.
C:\Users\vut2kor\Desktop\Extract.pl syntax OK


BillKSmith
Veteran

Nov 5, 2013, 11:31 AM


Views: 14523
Re: [vdthoughts] Extract specific data (user ID) from log file

Try to understand your warning messages.


Quote
Applying pattern match (m//) to @array will act on scalar(@array) at C:\Users\vut2kor\Desktop\Extract.pl line 9.


The match operator is meant to operate on a string. You have specified an array (@matches). Perl will convert the array to a scalar (number of elements in the array) This is definitly not what you intend!


Quote
Name "main::line" used only once: possible typo at C:\Users\vut2kor\Desktop\Extract.pl line 7.


The variable $line (in package main) is used only once (on line 7)


Quote
Name "main::matches" used only once: possible typo at C:\Users\vut2kor\Desktop\Extract.pl line 9.


Same story for $matches on line 9.


Quote
C:\Users\vut2kor\Desktop\Extract.pl syntax OK


Your code is valid Perl, but given the warnings above, it probably will not do what you intend.


You should always use the following two statements at the beginning of your perl scripts:


Code
use strict; 
use warnings;


It may seem like an inconvenience that they require you to declare (with my) all your variables. The effort will be repaid with less debug time.

Your open statements do not do what you think. Always use the three argument form (Refer: perldoc -f open).

Always use lexical filehandles.


Code
open $LOGFILE, '<', "C:\\Users\\vut2kor\\Desktop\\lic_usage.log" 
or die "Could not open log file.";


Your regex will fail when an ID comes at the end of a line (no comma).



Note: [perl][/perl] tags do not work on this site. Use code instead.
Good Luck,
Bill

(This post was edited by BillKSmith on Nov 5, 2013, 9:22 PM)


BillKSmith
Veteran

Nov 6, 2013, 7:41 AM


Views: 14508
Re: [BillKSmith] Extract specific data (user ID) from log file

Here is the "front end" program that I described in post 3.
I have attached the input and output files from my test.

Code
use strict; 
use warnings;
use English;
my $logfile = "lic_usage.log"; # Changed for my environment
open(my $LOG_IN, '<', $logfile) or die("Could not open log file.");
open my $TABLE_OUT, '>', 'excel_table.csv' or die "cannot open excel_table\n";

{
local $INPUT_RECORD_SEPARATOR = q();
local $OUTPUT_FIELD_SEPARATOR = q(,);
local $LIST_SEPARATOR = q(,);
while (my $log_record = <$LOG_IN>) {
my ($date) = $log_record =~ m/^ ( .+? ) $/xms;
my @IDs = $log_record =~ m/(?<=ID\sprincipal:\s) [^,\n]+ /xmsg;
print {$TABLE_OUT} $date, "@IDs\n";
}
}
close $TABLE_OUT;
close $LOG_IN;

Good Luck,
Bill
Attachments: lic_usage.log (2.02 KB)
  excel_table.csv (0.64 KB)
  excel_table.xlsx (9.24 KB)