CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
Extract Data from Logs

 



Amrit
Novice

Jan 6, 2012, 5:00 AM

Post #1 of 4 (1398 views)
Extract Data from Logs Can't Post

Hi ,

I have a large log file with lines as follows :

Source = da1
Products = [p=product1]
FieldSet =
Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value1
dateTime1
start1
ab1
] (processRequest)
Jan 04 10:48:46 akd haijkosa jkalkjd klajd

Source = da2
Products = [p=product2]
FieldSet =
Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value2
dateTime2
start2
ab2
] (processRequest)
Jan 04 10:48:46 akd haijkosa jkalkjd klajd

I need data in the following format

Source,Product,Field
da1,[p=product1],value1 dateTime1 start1 ab1
da2,[p=product2],value2 dateTime2 start2 ab2

I need to extract the data from file from "Source = ","Products =","Fields ="

Note : Fields columns are spread over multiline as below.I need it as value1 dateTime1 start1 ab1 in csv file in fields column.

Fields:
value1
dateTime1
start1
ab1


BillKSmith
Veteran

Jan 6, 2012, 7:18 AM

Post #2 of 4 (1396 views)
Re: [Amrit] Extract Data from Logs [In reply to] Can't Post

As long as every block of data is in the same format, each itteration of a while loop can read all the data in a block, reformat it, and write one output record. Some care is required on how to end the loop. The following code assumes that the blank line you show between blocks is a separator which does not occur after the last block.

Note that the general structure applies even if I did not understand field correctly.


Code
  

use strict;
use warnings;
print "Source,Product,Field\n";
while (1) {
chomp( my $source = <DATA> );
$source =~ s/.+= (\S+)/$1/;
chomp( my $product = <DATA> );
$product =~ s/[^=]+= (\S+)/$1/;
<DATA>; # FieldSet
<DATA>; # Request
<DATA>; # FiedlSetType
<DATA>; # FieldSet Option
<DATA>; # Field ID
<DATA>; # Fields
chomp( my $value = <DATA> );
chomp( my $datetime = <DATA> );
chomp( my $start = <DATA> );
chomp( my $ab = <DATA> );
<DATA>; # ProcessRequest
<DATA>; #
print "$source,$product,$value $datetime $start $ab\n";
last if(!<DATA>); # record Separator
}
__DATA__
Source = da1
Products = [p=product1]
FieldSet =
Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value1
dateTime1
start1
ab1
] (processRequest)
Jan 04 10:48:46 akd haijkosa jkalkjd klajd

Source = da2
Products = [p=product2]
FieldSet =
Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value2
dateTime2
start2
ab2
] (processRequest)
Jan 04 10:48:46 akd haijkosa jkalkjd klajd

Good Luck,
Bill


Amrit
Novice

Jan 9, 2012, 12:04 AM

Post #3 of 4 (1335 views)
Re: [BillKSmith] Extract Data from Logs [In reply to] Can't Post

My bad.Your reply works great Cool

I forgot to mention the the fields value may differ and it sometimes mite not contain anything.

Source = da2
Products = [p=product2]
FieldSet = Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value2
dateTime2
start2
ab2
I
forgot
asss1
aad1
pla
qaw
ww
] (processRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd

rocessRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd
processRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd
processRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd


Source = da2
Products = [p=product2]
FieldSet = Request :
FieldSetType: Pp
FieldSet Option:
Field Id: 0
Fields:
value2
] (processRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd

rocessRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd
processRequest) Jan 04 10:48:46 akd haijkosa jkalkjd klajd


Source,Product,Field
da1,[p=product1],value1 dateTime1 start1 ab1
da2,[p=product2],value2,dateTime2,start2,ab2,I,forgot,asss1,aad1, pla,qaw,ww
da2,[p=product2],value2

sometimes it may be 0,1 or any number of value.

Your help if higly apprecited


(This post was edited by Amrit on Jan 9, 2012, 12:09 AM)


BillKSmith
Veteran

Jan 9, 2012, 12:25 PM

Post #4 of 4 (1289 views)
Re: [Amrit] Extract Data from Logs [In reply to] Can't Post

Assume blank line between input blocks. Set the special variable $/ (refer: perldoc perlvar) to a null string. Read a block. If it does not contain all the keywords, go to next block. Extract the data with regular expressions. Using regular expressions with multi-line strings requires extra care. You probably need both the /m and /s options.
Good Luck,
Bill

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives