Perl Guru Forums: Perl Programming Help: Intermediate: perl one liner

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
perl one liner - need some advice

jeffersno1
Novice

Dec 5, 2011, 8:30 PM

Post #1 of 7 (3558 views)

perl one liner - need some advice	Can't Post

Hi All,

Wondered if someone could help me here with a one liner for perl???

What id like to do is run a perl command against a 55GB file (1mnths logs) to substitute 1 character "/" for ","

log file snippet
2011-10-01 00:00:00.861,RADIUS-IU,447555552255,11.91.11.173,apn.net.uk
2011-10-01 00:00:00.863,CCR-U,4,,PGHLD01-1,447555555555,122.22.222.222,apn.net.uk,domestic
2011-10-01 00:00:00.863,DIA_CCR_U,133.11.114.215,447555555555,2202,4115788/78923,IR0,5242880,
2011-10-01 00:00:00.675,DIA_CCR_T,119.11.137.209,447111111116,2200,0/0,IR2,-,
2011-10-01 00:00:00.675,DIA_CCR_T,111.11.137.209,447111111116,2202,422/482,IR2,-,
2011-10-01 00:00:00.693,DIA_CCR_U,111.105.88.11,447912345678,2202,4114441/79882,IR0,5242880,
2011-10-01 00:00:00.773,DIA_CCR_U,11.111.85.88,447512345678,2202,141/173,IR1,10485760,
2011-10-01 00:00:00.634,RADIUS-Stop,447787878787,111.11.131.119,apn.net.uk
2011-10-01 00:00:00.648,logoutByIpAddress,447133333396,111.91.111.109,ok

what id like to do is substitute the "/" for a comma but only on lines where there is a DIA_CCR. Does anyone know if this is possible from the command line?? Also the reason id like to use perl or sed is because its not so CPU or memory hungry and this is being run against a live server...

I've already used some one liners to substitute the date format, the reason I'm doing this is so i can load it into SQL and run queries against the data.

Here are a few commands I've already ran on the file

Code

sed 's/:/./3' file1 > file2 
 
perl -pe 's/_/ /' file1 > file2

any help appreciated.

Many thanks

Jeffers

wickedxter
User

Dec 6, 2011, 3:56 AM

Post #2 of 7 (3544 views)

Re: [jeffersno1] perl one liner - need some advice [In reply to]	Can't Post

Code

#open file read one line at a time 
# $line is a line of the file 
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/); 
Code  
it would be better to read one line of the file at a time so its not consuming 55gb of memory trying to slurp the whole file

jeffersno1
Novice

Dec 7, 2011, 1:28 PM

Post #3 of 7 (3519 views)

Re: [wickedxter] perl one liner - need some advice [In reply to]	Can't Post

Hi wickedxter

thanks for the recommendation, it works on small files but large 20GB files it runs out of memory.

Is there anything i can do to stop it using so much memory??

thanks

Code

20> more oct.pl  
#!/usr/bin/perl -w 
  
#use strict; 
use POSIX; 
  
open ( OUT, ">./results.csv"); 
  
$data = "/tmp/tester"; 
  
open(LOGFILE,"$data") or die("Could not open log file."); 
foreach $line (<LOGFILE>) { 
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);  
print OUT "$line"; 
} 
close(LOGFILE); 
close(OUT);

BillKSmith
Veteran

Dec 7, 2011, 1:59 PM

Post #4 of 7 (3512 views)

Re: [jeffersno1] perl one liner - need some advice [In reply to]	Can't Post

Code

perl -pe 's!(DIA_CCR.*)/!$1,!' file1 > file2

This does not seem much different from the example you say did work.
Good Luck,
Bill

(This post was edited by BillKSmith on Dec 7, 2011, 7:20 PM)

jeffersno1
Novice

Dec 7, 2011, 2:09 PM

Post #5 of 7 (3510 views)

Re: [BillKSmith] perl one liner - need some advice [In reply to]	Can't Post

You've not read the whole thread Bill,

yes the script works but the whole point is to run the script against large files 20GB minimum, this solution will only work with files smaller than current memory on the system.

thanks

jeffersno1
Novice

Dec 7, 2011, 2:15 PM

Post #6 of 7 (3508 views)

Re: [jeffersno1] perl one liner - need some advice [In reply to]	Can't Post

Bill

Please accept my apologies, i didnt even see your code and suggestion.

That was just what i was after, !!!! and there is me saying you havent read the whole thread Crazy

Appreciate you help, and sorry again

Your a gent Blush

wickedxter
User

Dec 7, 2011, 5:34 PM

Post #7 of 7 (3470 views)

Re: [jeffersno1] perl one liner - need some advice [In reply to]	Can't Post

Code

 
 
#!/usr/bin/perl -w  
   
#use strict;  
use POSIX;  
   
open ( OUT, ">./results.csv");  
   
$data = "/tmp/tester";  
   
open(LOGFILE,"$data") or die("Could not open log file.");  
while ($line = <LOGFILE>) {  
    $line =~ s/\//,/ if($line =~ /DIA_CCR_U/);   
print OUT "$line";  
#why are you printing this and creating anouther large file? 
}  
close(LOGFILE);  
close(OUT); 
 
Code  
Try useing a while loop rather then a foreach, after reading around the web found and article that explained some detail about the deference of the two and the while loop worked the fastest for large files. 
http://wildpointer.org/2009/11/15/efficient-reads-on-perl-file-handles-while-vs-foreach/ 


	

(This post was edited by wickedxter on Dec 7, 2011, 5:36 PM)

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives