Home: Perl Programming Help: Intermediate:
perl one liner - need some advice



jeffersno1
Novice

Dec 5, 2011, 8:30 PM


Views: 4502
perl one liner - need some advice

Hi All,

Wondered if someone could help me here with a one liner for perl???

What id like to do is run a perl command against a 55GB file (1mnths logs) to substitute 1 character "/" for ","

log file snippet
2011-10-01 00:00:00.861,RADIUS-IU,447555552255,11.91.11.173,apn.net.uk
2011-10-01 00:00:00.863,CCR-U,4,,PGHLD01-1,447555555555,122.22.222.222,apn.net.uk,domestic
2011-10-01 00:00:00.863,DIA_CCR_U,133.11.114.215,447555555555,2202,4115788/78923,IR0,5242880,
2011-10-01 00:00:00.675,DIA_CCR_T,119.11.137.209,447111111116,2200,0/0,IR2,-,
2011-10-01 00:00:00.675,DIA_CCR_T,111.11.137.209,447111111116,2202,422/482,IR2,-,
2011-10-01 00:00:00.693,DIA_CCR_U,111.105.88.11,447912345678,2202,4114441/79882,IR0,5242880,
2011-10-01 00:00:00.773,DIA_CCR_U,11.111.85.88,447512345678,2202,141/173,IR1,10485760,
2011-10-01 00:00:00.634,RADIUS-Stop,447787878787,111.11.131.119,apn.net.uk
2011-10-01 00:00:00.648,logoutByIpAddress,447133333396,111.91.111.109,ok


what id like to do is substitute the "/" for a comma but only on lines where there is a DIA_CCR. Does anyone know if this is possible from the command line?? Also the reason id like to use perl or sed is because its not so CPU or memory hungry and this is being run against a live server...

I've already used some one liners to substitute the date format, the reason I'm doing this is so i can load it into SQL and run queries against the data.

Here are a few commands I've already ran on the file


Code
sed 's/:/./3' file1 > file2 

perl -pe 's/_/ /' file1 > file2


any help appreciated.

Many thanks

Jeffers


wickedxter
User

Dec 6, 2011, 3:56 AM


Views: 4488
Re: [jeffersno1] perl one liner - need some advice


Code
#open file read one line at a time 
# $line is a line of the file
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);

Code
 
it would be better to read one line of the file at a time so its not consuming 55gb of memory trying to slurp the whole file


jeffersno1
Novice

Dec 7, 2011, 1:28 PM


Views: 4463
Re: [wickedxter] perl one liner - need some advice

Hi wickedxter

thanks for the recommendation, it works on small files but large 20GB files it runs out of memory.

Is there anything i can do to stop it using so much memory??

thanks


Code
20> more oct.pl  
#!/usr/bin/perl -w

#use strict;
use POSIX;

open ( OUT, ">./results.csv");

$data = "/tmp/tester";

open(LOGFILE,"$data") or die("Could not open log file.");
foreach $line (<LOGFILE>) {
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);
print OUT "$line";
}
close(LOGFILE);
close(OUT);



BillKSmith
Veteran

Dec 7, 2011, 1:59 PM


Views: 4456
Re: [jeffersno1] perl one liner - need some advice


Code
perl -pe 's!(DIA_CCR.*)/!$1,!' file1 > file2



This does not seem much different from the example you say did work.
Good Luck,
Bill

(This post was edited by BillKSmith on Dec 7, 2011, 7:20 PM)


jeffersno1
Novice

Dec 7, 2011, 2:09 PM


Views: 4454
Re: [BillKSmith] perl one liner - need some advice

You've not read the whole thread Bill,

yes the script works but the whole point is to run the script against large files 20GB minimum, this solution will only work with files smaller than current memory on the system.

thanks


jeffersno1
Novice

Dec 7, 2011, 2:15 PM


Views: 4452
Re: [jeffersno1] perl one liner - need some advice

Bill

Please accept my apologies, i didnt even see your code and suggestion.

That was just what i was after, !!!! and there is me saying you havent read the whole thread Crazy

Appreciate you help, and sorry again

Your a gent Blush


wickedxter
User

Dec 7, 2011, 5:34 PM


Views: 4414
Re: [jeffersno1] perl one liner - need some advice


Code
 

#!/usr/bin/perl -w

#use strict;
use POSIX;

open ( OUT, ">./results.csv");

$data = "/tmp/tester";

open(LOGFILE,"$data") or die("Could not open log file.");
while ($line = <LOGFILE>) {
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);
print OUT "$line";
#why are you printing this and creating anouther large file?
}
close(LOGFILE);
close(OUT);


Code
 
Try useing a while loop rather then a foreach, after reading around the web found and article that explained some detail about the deference of the two and the while loop worked the fastest for large files.
http://wildpointer.org/2009/11/15/efficient-reads-on-perl-file-handles-while-vs-foreach/


(This post was edited by wickedxter on Dec 7, 2011, 5:36 PM)