CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
perl one liner - need some advice

 



jeffersno1
Novice

Dec 5, 2011, 8:30 PM

Post #1 of 7 (3881 views)
perl one liner - need some advice Can't Post

Hi All,

Wondered if someone could help me here with a one liner for perl???

What id like to do is run a perl command against a 55GB file (1mnths logs) to substitute 1 character "/" for ","

log file snippet
2011-10-01 00:00:00.861,RADIUS-IU,447555552255,11.91.11.173,apn.net.uk
2011-10-01 00:00:00.863,CCR-U,4,,PGHLD01-1,447555555555,122.22.222.222,apn.net.uk,domestic
2011-10-01 00:00:00.863,DIA_CCR_U,133.11.114.215,447555555555,2202,4115788/78923,IR0,5242880,
2011-10-01 00:00:00.675,DIA_CCR_T,119.11.137.209,447111111116,2200,0/0,IR2,-,
2011-10-01 00:00:00.675,DIA_CCR_T,111.11.137.209,447111111116,2202,422/482,IR2,-,
2011-10-01 00:00:00.693,DIA_CCR_U,111.105.88.11,447912345678,2202,4114441/79882,IR0,5242880,
2011-10-01 00:00:00.773,DIA_CCR_U,11.111.85.88,447512345678,2202,141/173,IR1,10485760,
2011-10-01 00:00:00.634,RADIUS-Stop,447787878787,111.11.131.119,apn.net.uk
2011-10-01 00:00:00.648,logoutByIpAddress,447133333396,111.91.111.109,ok


what id like to do is substitute the "/" for a comma but only on lines where there is a DIA_CCR. Does anyone know if this is possible from the command line?? Also the reason id like to use perl or sed is because its not so CPU or memory hungry and this is being run against a live server...

I've already used some one liners to substitute the date format, the reason I'm doing this is so i can load it into SQL and run queries against the data.

Here are a few commands I've already ran on the file


Code
sed 's/:/./3' file1 > file2 

perl -pe 's/_/ /' file1 > file2


any help appreciated.

Many thanks

Jeffers


wickedxter
User

Dec 6, 2011, 3:56 AM

Post #2 of 7 (3867 views)
Re: [jeffersno1] perl one liner - need some advice [In reply to] Can't Post


Code
#open file read one line at a time 
# $line is a line of the file
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);

Code
 
it would be better to read one line of the file at a time so its not consuming 55gb of memory trying to slurp the whole file


jeffersno1
Novice

Dec 7, 2011, 1:28 PM

Post #3 of 7 (3842 views)
Re: [wickedxter] perl one liner - need some advice [In reply to] Can't Post

Hi wickedxter

thanks for the recommendation, it works on small files but large 20GB files it runs out of memory.

Is there anything i can do to stop it using so much memory??

thanks


Code
20> more oct.pl  
#!/usr/bin/perl -w

#use strict;
use POSIX;

open ( OUT, ">./results.csv");

$data = "/tmp/tester";

open(LOGFILE,"$data") or die("Could not open log file.");
foreach $line (<LOGFILE>) {
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);
print OUT "$line";
}
close(LOGFILE);
close(OUT);



BillKSmith
Veteran

Dec 7, 2011, 1:59 PM

Post #4 of 7 (3835 views)
Re: [jeffersno1] perl one liner - need some advice [In reply to] Can't Post


Code
perl -pe 's!(DIA_CCR.*)/!$1,!' file1 > file2



This does not seem much different from the example you say did work.
Good Luck,
Bill

(This post was edited by BillKSmith on Dec 7, 2011, 7:20 PM)


jeffersno1
Novice

Dec 7, 2011, 2:09 PM

Post #5 of 7 (3833 views)
Re: [BillKSmith] perl one liner - need some advice [In reply to] Can't Post

You've not read the whole thread Bill,

yes the script works but the whole point is to run the script against large files 20GB minimum, this solution will only work with files smaller than current memory on the system.

thanks


jeffersno1
Novice

Dec 7, 2011, 2:15 PM

Post #6 of 7 (3831 views)
Re: [jeffersno1] perl one liner - need some advice [In reply to] Can't Post

Bill

Please accept my apologies, i didnt even see your code and suggestion.

That was just what i was after, !!!! and there is me saying you havent read the whole thread Crazy

Appreciate you help, and sorry again

Your a gent Blush


wickedxter
User

Dec 7, 2011, 5:34 PM

Post #7 of 7 (3793 views)
Re: [jeffersno1] perl one liner - need some advice [In reply to] Can't Post


Code
 

#!/usr/bin/perl -w

#use strict;
use POSIX;

open ( OUT, ">./results.csv");

$data = "/tmp/tester";

open(LOGFILE,"$data") or die("Could not open log file.");
while ($line = <LOGFILE>) {
$line =~ s/\//,/ if($line =~ /DIA_CCR_U/);
print OUT "$line";
#why are you printing this and creating anouther large file?
}
close(LOGFILE);
close(OUT);


Code
 
Try useing a while loop rather then a foreach, after reading around the web found and article that explained some detail about the deference of the two and the while loop worked the fastest for large files.
http://wildpointer.org/2009/11/15/efficient-reads-on-perl-file-handles-while-vs-foreach/


(This post was edited by wickedxter on Dec 7, 2011, 5:36 PM)

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives