CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
collect log files remotely and store as 1 file

 



jeffersno1
Novice

Nov 1, 2012, 4:05 PM

Post #1 of 10 (3463 views)
collect log files remotely and store as 1 file Can't Post

Hi guys,

I've been asked to write a script by the company i work for but i'm having trouble getting the data.

details are:
script will run from logserv daily via cron and collect files from 4 different servers for the previous day
collects files from /home/sys1/event_logs_201211010000.audit

Now i can put the public key on each machine, no issues there, and just use a system call to retrieve the file, my problem is, if the application restart then another files with a different time stamp will be generated.
eg
/home/sys1/event_log_2012110172314.audit

Ideally id like all 4 files (1 from each server from the previous day) to be collected and stored in 1 file on the logserv in a given directory.

Can anyone offer any suggestions as to how i get round the multiple files should the app restart?

Many thanks

Jeffers


Laurent_R
Veteran / Moderator

Nov 2, 2012, 3:48 AM

Post #2 of 10 (3458 views)
Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to] Can't Post

If you do a glob with a string having only the date, such as:


Code
my @list_of_files = glob "/home/sys1/event_log_20121101*";


you will get a list of files with that date.


jeffersno1
Novice

Nov 2, 2012, 1:28 PM

Post #3 of 10 (3453 views)
Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to] Can't Post

Hi Laurent_R,

Thanks for the reply. Ive seen glob used a few times, how can i use this to copy the files from a remote system?

Thanks


Laurent_R
Veteran / Moderator

Nov 3, 2012, 1:33 AM

Post #4 of 10 (3382 views)
Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to] Can't Post

You first said you have no problem to retrieve the files and now ask how to do it. You don't say enough about your environment, but I guess that you can use FTP to transfer your files (see the Net::FTP module on the CPAN). Since you mentioned something about secured connection, you might want to take a look at the Net-FTPSSL module (http://search.cpan.org/~kral/Net-FTPSSL-0.04/FTPSSL.pm) or some other FTP modules on the CPAN.


jeffersno1
Novice

Nov 5, 2012, 4:44 AM

Post #5 of 10 (3221 views)
Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to] Can't Post

Hi Laurent,

Sorry, let me explain a bit further, I need to write a script that collects files from 4 different machines and store them as 1, to get the data i connect to each server a store the names of the files into an array.

Code
use POSIX 'strftime'; 
my $date = strftime("%Y%m%d", localtime(time - 86400));

@sys1list = system ("/usr/bin/ssh admin\@sys1 '/bin/ls -1 /$main_dir/event_log_$date??????.audit'");

Then i connect again and collect all the files in the array which works

But i cant get the contents of the files into 1 file, using the code below i just get the elements of the array Unsure


Code
foreach (@sys1list){ 
$_ =~ s/\/\//\//g;
print " LINE = $_\n";
$srcfile=$_;
$dstfile=basename $_;
system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`);
system (`/bin/cat /home/sys1/data/'$srcfile' >> /home/sys1/data/log_$date`); // doesnt work
/// how can i get the data from the files instead of the array names ??
}

Once I've collected the data from all 4 machines i would like to cat all of these files into 1 file

I've tried using glob but couldn't get it to work. I've managed to collect the data but am struggling to get these files into 1. I also don't like the idea of connecting twice, any suggestions?

Many thanks for your help!

Jeffers


(This post was edited by jeffersno1 on Nov 5, 2012, 5:08 AM)


Laurent_R
Veteran / Moderator

Nov 5, 2012, 10:27 AM

Post #6 of 10 (3206 views)
Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to] Can't Post

You have this:


Code
        system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`);  
system (`/bin/cat /home/sys1/data/'$srcfile' >> /home/sys1/data/log_$date`);


I think you have a mistake in your file names. Your local copy of the file goes into $dstname and then you are trying to to cat $srcfile into your log_$date file. I believe you should cat $dstname.

In addition, I don't think you should use single quotes around the variables containing the file names in your commands because the name of the file will not be interpolated.


rovf
Veteran

Nov 6, 2012, 1:30 AM

Post #7 of 10 (3173 views)
Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to] Can't Post


Quote
In addition, I don't think you should use single quotes around the variables containing the file names in your commands because the name of the file will not be interpolated.


The real problem are the backquotes:


Quote

Code
system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`);



They should be double-quotes; I guess this is just a typo, because backquotes don't make sense here. Hence we end up with:


Code
system ("/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'");


and in this code, the single quotes do make sense: $srcfile is interpolated (because the string is in double quotes), and something like


Code
/usr/bin/scp admin\@sys1:'my source file' ...


is passed to the shell, allowing filenames containing spaces, wildcard characters (as part of the name) and similar weird stuff.


jeffersno1
Novice

Nov 7, 2012, 10:48 AM

Post #8 of 10 (3061 views)
Re: [rovf] collect log files remotely and store as 1 file [In reply to] Can't Post

Hi Rovf,

Thanks for the tip, I've made the amendment and have nearly got it finished, 1 last niggle!!!!!!

The below works brilliantly but if i copy a file with the same date/time from more than 1 host it will get overwritten. What I'm trying to do now is add the $host to the beginning of the file saved Unsure I've tried various ways but cant get it to work. The dstfile and basename variables from "use File:Basename" are screwing it up Mad

Code
#!/usr/bin/perl 

use warnings;
use POSIX 'strftime';
use File::Basename;
my $date = strftime("%Y%m%d", localtime(time - 86400));

my $main_dir = '/home/core/sys/logs';
my $sis_main_dir = '/home/otpuser/pdf';

$corelist = "sys1 sys2 sys3 sys4";
@cores=split(' ',$corelist);

foreach (@cores)
{
$host=$_;
@array = command_to_array("/usr/bin/ssh core\@$host '/bin/ls -1 /$main_dir/log_$date??????.audit'");
print "host = $host\n";
foreach (@array){
$_ =~ s/\/\//\//g;
print " LINE = $_\n";
$srcfile=$_;
$dstfile=basename $_;
system ("/usr/bin/scp core\@$host:'$srcfile' /home/otpuser/pdf/RA_data/'$dstname'");
}
}



Any suggestions?

Thanks again for your help, its


Laurent_R
Veteran / Moderator

Nov 7, 2012, 11:01 AM

Post #9 of 10 (3058 views)
Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to] Can't Post

I did not tryn but probably this would work:


Code
@array = command_to_array("/usr/bin/ssh core\@$host '/bin/ls -1 /$main_dir/log_$date$_.audit'");


BTW, this:


Code
$corelist = "sys1 sys2 sys3 sys4";  
@cores=split(' ',$corelist);


could be written:


Code
my @cores = qw (sys1 sys2 sys3 sys4);



rovf
Veteran

Nov 7, 2012, 12:18 PM

Post #10 of 10 (3049 views)
Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to] Can't Post

I think the command generated is not what you expect it to be. When doing this type of stuff, I rarely use




Code
system("My Command $goes $here");


but something like (please adapt it to your taste):


Code
sub SYSTEM($) { 
my $command=shift;
chomp $command;
print "EXECUTING: [$command]\n" if $i_am_still_debugging;
my $retval = system($command);
print "Exit Code: ",($retval >> 8), ", Return Code:",$retval & 0xff,"\n"
if $i_am_still_debugging;
$retval
}

....

SYSTEM("My Command $goes $here");


This makes it easier to see what is going on.

BTW, you don't need $corelist in your code. You can write instead:


Code
my @cores = qw(sys1 sys2 sys3 sys4);


Finally, do your self a favour and make your program run under


Code
use strict;


 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives