Perl Guru Forums: Perl Programming Help: Intermediate: collect log files remotely and store as 1 file

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
collect log files remotely and store as 1 file

jeffersno1
Novice

Nov 1, 2012, 4:05 PM

Post #1 of 10 (6511 views)

collect log files remotely and store as 1 file	Can't Post

Hi guys,

I've been asked to write a script by the company i work for but i'm having trouble getting the data.

details are:
script will run from logserv daily via cron and collect files from 4 different servers for the previous day
collects files from /home/sys1/event_logs_201211010000.audit

Now i can put the public key on each machine, no issues there, and just use a system call to retrieve the file, my problem is, if the application restart then another files with a different time stamp will be generated.
eg
/home/sys1/event_log_2012110172314.audit

Ideally id like all 4 files (1 from each server from the previous day) to be collected and stored in 1 file on the logserv in a given directory.

Can anyone offer any suggestions as to how i get round the multiple files should the app restart?

Many thanks

Jeffers

Laurent_R
Veteran / Moderator

Nov 2, 2012, 3:48 AM

Post #2 of 10 (6506 views)

Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to]	Can't Post

If you do a glob with a string having only the date, such as:

Code

my @list_of_files = glob "/home/sys1/event_log_20121101*";

you will get a list of files with that date.

jeffersno1
Novice

Nov 2, 2012, 1:28 PM

Post #3 of 10 (6501 views)

Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to]	Can't Post

Hi Laurent_R,

Thanks for the reply. Ive seen glob used a few times, how can i use this to copy the files from a remote system?

Thanks

Laurent_R
Veteran / Moderator

Nov 3, 2012, 1:33 AM

Post #4 of 10 (6430 views)

Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to]	Can't Post

You first said you have no problem to retrieve the files and now ask how to do it. You don't say enough about your environment, but I guess that you can use FTP to transfer your files (see the Net::FTP module on the CPAN). Since you mentioned something about secured connection, you might want to take a look at the Net-FTPSSL module (http://search.cpan.org/~kral/Net-FTPSSL-0.04/FTPSSL.pm) or some other FTP modules on the CPAN.

jeffersno1
Novice

Nov 5, 2012, 4:44 AM

Post #5 of 10 (6269 views)

Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to]	Can't Post

Hi Laurent,

Sorry, let me explain a bit further, I need to write a script that collects files from 4 different machines and store them as 1, to get the data i connect to each server a store the names of the files into an array.

Code

use POSIX 'strftime'; 
my $date = strftime("%Y%m%d", localtime(time - 86400)); 
 
@sys1list = system ("/usr/bin/ssh admin\@sys1 '/bin/ls -1 /$main_dir/event_log_$date??????.audit'");

Then i connect again and collect all the files in the array which works

But i cant get the contents of the files into 1 file, using the code below i just get the elements of the array Unsure

Code

foreach (@sys1list){ 
        $_ =~ s/\/\//\//g; 
        print " LINE = $_\n"; 
        $srcfile=$_; 
        $dstfile=basename $_; 
        system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`); 
        system (`/bin/cat /home/sys1/data/'$srcfile' >> /home/sys1/data/log_$date`);   // doesnt work 
        /// how can i get the data from the files instead of the array names ?? 
}

Once I've collected the data from all 4 machines i would like to cat all of these files into 1 file

I've tried using glob but couldn't get it to work. I've managed to collect the data but am struggling to get these files into 1. I also don't like the idea of connecting twice, any suggestions?

Many thanks for your help!

Jeffers

(This post was edited by jeffersno1 on Nov 5, 2012, 5:08 AM)

Laurent_R
Veteran / Moderator

Nov 5, 2012, 10:27 AM

Post #6 of 10 (6254 views)

Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to]	Can't Post

You have this:

Code

        system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`);  
        system (`/bin/cat /home/sys1/data/'$srcfile' >> /home/sys1/data/log_$date`);

I think you have a mistake in your file names. Your local copy of the file goes into $dstname and then you are trying to to cat $srcfile into your log_$date file. I believe you should cat $dstname.

In addition, I don't think you should use single quotes around the variables containing the file names in your commands because the name of the file will not be interpolated.

rovf
Veteran

Nov 6, 2012, 1:30 AM

Post #7 of 10 (6221 views)

Re: [Laurent_R] collect log files remotely and store as 1 file [In reply to]	Can't Post

Quote

In addition, I don't think you should use single quotes around the variables containing the file names in your commands because the name of the file will not be interpolated.

The real problem are the backquotes:

Quote

Code

system (`/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'`);

They should be double-quotes; I guess this is just a typo, because backquotes don't make sense here. Hence we end up with:

Code

system ("/usr/bin/scp admin\@sys1:'$srcfile' /home/sys1/data/'$dstname'");

and in this code, the single quotes do make sense: $srcfile is interpolated (because the string is in double quotes), and something like

Code

/usr/bin/scp admin\@sys1:'my source file' ...

is passed to the shell, allowing filenames containing spaces, wildcard characters (as part of the name) and similar weird stuff.

jeffersno1
Novice

Nov 7, 2012, 10:48 AM

Post #8 of 10 (6109 views)

Re: [rovf] collect log files remotely and store as 1 file [In reply to]	Can't Post

Hi Rovf,

Thanks for the tip, I've made the amendment and have nearly got it finished, 1 last niggle!!!!!!

The below works brilliantly but if i copy a file with the same date/time from more than 1 host it will get overwritten. What I'm trying to do now is add the $host to the beginning of the file saved Unsure

I've tried various ways but cant get it to work. The dstfile and basename variables from "use File:Basename" are screwing it up Mad

Code

#!/usr/bin/perl 
  
use warnings; 
use POSIX 'strftime'; 
use File::Basename; 
my $date = strftime("%Y%m%d", localtime(time - 86400)); 
 
my $main_dir = '/home/core/sys/logs'; 
my $sis_main_dir = '/home/otpuser/pdf'; 
  
$corelist = "sys1 sys2 sys3 sys4"; 
@cores=split(' ',$corelist); 
  
foreach (@cores) 
{ 
        $host=$_; 
        @array = command_to_array("/usr/bin/ssh core\@$host '/bin/ls -1 /$main_dir/log_$date??????.audit'"); 
        print "host = $host\n"; 
        foreach (@array){ 
                $_ =~ s/\/\//\//g; 
                print " LINE = $_\n"; 
                $srcfile=$_; 
                $dstfile=basename $_; 
                 system ("/usr/bin/scp core\@$host:'$srcfile' /home/otpuser/pdf/RA_data/'$dstname'");  
        } 
}

Any suggestions?

Thanks again for your help, its

Laurent_R
Veteran / Moderator

Nov 7, 2012, 11:01 AM

Post #9 of 10 (6106 views)

Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to]	Can't Post

I did not tryn but probably this would work:

Code

@array = command_to_array("/usr/bin/ssh core\@$host '/bin/ls -1 /$main_dir/log_$date$_.audit'");

BTW, this:

Code

$corelist = "sys1 sys2 sys3 sys4";  
@cores=split(' ',$corelist);

could be written:

Code

my @cores = qw (sys1 sys2 sys3 sys4);

rovf
Veteran

Nov 7, 2012, 12:18 PM

Post #10 of 10 (6097 views)

Re: [jeffersno1] collect log files remotely and store as 1 file [In reply to]	Can't Post

I think the command generated is not what you expect it to be. When doing this type of stuff, I rarely use

Code

system("My Command $goes $here");

but something like (please adapt it to your taste):

Code

sub SYSTEM($) { 
  my $command=shift; 
  chomp $command; 
  print "EXECUTING: [$command]\n" if $i_am_still_debugging; 
  my $retval = system($command); 
  print "Exit Code: ",($retval >> 8), ", Return Code:",$retval & 0xff,"\n" 
    if $i_am_still_debugging; 
  $retval 
} 
 
.... 
 
SYSTEM("My Command $goes $here");

This makes it easier to see what is going on.

BTW, you don't need $corelist in your code. You can write instead:

Code

my @cores = qw(sys1 sys2 sys3 sys4);

Finally, do your self a favour and make your program run under

Code

use strict;

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives