CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
Capturing STDOUT from Parallel::ForkManager with SSH

 



zohman
Novice

May 5, 2016, 6:12 AM

Post #1 of 4 (1920 views)
Capturing STDOUT from Parallel::ForkManager with SSH Can't Post

Hi,

I have a request where i need to query 500 servers,
right now i'm querying them in a loop, like this:

(This is illustration only)


Code
my $results = {}; 
my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com');

foreach my $server (@servers) {
chomp( my $stdout = qx{ ssh $server domainname 2>&1 } );
$results->{$server} = $stdout;
}


but it's taking more then 15 minutes until $results get populated and the script finish.

i searched for some good an quick method to fork the SSH and found the Parallel::ForkManager package,
after i understood what it does i tried to use it like this:


Code
use Parallel::ForkManager; 

my $maxProbes = 100; # Max Processes
my $pm = new Parallel::ForkManager($maxProbes);

my $results = {};
my @servers = ('start.pdx.intel.com','start.sc.intel.com','start.ir.intel.com','start.png.intel.com');

foreach my $server (@servers) {
$pm->start and next;
chomp( my $stdout = qx{ ssh $server domainname 2>&1 } );
print "$stdout\n";
$results->{$server} = $stdout;
$pm->finish;
}

$pm->wait_all_children;


All query's from 500 servers finished in 30 seconds!

The thing is that - print "$stdout\n"; - prints all the output to the screen as expected,
but $results hash reminds empty and everything i do trying to capture $stdout for later use in the script, fails.

i can use Parallel::ForkManager but i'm a little restricted using external packages,
so if someone know something generic or using this package it will help.

Thanks,
Zohman.


zohman
Novice

May 5, 2016, 1:52 PM

Post #2 of 4 (1910 views)
Re: [zohman] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to] Can't Post

I managed to find some workaround with File::Temp, printing the data to tmp file
then read from it to a hash, the file will be deleted at the end.
i don't know how elegant it is but it's doing my job, what do you think?


Code
use Parallel::ForkManager;  
use File::Temp qw( tempfile );

my $tmp_fh = new File::Temp( UNLINK => 1 );
my $maxProbes = 100; # Max Processes
my $pm = new Parallel::ForkManager($maxProbes);

my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com');

foreach my $server (@servers) {
$pm->start and next;
my $stdout = qx { ssh $server domainname 2>&1 };
print $tmp_fh "$server,$stdout";
$pm->finish;
}
$pm->wait_all_children;

open(RDATA,"$tmp_fh") or die "cannot open $tmp_fh: $!";
my @r_data = <RDATA>;
close(RDATA);

my %results = map { chomp; @_=split ','; $_[0] => $_[1] } @r_data;


Regards,
Zohman.


FishMonger
Veteran / Moderator

May 5, 2016, 2:13 PM

Post #3 of 4 (1907 views)
Re: [zohman] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to] Can't Post

I have not used Parallel::ForkManager , so I can't say what the best solution is, but using File::Temp should be ok. The documentation for the Parallel::ForkManager module suggests and gives an example using Storable instead of File::Temp.


zohman
Novice

May 6, 2016, 12:31 AM

Post #4 of 4 (1899 views)
Re: [FishMonger] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to] Can't Post

Thanks,
yes, i tried to work with that with no success,
but i figure it must be declared before first 'start' ($pm->start)
and need to tale what we want to store in $pm->finish,
so now it's working great and no need for tmpfile.


Code
use Parallel::ForkManager;   

my $maxProbes = 100; # Max Processes
my $pm = new Parallel::ForkManager($maxProbes);

my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com');

my %results;

$pm->run_on_finish( sub {
my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data) = @_;
$results{ $data->[0] } = $data->[1];
});

foreach my $server (@servers) {
$pm->start and next;
my $stdout = qx { ssh $server domainname 2>&1 };
$pm->finish(0, [ $server, $stdout ]);
}
$pm->wait_all_children;


Regards,
Zohman.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives