Perl Guru Forums: Perl Programming Help: Beginner: Capturing STDOUT from Parallel::ForkManager with SSH

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Beginner:
Capturing STDOUT from Parallel::ForkManager with SSH

zohman
Novice

May 5, 2016, 6:12 AM

Post #1 of 4 (2897 views)

Capturing STDOUT from Parallel::ForkManager with SSH	Can't Post

Hi,

I have a request where i need to query 500 servers,
right now i'm querying them in a loop, like this:

(This is illustration only)

Code

my $results = {}; 
my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com'); 
 
	foreach my $server (@servers) { 
		chomp( my $stdout = qx{ ssh $server domainname 2>&1 } ); 
		$results->{$server} = $stdout; 
	}

but it's taking more then 15 minutes until $results get populated and the script finish.

i searched for some good an quick method to fork the SSH and found the Parallel::ForkManager package,
after i understood what it does i tried to use it like this:

Code

use Parallel::ForkManager; 
 
my $maxProbes = 100; # Max Processes  
my $pm = new Parallel::ForkManager($maxProbes); 
 
my $results = {}; 
my @servers = ('start.pdx.intel.com','start.sc.intel.com','start.ir.intel.com','start.png.intel.com'); 
 
	foreach my $server (@servers) { 
			$pm->start and next; 
		    chomp( my $stdout = qx{ ssh $server domainname 2>&1 } ); 
			print "$stdout\n"; 
			$results->{$server} = $stdout; 
			$pm->finish; 
	} 
 
	$pm->wait_all_children;

All query's from 500 servers finished in 30 seconds!

The thing is that - print "$stdout\n"; - prints all the output to the screen as expected,
but $results hash reminds empty and everything i do trying to capture $stdout for later use in the script, fails.

i can use Parallel::ForkManager but i'm a little restricted using external packages,
so if someone know something generic or using this package it will help.

Thanks,
Zohman.

zohman
Novice

May 5, 2016, 1:52 PM

Post #2 of 4 (2887 views)

Re: [zohman] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to]	Can't Post

I managed to find some workaround with File::Temp, printing the data to tmp file
then read from it to a hash, the file will be deleted at the end.
i don't know how elegant it is but it's doing my job, what do you think?

Code

use Parallel::ForkManager;  
use File::Temp qw( tempfile ); 
 
my $tmp_fh = new File::Temp( UNLINK => 1 ); 
my $maxProbes = 100; # Max Processes   
my $pm = new Parallel::ForkManager($maxProbes);  
 
my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com'); 
  
	foreach my $server (@servers) {  
			$pm->start and next; 
			my $stdout = qx { ssh $server domainname 2>&1 }; 
			print $tmp_fh "$server,$stdout"; 
			$pm->finish; 
	}  
			$pm->wait_all_children; 
 
		open(RDATA,"$tmp_fh") or die "cannot open $tmp_fh: $!"; 
			my @r_data = <RDATA>; 
	        close(RDATA); 
 
		my %results = map { chomp; @_=split ','; $_[0] => $_[1] } @r_data;

Regards,
Zohman.

FishMonger
Veteran / Moderator

May 5, 2016, 2:13 PM

Post #3 of 4 (2884 views)

Re: [zohman] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to]	Can't Post

I have not used Parallel::ForkManager , so I can't say what the best solution is, but using File::Temp should be ok. The documentation for the Parallel::ForkManager module suggests and gives an example using Storable instead of File::Temp.

zohman
Novice

May 6, 2016, 12:31 AM

Post #4 of 4 (2876 views)

Re: [FishMonger] Capturing STDOUT from Parallel::ForkManager with SSH [In reply to]	Can't Post

Thanks,
yes, i tried to work with that with no success,
but i figure it must be declared before first 'start' ($pm->start)
and need to tale what we want to store in $pm->finish,
so now it's working great and no need for tmpfile.

Code

use Parallel::ForkManager;   
 
my $maxProbes = 100; # Max Processes    
my $pm = new Parallel::ForkManager($maxProbes);   
  
my @servers = ('host1.domain1.com','host2.domain2.com','host3.domain3.com','host4.domain4.com');  
 
my %results; 
 
	$pm->run_on_finish( sub {     
				my ($pid, $exit_code, $ident, $exit_signal, $core_dump, $data) = @_; 
					   $results{ $data->[0] } = $data->[1]; 
			    }); 
 
		foreach my $server (@servers) {   
				$pm->start and next;  
				my $stdout = qx { ssh $server domainname 2>&1 };  
				$pm->finish(0, [ $server, $stdout ]); 
		}   
				$pm->wait_all_children;

Regards,
Zohman.

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives