CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
How to pass parameters to Multi Core Engine that calls a subroutine?

 



bulrush
User

Aug 30, 2016, 8:10 AM

Post #1 of 2 (2509 views)
How to pass parameters to Multi Core Engine that calls a subroutine? Can't Post

I'm using MCE to try and speed up my grep function in perl since I have to grep on 3 million records but I'm not sure how to use MCE with parameters that get passed to a function. MCE docs here: https://metacpan.org/pod/MCE

I actually need to pass in 2 parameters to the function, and get back an array of values that matched a constructed key. So if my data looks like this:

Code
H1234<tab>140 
H1234-SS<tab>130
H1234-SB-12-LA<tab>110
H6155<tab>115
H6156<tab>
H1234-SB-12-LA<tab>112
H1234-SB-12-LA<tab>140


and I want to match "H1234.+140" the resulting array would contain:

Code
H1234<tab>140 
H1234-SB-12-LA<tab>140


MCE actually allows your Perl program to use all CPU cores in parallell processing. This is actually my test loop.

Code
for ($i=0; $i<=$#findthese; $i++) 
{
$t=$findthese[$i]; # Use fake price on end
my $mce = MCE->new(
max_workers => 4, # On windows use 4 workers.
user_func =>
sub
{#my($base,$oldpr)=@_;
my($procname,$k);
#my(@k);

$procname="guesspricemce";

# Method 2: first part of part num is correct.
$k='^'.$_[0];
if (length($_[1])>0)
{
$k.=".+_".$_[1];
}
@k=();
#@k=grep(/$k/, @prkeys); # Find all models that start with key.

foreach (@prkeys)
{
if (/$k/)
{
#$thing=$1;
push(@k,$&); # What is faster than push?
}
}

return; # anon function
}
);
#$t=guesspricewc4($t,140);

if (($i % 5)==0) # Print status message.
{
$s="$title: Test $i of $maxsample";
print "$s\n";
}
} # for i

I have to pass in 2 parameters to the anonymous function. The first param is a model number, the second is an old price. Then inside the anon function I construct my regex to grep with.

Any ideas how to pass in parameters to this anon function?

Thanks.
-----


(This post was edited by bulrush on Aug 30, 2016, 9:09 AM)


Laurent_R
Veteran / Moderator

Aug 30, 2016, 9:38 AM

Post #2 of 2 (2504 views)
Re: [bulrush] How to pass parameters to Multi Core Engine that calls a subroutine? [In reply to] Can't Post

In the case at hand, I would say: get the right (fast) algorithm, rather than trying to improve the speed of a poor algorithm with parallel processing.

Using parallel processing might improve your processing time by a factor of 3 or so (not more than 4, for sure, with 4 workers); using the right data structure and the right algorithm will probably speed up your process by a factor of 10 to 100, perhaps even more.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives