Perl Guru Forums: Perl Programming Help: Advanced: How to pass parameters to Multi Core Engine that calls a subroutine?

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Advanced:
How to pass parameters to Multi Core Engine that calls a subroutine?

bulrush
User

Aug 30, 2016, 8:10 AM

Post #1 of 2 (6359 views)

How to pass parameters to Multi Core Engine that calls a subroutine?	Can't Post

I'm using MCE to try and speed up my grep function in perl since I have to grep on 3 million records but I'm not sure how to use MCE with parameters that get passed to a function. MCE docs here: https://metacpan.org/pod/MCE

I actually need to pass in 2 parameters to the function, and get back an array of values that matched a constructed key. So if my data looks like this:

Code

H1234<tab>140 
H1234-SS<tab>130 
H1234-SB-12-LA<tab>110 
H6155<tab>115 
H6156<tab> 
H1234-SB-12-LA<tab>112 
H1234-SB-12-LA<tab>140

and I want to match "H1234.+140" the resulting array would contain:

Code

H1234<tab>140 
H1234-SB-12-LA<tab>140

MCE actually allows your Perl program to use all CPU cores in parallell processing. This is actually my test loop.

Code

for ($i=0; $i<=$#findthese; $i++) 
    { 
    $t=$findthese[$i]; # Use fake price on end 
    my $mce = MCE->new( 
       max_workers => 4, # On windows use 4 workers. 
       user_func => 
            sub  
            {#my($base,$oldpr)=@_; 
            my($procname,$k); 
            #my(@k); 
 
            $procname="guesspricemce"; 
 
            # Method 2: first part of part num is correct. 
            $k='^'.$_[0]; 
            if (length($_[1])>0) 
                { 
                $k.=".+_".$_[1]; 
                } 
            @k=(); 
            #@k=grep(/$k/, @prkeys); # Find all models that start with key. 
 
            foreach (@prkeys) 
                { 
                if (/$k/) 
                    { 
                    #$thing=$1; 
                    push(@k,$&); # What is faster than push? 
                    } 
                } 
 
            return; # anon function 
            } 
    ); 
    #$t=guesspricewc4($t,140); 
 
    if (($i % 5)==0) # Print status message. 
        { 
        $s="$title: Test $i of $maxsample"; 
        print "$s\n"; 
        } 
    } # for i

I have to pass in 2 parameters to the anonymous function. The first param is a model number, the second is an old price. Then inside the anon function I construct my regex to grep with.

Any ideas how to pass in parameters to this anon function?

Thanks.
-----

(This post was edited by bulrush on Aug 30, 2016, 9:09 AM)

Laurent_R
Veteran / Moderator

Aug 30, 2016, 9:38 AM

Post #2 of 2 (6354 views)

Re: [bulrush] How to pass parameters to Multi Core Engine that calls a subroutine? [In reply to]	Can't Post

In the case at hand, I would say: get the right (fast) algorithm, rather than trying to improve the speed of a poor algorithm with parallel processing.

Using parallel processing might improve your processing time by a factor of 3 or so (not more than 4, for sure, with 4 workers); using the right data structure and the right algorithm will probably speed up your process by a factor of 10 to 100, perhaps even more.

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives