CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
Sharing statics variable between Threads

 



rodeo
New User

Aug 30, 2013, 5:32 AM

Post #1 of 9 (2780 views)
Sharing statics variable between Threads Can't Post

Hello everybody, I have a problem, where I didn't find some solutions until now. I have a program for calculating. For it I must load large data (10GB). This files will be written in a structure. After that I call a function which calculate with this data and an input (load by a excel file). This funtion have 5 interlaced for-loops. The results are well and all runs fine but to get a result I must wait to 10 hours. I have found that I can do the calculates parallel with the Parallel::ForkManager, where the steps of the loop will be swap out at a extra core. The first tests was fine but the used RAM will be multiply by the number of cores. With the modules threads/threads::shared you can share hashes and arrays. For sharing the loaded data, i changed the structure to a hash. Unfortunately the calculating needs more than 24 hours so I canceled the programm. The same happens when I use onlny 2 cores. Now my question: it's there a possibility to share static variables (in the loops the data of the loaded data don't will be changem, I only reading from the structure). Or somebody have an other idea how I can release this problem? Thank you very much


FishMonger
Veteran / Moderator

Aug 30, 2013, 6:40 AM

Post #2 of 9 (2779 views)
Re: [rodeo] Sharing statics variable between Threads [In reply to] Can't Post


Quote
Now my question: it's there a possibility to share static variables (in the loops the data of the loaded data don't will be changem, I only reading from the structure).


If the data structure for the 10GB of data doesn't change during the course of the script, then you simply need to define that var at the file scope level prior to starting your threads. Each thread should then be able to access that data.

I would highly recommend profiling your script to see where it's spending its time and then work on optimizing the slow points.

Devel::NYTProf - Powerful fast feature-rich Perl source code profiler
http://search.cpan.org/~timb/Devel-NYTProf-5.05/lib/Devel/NYTProf.pm


FishMonger
Veteran / Moderator

Aug 30, 2013, 6:43 AM

Post #3 of 9 (2778 views)
Re: [rodeo] Sharing statics variable between Threads [In reply to] Can't Post

You haven't provided enough info for me to be sure on this point, but loading 10GB of data into a var leads me to question why you're not storing and accessing that data from a database.


rodeo
New User

Sep 2, 2013, 1:00 AM

Post #4 of 9 (2750 views)
Re: [FishMonger] Sharing statics variable between Threads [In reply to] Can't Post

The data comes from an external program which saved the data as a file. So first I must read the data and storing this in a database...

Can you give me an example for defining a variable at the file scope level priority?

Thanks a lot for your replies.


Code
my $maxProc=4; 
my $pm = new Parallel::ForkManager($maxProc);

for(my $idy=0;$idy<=$#ySurf;$idy++)
{
my $currY = $ySurf[$idy];
for(my $alfaIdx=0;$alfaIdx<=$#alfas;$alfaIdx++)
{
$pm->start and next;

my $alfa = $alfas[$alfaIdx];

my $norVectX = cos($alfa);
my $norVectY = 0;
my $norVectZ = sin($alfa);

my $norTransf=TransfNorVec($input->{az}, $input->{el}, $norVectX, $norVectY, $norVectZ, $Side);

my $intSurfx = $input->{MiddleX} - $input->{r}*$norVectX;
my $intSurfy = $currY;
my $intSurfz = ($input->{MiddleZ}+$input->{zOffset})-$input->{r}*$norVectZ;

my $center = Divide($input->{w}, $input->{h}, $input->{M}, $input->{N}, $intSurfx, $intSurfy, $intSurfz, $alfa);

my $surf=0;

for(my $idx=0;$idx<$center->{count};$idx++)
{
my $xT = $center->{Px}[$idx];
my $yT = $center->{Py}[$idx];
my $zT = $center->{Pz}[$idx];

my $xTr = coordSystTransf($input->{az}, $input->{el}, $input->{MiddleX}, $input->{MiddleY}, $input->{MiddleZ}, $input->{ZeroX}, $input->{ZeroY}, $input->{ZeroZ}, $xT, $yT, $zT, $Side);

my $sqDiag = ($input->{w}/$input->{M})**2+($input->{h}/$input->{N})**2;

my $P = new Points();
$P->{count}=0;

for(my $dataIdx=0;$dataIdx<$data->{count};$dataIdx++)
{
my $inSphere = Check($sqDiag, $xTr->{xe}, $xTr->{ye}, $xTr->{ze}, $data->{abs_x}[$dataIdx], $data->{abs_y}[$dataIdx], $data->{abs_z}[$dataIdx]);

if($inSphere == 1)
{
$P->{Px}[$P->{count}]=($data->{magX}[$dataIdx])*cos($data->{phX}[$dataIdx]);
$P->{Py}[$P->{count}]=($data->{magY}[$dataIdx])*cos($data->{phY}[$dataIdx]);
$P->{Pz}[$P->{count}]=($data->{magZ}[$dataIdx])*cos($data->{phZ}[$dataIdx]);
$P->{count}++;
}
}

my $Poynt = new Trans();
if(defined($P->{Px}[0]) && defined($P->{Py}[0]) && defined($P->{Pz}[0]))
{
$Poynt->{xe} = summe($P->{Px})/$P->{count};
$Poynt->{ye} = summe($P->{Py})/$P->{count};
$Poynt->{ze} = summe($P->{Pz})/$P->{count};

my $innerProd = $Poynt->{xe}*$norTransf->{xe}+$Poynt->{ye}*$norTransf->{ye}+$Poynt->{ze}*$norTransf->{ze};
$surf += ((($input->{w})/$input->{M})*(($input->{h})/$input->{N}))*$innerProd;
}
}


print "\ny= ".$currY."; alfa= ".$alfa .";\n";

if($surf == 0)
{
print 1e-100 ."dB\n";
}else{
print sprintf("%.4f", 20*log(abs($surf))) ." dB\n";
}

$pm->finish;
}
$pm->wait_all_children;
}



(This post was edited by rodeo on Sep 2, 2013, 1:01 AM)


FishMonger
Veteran / Moderator

Sep 2, 2013, 8:20 AM

Post #5 of 9 (2737 views)
Re: [rodeo] Sharing statics variable between Threads [In reply to] Can't Post

Variable Scoping in Perl: the basics http://www.perlmonks.org/?node_id=66677


Laurent_R
Enthusiast / Moderator

Sep 2, 2013, 10:46 AM

Post #6 of 9 (2732 views)
Re: [rodeo] Sharing statics variable between Threads [In reply to] Can't Post


In Reply To
Can you give me an example for defining a variable at the file scope level priority?


Just declare your variable at the top of your script, outside of any code block, function or loop. For example:


Code
#!/usr/bin/perl 
use strict;
use warnings;

my $var;

# the code for populating $var and calling the threads



rodeo
New User

Sep 5, 2013, 1:12 AM

Post #7 of 9 (2699 views)
Re: [Laurent_R] Sharing statics variable between Threads [In reply to] Can't Post

Thank you for the example.
But I must still write "use Devel::NYTProf" before I declare my variable?!
I will test it. Thanks a lot


rodeo
New User

Sep 5, 2013, 3:46 AM

Post #8 of 9 (2695 views)
Re: [FishMonger] Sharing statics variable between Threads [In reply to] Can't Post

I have a question to the Devel::NYTProf-Modul.
How the script know which variable is mean.
Can I use the variable "normally" or I need a function to access?

Here a simple example:


Code
use Parallel::ForkManager; 
use Devel::NYTProf;

my $test;

$test=(1..10);
my $pm=new Parallel::ForkManager(4);

for(my $i=0;$i<10;$i++)
{
$pm->start and next;
print $test[$i];
$pm->finish;
}


Variable $test has data from more than 10GB.


(This post was edited by rodeo on Sep 5, 2013, 3:53 AM)


FishMonger
Veteran / Moderator

Sep 5, 2013, 6:26 AM

Post #9 of 9 (2687 views)
Re: [rodeo] Sharing statics variable between Threads [In reply to] Can't Post

That's not how the module is used.

The module is not loaded via a use statement in your script. It's loaded/used by perl's debugger from the command line.

You should read over the module's documentation.
http://search.cpan.org/~timb/Devel-NYTProf-5.05/lib/Devel/NYTProf.pm

Code
SYNOPSIS 

# profile code and write database to ./nytprof.out
perl -d:NYTProf some_perl.pl

# convert database into a set of html files, e.g., ./nytprof/index.html
# and open a web browser on the nytprof/index.html file
nytprofhtml --open

# or into comma separated files, e.g., ./nytprof/*.csv
nytprofcsv


Are you still using your above posted code, or did you make any adjustments based on the suggestions made to you in your cross posted question on perlmonks?

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives