CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
Release memory from threads

 



KingNothing
Novice

Feb 22, 2012, 8:41 AM

Post #1 of 11 (11972 views)
Release memory from threads Can't Post

Hi there,
i have write a script that manages input files i give. It creates threads to make the whole process faster. The operation takes about 5 minutes with threads while doing it without theads it takes almost 40 minutes. The time is of course analog to the filesize and the number of the threads. For a file around 4.7Megabyte i create 408 threads which run for about 5 minutes and in that time the cpu is 100% working as i see in Task Manager on win7. When the threads finish their jobs i see that the cpu goes down at 15% (at task manager). But, the perl.exe process doesn't release the memory which has reached around 400megabyte and i have to close the program to release the memory. I tried, after the finish of threads, many ways to release the memory such as

Code
for($a=0;$a<=$filecounter;$a++){ 
$thr[$a]->exit();
}

and other like

Code
for($a=0;$a<=$filecounter;$a++){ 
$thr[$a]=undef;
}

or

Code
for($a=0;$a<=$filecounter;$a++){ 
die $thr[$a];
}

but i can't release the memory back to the os. I only manage to shut down the whole program with the ways i wrote above or take some warning messages.
I would appreciate the help and the opinion whoever can help me.
Thank you :)


budman
User

Feb 25, 2012, 7:02 AM

Post #2 of 11 (11821 views)
Re: [KingNothing] Release memory from threads [In reply to] Can't Post

Can you provide sample code of how you are setting up the threads.

Have you checked over the doc on threads
http://perldoc.perl.org/threads.html
"Thread Signalling" may help.

Using signals to should help the shutdown the threads correctly,
especially on exceptions.


KingNothing
Novice

Feb 25, 2012, 1:52 PM

Post #3 of 11 (11812 views)
Re: [budman] Release memory from threads [In reply to] Can't Post

Hi, this is how i create the threads

Code
for($a=0;$a<=$counter;$a++){ 
if($choice==1){
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::submit($file)});
}
else
{
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::reset($file)});
}
$result[$a]=$thr[$a]->is_running();
}

I used semaphores th shut down the threads but then the memory in use just get doubled and reached 800Megabyte. I closed the whole process because soon it would get out of memory. I have read the doc on perldoc page for the threads but really can't find the way out.


(This post was edited by KingNothing on Feb 25, 2012, 1:55 PM)


budman
User

Feb 25, 2012, 7:27 PM

Post #4 of 11 (11801 views)
Re: [KingNothing] Release memory from threads [In reply to] Can't Post

The loop really didn't help much.
I created an example basing it on your loop contents.

Do you have loops that check if the threads are ok to join (is_joinable)
and ok to exit (is_running)?

About join, this means block or wait until the thread is complete
and then release the thread and do any OS cleanup thats needed.
Its important to join. Detach means to ignore the thread, which will just exit and release when its done.


I ran this on linux, adding the sleep after everything was done,
allowed the system time to free memory. In this run, it freed up ~80KB.

Amt Free Mem at Start : 2304404
Amt Free Mem w/Threads : 2299016 (-5388)
Amt Free Mem at End : 2387748 (83344)




Code
#!/usr/bin/perl 
use strict;
use warnings;

# example of the ext1.pm
{
package ext1;
use Exporter;
our @EXPORTS_OK=qw(submit reset);

sub submit {
my ($id, $file) = @_;
print "Thread $id: submit $file\n";
keys ( my %h ) = 1000;
sleep(2);
}

sub reset {
my ($id, $file) = @_;
print "Thread $id: reset $file\n";
keys ( my %h ) = 1000;
sleep(15);
}
1;
}

# would normally choose use ext1 if was separate file ext1.pm
# but since I have the package in the same file, I don't want to run require.
import ext1;
#use ext1;

use threads;
use Data::Dumper;
$|++;

my @files = (['file1',1],['file2',1],['file3',0],['file4',1],['file5',1]);
my $options = {'exit'=>'thread_only'};
my $bar = "=" x 40;

# open the threads

my $start_mem = free_mem();
print "$bar\nSubmit processes\n$bar\n";
my @results;
foreach my $i (0 .. $#files) {
my ($file,$choice) = @{$files[$i]};
if ( $choice == 1 ) {
$results[$i] = threads->create( $options, sub { &ext1::submit($i,$file) } );
} else {
$results[$i] = threads->create( $options, sub { &ext1::reset($i,$file) } );
}
}

# once threads are submitted, then need to either be joined or detached.

# wait for each thread to be ready to be joined.

sleep(2);
my $thread_mem = free_mem();
print "$bar\nJoining threads\n$bar\n";
foreach my $i (0..$#results) {
print "thread $i: ";
while ( !$results[$i]->is_joinable() ) { print "."; sleep(1) }
print "ready\n";
#$results[$i]->join; # can join here as well
}
# join all the threads
foreach my $r (@results) {
$r->join;
}

# check that all threads freed up

print "$bar\nThread results\n$bar\n";
foreach my $i (0 .. $#results) {
print "thread $i: ";
while ( $results[$i]->is_running() ) { print "."; sleep(1) }
print "finished\n";
#print Dumper($results[$i]);
}

sleep(10);

print "$bar\nMemory\n$bar\n";
my $finish_mem = free_mem();

printf "Amt Free Mem at Start : %d\n", $start_mem;
printf "Amt Free Mem w/Threads : %d (%d)\n", $thread_mem, $thread_mem - $start_mem;
printf "Amt Free Mem at End : %d (%d)\n", $finish_mem, $finish_mem - $start_mem;
print "done\n";
exit 0;

sub free_mem {
return (split /\s+/, qx(free | grep ^Mem:) )[3];
}




Output:

Code
======================================== 
Submit processes
========================================
Thread 0: submit file1
Thread 1: submit file2
Thread 2: reset file3
Thread 3: submit file4
Thread 4: submit file5
========================================
Joining threads
========================================
thread 0: ready
thread 1: ready
thread 2: .............ready
thread 3: ready
thread 4: ready
========================================
Thread results
========================================
thread 0: finished
thread 1: finished
thread 2: finished
thread 3: finished
thread 4: finished
========================================
Memory
========================================
Amt Free Mem at Start : 2304404
Amt Free Mem w/Threads : 2299016 (-5388)
Amt Free Mem at End : 2387748 (83344)
done



(This post was edited by budman on Feb 25, 2012, 7:43 PM)


KingNothing
Novice

Feb 27, 2012, 3:53 AM

Post #5 of 11 (11684 views)
Re: [budman] Release memory from threads [In reply to] Can't Post

Hi, sorry for my late reply. Also, sorry for not posting the whole block of code. Here is the whole part where i use threads:

Code
for($a=0;$a<=$counter;$a++){  
if($choice==1){
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::submit($file)});
}
else
{
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::reset($file)});
}
$result[$a]=$thr[$a]->is_running();
}
$sum_a=sum(@result);
while($sum_a!=0)
{
for($a=0;$a<=$counter;$a++){
$result[$a]=$thr[$a]->is_running();}
$sum_a=sum(@result);
}
print("Operation finished\n");

I will check the code you posted and i will come back soon to reply. As you see, i have a while loop which won't stop until the $sum_a isn't zero (when it becomes zero then all threads have finished their job).
Anyway, thank you very much for your interest. I will reply soon to your code.

EDIT
As you see, i don't know what to do after the block of code i posted above where i am pretty sure that the threads have finished their jobs but the memory remains. I really don't know how to release the memory and i have tryed the options in my first post. All the options i have tryed from my first post have been tested after the block of code i posted here in my third post. I would really appreciate it if you could help me somehow with my problem.


(This post was edited by KingNothing on Feb 27, 2012, 4:13 AM)


budman
User

Feb 27, 2012, 9:07 AM

Post #6 of 11 (11671 views)
Re: [KingNothing] Release memory from threads [In reply to] Can't Post

Hi,

To use threads you need to do these steps:
1) create the threads (OK)
2) wait til they can be joined (you need to do this)
3) join the threads (you need to do this - this initiates the cleanup when done)
4) check that all threads exited (OK)

You need to do steps 2 and 3, otherwise, the threads remain open.

There are two ways to close a thread, join or detach. (You aren't doing either)

If you care about the output, and want to retrieve it, then you join each thread when they able to join. You must check if they can join, otherwise the parent can send the signal, and the child never sees it.

If you are not concerned about the output, then you can use detach, which tells the thread to cleanup when you are done, the parent doesn't care.

Either way, the memory cleanup is handled by join and detach.
When it's done depends on which one is used.


KingNothing
Novice

Feb 27, 2012, 10:22 AM

Post #7 of 11 (11667 views)
Re: [budman] Release memory from threads [In reply to] Can't Post

I tried this (the new code is inside the comments):

Code
 
for($a=0;$a<=$counter;$a++){
if($choice==1){
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::submit($file)});
}
else
{
$thr[$a] = threads->create({'exit'=> 'thread_only'},sub {&ext1::reset($file)});
}
$result[$a]=$thr[$a]->is_running();
}

##############################
for($a=0;$a<=$counter;$a++){
$result_join=$thr[$a]->is_joinable();
while($result_join!=1){
if ($thr[$a]->is_joinable()) {
$thr[$a]->threads->detach();
$result_join=1;
}}}
##############################
$sum_a=sum(@result);
while($sum_a!=0)
{
for($a=0;$a<=$counter;$a++){
$result[$a]=$thr[$a]->is_running();}
$sum_a=sum(@result);
}
print("Operation finished\n");


But i get this error

Thread 1 terminated abnormally: Can't locate object method "threads" via package
"threads" at ext1.pm line 1024.


The program doesn't release the memory and when i shut it down i get the below info (i put it just for information, as i had output messages like that and before)
Perl exited with active threads:
0 running and unjoined
2 finished and unjoined
0 running and detached

Sorry but I really don't know where to put the things you told me. I don't care about the output of the threads since they finish their jobs and as you told me I used the detach choice but I guess I have more errors in my code. :(


budman
User

Mar 4, 2012, 9:05 AM

Post #8 of 11 (11367 views)
Re: [KingNothing] Release memory from threads [In reply to] Can't Post

The error means threads weren't joined or detached.

I think I see what you are trying to accomplish using detach.


Code
threads->yield();  
my @list = threads->list();
printf ("Thread %d running %d\n", $_->tid, $_->is_running) for @list;
print "$bar\nContinuing\n$bar\n";
while ( @list = threads->list() ) {
foreach my $t (@list) {
if ( ! $t->is_running() ) {
printf "Thread %d is no longer running\n", $t->tid;
$t->detach;
if ( $t->is_detached ) { print "\tdetached\n" }
else { warn "Error: could not detach\n" }
my (@remaining) = threads->list(threads::running);
print "\t",scalar(@remaining)," threads remain\n";
}
}
# do whatever else
sleep(1);
}

Output:

========================================
Submit processes
========================================
Thread 1 running 1
Thread 2 running 1
Thread 3 running 1
Thread 4 running 1
Thread 5 running 1
========================================
Continuing
========================================
Thread 5 is no longer running
detached
4 threads remain
Thread 3 is no longer running
detached
3 threads remain
Thread 4 is no longer running
detached
2 threads remain
Thread 1 is no longer running
detached
1 threads remain
Thread 2 is no longer running
detached
0 threads remain
========================================


Let me clarify a bit more about join and detach, I never really used detach, my understanding was it let the processes complete when done, that was incorrect, any processes that are detached will be killed silently when parent quits.

Joinable means you want to capture data from the threads. You use is_joinable to allow you to do other things, while the thread is running. If you just issued the join without checking, the loop would wait (block) until the thread completed.


Code
threads->yield(); # tells OS to let threads have cpu time 
while ( my @list = threads->list ) {
foreach my $t (threads->list(threads::joinable) ) {
my @result = $t->join;
# do whatever to the results from the thread
my (@remaining) = threads->list(threads::running);
print "\t",scalar(@remaining)," threads remain\n";
}
# do other stuff
sleep(1);
}



Detach means you don't care about the output of the thread, you are done with it. Once detached, you cannot rejoin it. You monitor it by checking is_running and if detach was successful with is_detached.

Check out this tutorial on threads
http://perldoc.perl.org/threads.html

You may want to look into Fork::Manager on CPAN, that does a lot of the work in the background.


(This post was edited by budman on Mar 4, 2012, 9:45 AM)


KingNothing
Novice

Mar 7, 2012, 6:41 AM

Post #9 of 11 (11131 views)
Re: [budman] Release memory from threads [In reply to] Can't Post

I managed to detach all the finished threads but the memory remains Mad
Anyway, i spent free time from a week to achieve the memory release but found nothing. I also tried fork() but I have issues over there too MadMad
I can't found anything else on net helping me.
The only I have do - but can't let it like that for sure - is to restart the program when the threads finish. But, this is only for now, I can't leave it like that.
If you have any more ideas for a possible solution you just can post it in the thread.
Anyway, no matter the result, i really thank you for your interest and your willing to help me Smile


budman
User

Mar 13, 2012, 11:32 PM

Post #10 of 11 (10792 views)
Re: [KingNothing] Release memory from threads [In reply to] Can't Post

I did find some notes about perl threads under Windows, they don't always work the best. Perl uses its own threads, and from what I read, each thread launches is own Perl environment, so the overhead can be quite high if you are loading up 400+ threads.

To see if there is another possible solution, I need to know more about what is going on with ext1 reset and resume. Also, how the files are processed, or fed into the script. Is this a resident app like a daemon or tsr?

Because of the way perl is processing, you may need to treat these files as sessions, so the perl.exe exec can exit and do cleanup. Its hard to tell, I would definitely need more details, even a brief outline of the process so I can see how files are spawned into threads.


budman
User

Mar 14, 2012, 12:26 AM

Post #11 of 11 (10790 views)
Re: [budman] Release memory from threads [In reply to] Can't Post

You can check if there was an error with a thread when its detached.


Code
 
$t->detach;
if ( my $err = $t->error() ) { print "Error: $err\n" }


This will force the thread to be evaled and report any errors if they occur.

What version of Perl are you using?
It's recommended you use 5.12 or newer due to the many bug fixes applied for threads (There was one for memory leak).

Here is a test script to test for memory leaks in threads:
https://bugzilla.redhat.com/attachment.cgi?id=442606
May need to adapt it to Windows. I don't know how to grab stat for memory, other than the DOS mem command. Maybe there is a module for Win32.



I tested it under 5.14, no leaks detected.


(This post was edited by budman on Mar 14, 2012, 12:50 AM)

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives