CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
calling process in threads with IO

 



Xhonzi
New User

Oct 24, 2011, 5:31 AM

Post #1 of 8 (9147 views)
calling process in threads with IO Can't Post

Hello,
I have found that once I call external processes (which generates files) from plenty of threads, sometimes file handles aren't closed and fail on opening or on unlinking of these files. It's very random problem and occurs only e.g. 0-3 times for 200 repeats. But I've tried Win32::Job and for it seems count of these problems increases. I don't know where's the problem:

1) antivirus realtime scan - I have no oportunity to disable it so I can't be sure this is not the reason. But if it's processed only in one thread I've not encountered this problem.
2) I had been searching internet if there's no problem with IO requests when comes from 2 threads in same time if one doesn't block the other, but found no solution.

Have you ever faced such problem once anyone?

Thank you in advance.

#runProcXtimes.pl
use warnings;
use strict;
use threads;
use PROCmod;

foreach my $i(1..200){
map {threads->create(\&subT)} 1..10
sub subT{
PROCmod::run();
}
foreach(threads->list){
while(!$_->is_joinable){;}
$_->join;
}
}

#PROCmod.pm
package PROCmod;

sub run{
my $tID=threads->self->tid;

system("perl -e \"print STDOUT 'stdout';print STDERR 'stderr';\" >o\\stdout_$tID 2>o\\stderr_$tID")==0 or die "$tID:$!";
open(my $SO,'<',"o\\stdout_$tID") or die "open o\\stdout_$tID:$!";
close $SO;
unlink "o\\stdout_$tID" or die "unlink o\\stdout_$tID:$!";
open(my $SE,'<',"o\\stderr_$tID") or die "open o\\stderr_$tID:$!";
close $SE;
unlink "o\\stderr_$tID" or die "unlink o\\stderr_$tID:$!";
}
1;


(This post was edited by Xhonzi on Oct 24, 2011, 5:59 AM)


rovf
Veteran

Oct 28, 2011, 5:07 AM

Post #2 of 8 (8769 views)
Re: [Xhonzi] calling process in threads with IO [In reply to] Can't Post

I had similar effects on Windows too.

Of course it makes sense to check the return code of close. This would at least give some diagnostics, though it would not solve the general problem.

I have the impression that Windows, sometimes, does not immediately release a file after closing, so when another thread wants to open it, it is blocked by "being in use by another process".

You said that your program sometimes fails to open a file, but you didn't say what error message you get.


Xhonzi
New User

Oct 28, 2011, 2:12 PM

Post #3 of 8 (8751 views)
Re: [rovf] calling process in threads with IO [In reply to] Can't Post

Nice to hear I'm not the only one with such problems.
Once it fails it fails always with Permission denied for opening same like for unlinking. File opening failed only when I tried Win32::Job(+file given by path not by file handle). Anyway both have problems on windows with unlinking.

I've tried example I posted on Windows XP, Windows 7 and Knoppix, Porteus Linux. Seems it works well only with Linux. Now the question is what's the reason?
I'm thinking of:
1) bad threads implemetation of ActivePerl on Windows.
2) internal problem of Windows itself.

I wanted to try make the same for .Net but the Process class implementation in there doesn't offer saving to file directly (you have to save file on your own so at the end its saved outside the process) so even I find it works it proves nothing.


rovf
Veteran

Oct 28, 2011, 2:27 PM

Post #4 of 8 (8748 views)
Re: [Xhonzi] calling process in threads with IO [In reply to] Can't Post

Looks similar to the problem I'm experiencing from time to time. In my case, it happens most of the time with files located on a network path, though.

What you could do to narrow down the cause:

- If you get "permission denied", sleep for a couple of seconds and try again.

- Before going to sleep, you could also use an utility such as http://technet.microsoft.com/en-us/sysinternals/bb896655 to find, whether another process keeps the file locked.

Please post your findings here, if any...

Ronald


Xhonzi
New User

Nov 1, 2011, 10:44 PM

Post #5 of 8 (8135 views)
Re: [rovf] calling process in threads with IO [In reply to] Can't Post

Hello,
thank you for the post. I didn't know this tool. Seems interesting. I've to test it on my home PC, I don't have admin rights in here. I don't know why it's needed especially when ProcessExp is based on this tool and even accounts with limited rights can be used. Once I have time to test it at home I'll post result here.


rovf
Veteran

Nov 2, 2011, 5:18 AM

Post #6 of 8 (8110 views)
Re: [Xhonzi] calling process in threads with IO [In reply to] Can't Post


Quote
I don't have admin rights in here. I don't know why it's needed especially when ProcessExp is based on this tool


Indeed, this is weird. However, if you have ProcessExp running, I think you can use this one too (look at the "Find" menu).

Ronald


Xhonzi
New User

Nov 2, 2011, 5:44 AM

Post #7 of 8 (8107 views)
Re: [rovf] calling process in threads with IO [In reply to] Can't Post

Yep, Find menu works fine to solve common file locks but the problem is that the time period for which process locks file is very short. So the only way is to check it out in the code directly after opening/unlinking fails. For this purpose I think I can use the tool you gave me link to I guess.


rovf
Veteran

Nov 4, 2011, 9:24 AM

Post #8 of 8 (7728 views)
Re: [Xhonzi] calling process in threads with IO [In reply to] Can't Post

If the time period is so short, I fear that, until the tool starts running, the issue is resolved already.

I would therefore approach it like this: Place a sleep(1) *before* you unlink or open. If the error still occurs, replace the sleep by execution of the tool.

Ronald

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives