kirylm
New User
Jul 9, 2010, 2:08 PM
Post #1 of 2
(3287 views)
|
|
Pipe I/O Very Large files
|
Can't Post
|
|
Hi all,
I have the following situation:
I have two sets of pipes open:
- in1, out1, err1 for processA
- in2, out2, err2 for processB
Pipes were open with IPC::Open3().
I/O is done using sysread() and syswrite() (no mixing with buffered I/O).
What i want to accomplish:
send a request to in1;
while there is some data on out1 {
receive partial output from out1 (watch 4 errors on err1);
write that output to in2;
}
receive confirmation from out2 (watch 4 errors on err2);
The data that is coming from out1 could be very large (up to 250 GB) and I only have 32 GB of RAM on that machine. I need a solution that will not require any disk I/O, only RAM.
I tried to come up with some kind of a batch solution where I read 5 GB, then I write them, then I read another 5 GB - write them. However, this is not working... The program blocks on the first read (i assume because not all data was read). However, in case only 1 read was required (data < 5 GB), everything works.
Questions:
- what will happen if I try to fetch more then 32 GB of data into a scalar? Is there some build-in process in Perl that will automatically go disk-based if there is not enough RAM?
- how can I only read n bytes of data and continue execution (i.e. write whatever was read so far) and come back to read the rest later?
- For my solution, am I using the right functions (sysread/syswrite)?
- I suspect I need to implement asynch I/O. I never did that before. If that is the case, any suggestions are really appreciated.
Thanks all!
PS - I can post the code if needed.
|
MAIN
SEARCH
WHO'S 
LOG 