
kirylm
New User
Jul 9, 2010, 2:08 PM
Post #1 of 2
(3624 views)
|
Pipe I/O Very Large files
|
Can't Post
|
|
Hi all, I have the following situation: I have two sets of pipes open: - in1, out1, err1 for processA - in2, out2, err2 for processB Pipes were open with IPC::Open3(). I/O is done using sysread() and syswrite() (no mixing with buffered I/O). What i want to accomplish: send a request to in1; while there is some data on out1 { receive partial output from out1 (watch 4 errors on err1); write that output to in2; } receive confirmation from out2 (watch 4 errors on err2); The data that is coming from out1 could be very large (up to 250 GB) and I only have 32 GB of RAM on that machine. I need a solution that will not require any disk I/O, only RAM. I tried to come up with some kind of a batch solution where I read 5 GB, then I write them, then I read another 5 GB - write them. However, this is not working... The program blocks on the first read (i assume because not all data was read). However, in case only 1 read was required (data < 5 GB), everything works. Questions: - what will happen if I try to fetch more then 32 GB of data into a scalar? Is there some build-in process in Perl that will automatically go disk-based if there is not enough RAM? - how can I only read n bytes of data and continue execution (i.e. write whatever was read so far) and come back to read the rest later? - For my solution, am I using the right functions (sysread/syswrite)? - I suspect I need to implement asynch I/O. I never did that before. If that is the case, any suggestions are really appreciated. Thanks all! PS - I can post the code if needed. |