CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
Search Posts SEARCH
Who's Online WHO'S
Log in LOG

Home: Perl Programming Help: Advanced:
Pipe I/O Very Large files


New User

Jul 9, 2010, 2:08 PM

Post #1 of 2 (3942 views)
Pipe I/O Very Large files Can't Post

Hi all,

I have the following situation:

I have two sets of pipes open:
- in1, out1, err1 for processA
- in2, out2, err2 for processB

Pipes were open with IPC::Open3().
I/O is done using sysread() and syswrite() (no mixing with buffered I/O).

What i want to accomplish:

send a request to in1;
while there is some data on out1 {
receive partial output from out1 (watch 4 errors on err1);
write that output to in2;
receive confirmation from out2 (watch 4 errors on err2);

The data that is coming from out1 could be very large (up to 250 GB) and I only have 32 GB of RAM on that machine. I need a solution that will not require any disk I/O, only RAM.

I tried to come up with some kind of a batch solution where I read 5 GB, then I write them, then I read another 5 GB - write them. However, this is not working... The program blocks on the first read (i assume because not all data was read). However, in case only 1 read was required (data < 5 GB), everything works.

- what will happen if I try to fetch more then 32 GB of data into a scalar? Is there some build-in process in Perl that will automatically go disk-based if there is not enough RAM?
- how can I only read n bytes of data and continue execution (i.e. write whatever was read so far) and come back to read the rest later?
- For my solution, am I using the right functions (sysread/syswrite)?
- I suspect I need to implement asynch I/O. I never did that before. If that is the case, any suggestions are really appreciated.

Thanks all!

PS - I can post the code if needed.




Jul 14, 2010, 6:29 PM

Post #2 of 2 (3823 views)
Re: [kirylm] Pipe I/O Very Large files [In reply to] Can't Post

can you please post sample that i can play with, i have a mechanism where i am using pipes(fifos) as well, but i am not using IPC module, i am just using POSIX select and read, and i am thinking that select is what you want, because if something writes into pipe, select will block it, read information into variable, which should let you process it, and then will return fifo back to another process, at that point another process will be able to write some more information into pipe, and so on.... so you can accomplish what you desire through blocking pretty much...

sounds like a lot of fun :)


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives