CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
Search Posts SEARCH
Who's Online WHO'S
Log in LOG

Home: Perl Programming Help: Beginner:
How to list into separate files and Fork() function (Updated)



Jul 17, 2014, 12:33 PM

Post #1 of 3 (935 views)
How to list into separate files and Fork() function (Updated) Can't Post

Hello, All:

Okay, honestly I am stumped, for I don't no an algorithm to take a list of nodes (elements, whatever) say 100 or so, and have a Perl script run where I take the 100 nodes into a Filehand <FH> and read into an array as normal. Yet, what I need is away to take the input of the 100 and split them up into files of 25 files or temporary arrays. Next have the x number of files or arrays to start a separate process so they run in parallel.

Attached you will find the script I tried to adapt and use but after the nodechunks I get lost in the fork, and the children and child function. help.

(This post was edited by preston3271 on Jul 18, 2014, 6:51 AM)
Attachments: nodelist.rtf (32.5 KB)
  Sample Script.rtf (35.4 KB)

Veteran / Moderator

Jul 18, 2014, 10:09 AM

Post #2 of 3 (898 views)
Re: [preston3271] How to list into separate files and Fork() function (Updated) [In reply to] Can't Post

Am I correct to understand that what you call nodes are just files that you want to read through?

It also seems that the splitting into 25 files is aimed only at load balancing, the data is not dispatched on ther basis of its content.

If you have a single process reading all the files in order to split them into other files before handling them to child process, you might lose the benefit of parallel processing.

A much simpler technique your be to make an array of your file names (effectively a queue of file names) and to assign one file to each of your 25 children, which will read the data and process it. Once one process has completed, just assign a new file to it, and so forth until you run out of files.

In most cases, this should be more efficient, because you are reading your data only once and immediately in parallel processes. If the files have significantly different sizes, then start by sorting your file array by size, and assign the large files first to the children, this will ensure close-to-optimal load balancing in most cases (the only pathological case where this might not be optimal is if there are huge sizes difference between files, with a few extremely large and others extremely small).

As for dispatchning the work between the children, it depends how you implement the launching of parallel processes or threads. Either the parent process can also be the master, monitor the children and hand out work to any child running out of work, or the children can take a filename and remove it from the queue when irt completes some work and becomes idle.

There are many many implementation details missing in the above, but you are not supplying enough information on what you are really trying to do (which is more important than how you contemplate to do it, as you can see I am suggesting a slightly modified way of doing it).

Veteran / Moderator

Jul 18, 2014, 11:09 AM

Post #3 of 3 (895 views)
Re: [preston3271] How to list into separate files and Fork() function (Updated) [In reply to] Can't Post

You probably should consider using Parallel::ForkManager - A simple parallel processing fork manager


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives