CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
How to list into separate files and Fork() function (Updated)

 



preston3271
Novice

Jul 17, 2014, 12:33 PM

Post #1 of 3 (399 views)
How to list into separate files and Fork() function (Updated) Can't Post

Hello, All:

Okay, honestly I am stumped, for I don't no an algorithm to take a list of nodes (elements, whatever) say 100 or so, and have a Perl script run where I take the 100 nodes into a Filehand <FH> and read into an array as normal. Yet, what I need is away to take the input of the 100 and split them up into files of 25 files or temporary arrays. Next have the x number of files or arrays to start a separate process so they run in parallel.

Attached you will find the script I tried to adapt and use but after the nodechunks I get lost in the fork, and the children and child function. help.


(This post was edited by preston3271 on Jul 18, 2014, 6:51 AM)
Attachments: nodelist.rtf (32.5 KB)
  Sample Script.rtf (35.4 KB)


Laurent_R
Veteran / Moderator

Jul 18, 2014, 10:09 AM

Post #2 of 3 (362 views)
Re: [preston3271] How to list into separate files and Fork() function (Updated) [In reply to] Can't Post

Am I correct to understand that what you call nodes are just files that you want to read through?

It also seems that the splitting into 25 files is aimed only at load balancing, the data is not dispatched on ther basis of its content.

If you have a single process reading all the files in order to split them into other files before handling them to child process, you might lose the benefit of parallel processing.

A much simpler technique your be to make an array of your file names (effectively a queue of file names) and to assign one file to each of your 25 children, which will read the data and process it. Once one process has completed, just assign a new file to it, and so forth until you run out of files.

In most cases, this should be more efficient, because you are reading your data only once and immediately in parallel processes. If the files have significantly different sizes, then start by sorting your file array by size, and assign the large files first to the children, this will ensure close-to-optimal load balancing in most cases (the only pathological case where this might not be optimal is if there are huge sizes difference between files, with a few extremely large and others extremely small).

As for dispatchning the work between the children, it depends how you implement the launching of parallel processes or threads. Either the parent process can also be the master, monitor the children and hand out work to any child running out of work, or the children can take a filename and remove it from the queue when irt completes some work and becomes idle.

There are many many implementation details missing in the above, but you are not supplying enough information on what you are really trying to do (which is more important than how you contemplate to do it, as you can see I am suggesting a slightly modified way of doing it).


FishMonger
Veteran / Moderator

Jul 18, 2014, 11:09 AM

Post #3 of 3 (359 views)
Re: [preston3271] How to list into separate files and Fork() function (Updated) [In reply to] Can't Post

You probably should consider using Parallel::ForkManager - A simple parallel processing fork manager
https://metacpan.org/pod/Parallel::ForkManager

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives