CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Intermediate:
simultaneous exection of script on webpage

 



ttamlrep
Novice

Dec 17, 2014, 12:19 PM

Post #1 of 13 (5181 views)
simultaneous exection of script on webpage Can't Post

I have a Perl script that will be run behind a webpage. The user will input a URL and an e-mail address in the webpage and the Perl program get this info and then go out to a particular website (the URL) get information from that site, process the information, and send the results to the user's e-mail address. This process can take 45 min to an hour.

What if the user wants to input a second URL while the first is still processing, in other words, how can I make the script so that it can execute simultaneously on different requests ?

I just need to be pointed in the right direction. I am sort of clueless as to even start how I would deal with this. My first thought, would be to have the website reload after each query, but the person making the web page asked me how my Perl script would do this.


FishMonger
Veteran / Moderator

Dec 17, 2014, 12:24 PM

Post #2 of 13 (5180 views)
Re: [ttamlrep] simultaneous exection of script on webpage [In reply to] Can't Post

The script that gets and processes the info from the submitted url should be executed as a background process so that the user does not need to wait for it to complete before submitting a new request.


ttamlrep
Novice

Dec 17, 2014, 12:38 PM

Post #3 of 13 (5178 views)
Re: [FishMonger] simultaneous exection of script on webpage [In reply to] Can't Post

Thank you very much FishMonger for your reply.

I was able to google Perl and background process and from a 2008 post I found that I should fork and then exit.

Does this sound about right to you ?


FishMonger
Veteran / Moderator

Dec 17, 2014, 12:47 PM

Post #4 of 13 (5176 views)
Re: [ttamlrep] simultaneous exection of script on webpage [In reply to] Can't Post

Yes, that is what you need to do.


ttamlrep
Novice

Dec 17, 2014, 12:51 PM

Post #5 of 13 (5174 views)
Re: [FishMonger] simultaneous exection of script on webpage [In reply to] Can't Post

Thank you, again !


Zhris
Enthusiast

Dec 17, 2014, 4:44 PM

Post #6 of 13 (5038 views)
Re: [ttamlrep] simultaneous exection of script on webpage [In reply to] Can't Post

Just to provide an alternative technique, you could store the users email and url in a database table, then your secondary script could be called regularly as a cronjob, processing new entries in this table accordingly. Using this technique, it might be easier to manage processes, ensuring your system isn't overwhelmed, and individual processes can be seamlessly restarted or continued automatically if a system error occurred etc. Nonetheless though, creating a background process directly from your CGI can be implemented in such a way that my points above would also be covered.

Regards,

Chris


ttamlrep
Novice

Dec 17, 2014, 5:57 PM

Post #7 of 13 (5031 views)
Re: [Zhris] simultaneous exection of script on webpage [In reply to] Can't Post

Thank you very much for your reply, Chris. This is very clever.

If the script is running everytime anew, how would I get it to know what is the correct entry in the database table to use, so that it does not repeat itself ? Delete the entry, I guess ?


Zhris
Enthusiast

Dec 17, 2014, 6:55 PM

Post #8 of 13 (5021 views)
Re: [ttamlrep] simultaneous exection of script on webpage [In reply to] Can't Post

I can think of a few ways this could be implemented.

Your delete the record suggestion would work well, but I would choose to mark records perhaps as "being processed", then "processed" under a status column instead. This way you can simply select records that have a null or "unprocessed" status. I would only consider deleting or archiving records if the table became too large.

Another option would be to store a past datetime in another table or other. Select records by ascending order of date that were created after this datetime. Once your script has finished, update the datetime to the datetime of the newest record you selected, OR, after each single record has been processed, update the datetime to the datetime of that record.

Ensure you lock your script when it begins and unlock it when it ends, this will prevent manual or overlapping cron runs which otherwise may attempt to re-process records currently being processed. You may also want to consider marking those records that error-ed or took too long to complete etc etc etc perhaps a threshold number of times, they can then be analyzed, then processed manually if need be.

If you need a more detailed implementation plan, feel free to ask. I probably haven't covered all possible race conditions above.

Regards,

Chris


(This post was edited by Zhris on Dec 17, 2014, 6:57 PM)


ttamlrep
Novice

Dec 17, 2014, 8:25 PM

Post #9 of 13 (5018 views)
Re: [Zhris] simultaneous exection of script on webpage [In reply to] Can't Post

Thank you very much, Chris. This is really nice.

One question, I am clueless as to how to lock a script. Could you provide just a quick explanation to get me started and I can google the details ?


Zhris
Enthusiast

Dec 18, 2014, 5:40 AM

Post #10 of 13 (4949 views)
Re: [ttamlrep] simultaneous exection of script on webpage [In reply to] Can't Post

I'm not certain what the best technique is but a trick I learnt at my last company was to create a plain text file for every script, which can be opened and flocked / locked at the start, then closed (and subsequently unflocked / unlocked) at the end. This way, if the script is cross-run, it will error upon attempting to lock when it is already locked. This post describes some excellent approaches to achieve the above.

Chris


(This post was edited by Zhris on Dec 18, 2014, 5:44 AM)


FishMonger
Veteran / Moderator

Dec 18, 2014, 6:22 AM

Post #11 of 13 (4944 views)
Re: [Zhris] simultaneous exection of script on webpage [In reply to] Can't Post

"Locking" the script implies that only 1 instance of the script should be running at any given time, which goes against one of the requirements in the opening post.

Quote
how can I make the script so that it can execute simultaneously on different requests ?


Now, the new question/requirement.

Quote
If the script is running everytime anew, how would I get it to know what is the correct entry in the database table to use, so that it does not repeat itself ? Delete the entry, I guess ?

We don't have enough info on what the script does or how the data is being used to be able to properly answer that question. However, instead of locking the script to a single instance, it may be more appropriate to do the locking at the DB (either table locking or row locking depending on the storage engine). Using a storage engine that supports transactions would be best and is what is needed to support row level locking.

If the processing script takes 45+ minutes to complete, my first suggestion would be to profile the script to see if there are bottlenecks that can be optimized.


(This post was edited by FishMonger on Dec 18, 2014, 6:23 AM)


Zhris
Enthusiast

Dec 18, 2014, 11:06 AM

Post #12 of 13 (4936 views)
Re: [FishMonger] simultaneous exection of script on webpage [In reply to] Can't Post

With regards to script level locking, it is common to implement a locking mechanism when dealing with cronjobs, its purpose is to prevent overlapping runs if a previous run takes longer than expected, its unrelated to preventing records being processed more than once, although inadvertently will aid with this. Depending on what the actual script does, locking is nearly always desirable as a precaution, overlaps are rarely necessary. Internal forking will take care of distributing the work load for the current batch of records.

With regards to record level locking, your suggestion of locking the row is an excellent idea, and might indeed mean that script level locking would become unnecessary. It won't help in deciding which batch of records need selecting for processing, but I feel I covered this with two ideas above ( checking status and/or datetime ).

Regards,

Chris


(This post was edited by Zhris on Dec 18, 2014, 11:15 AM)


ttamlrep
Novice

Dec 18, 2014, 11:48 AM

Post #13 of 13 (4926 views)
Re: [Zhris] simultaneous exection of script on webpage [In reply to] Can't Post

Thank you very much, Chris and FishMonger for your replies. I am learning a lot. This is very helpful.

Locking with cronjobs is something I was totally unaware that existed and which could be beneficial for me. I often set up a crontab in which our server is running 24 hrs. with a list of jobs and sometimes some jobs have taken much longer than expected running into new jobs and resulting in the 15 min load average going well over our cpu number.

Thanks for all the help. I now have something to think about and decide which will be the best solution for me.

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives