CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX 		Search Posts SEARCH
POSTS 		Who's Online WHO'S
ONLINE 		Log in LOG
IN

Home: Perl Programming Help: Intermediate:
LWP Multiple Times

 


perlkid
stranger

Jun 10, 2000, 4:30 AM

Post #1 of 6 (452 views)
LWP Multiple Times Can't Post
 
Hello,

I have a data base full of urls. 275 urls to be exact. How can I get the information from the urls and print the information to a database specified in the second field of the database with the urls using lwp?

I tried

use LWP::Simple;
foreach $line (@all)
{
@array=split(/\|/, $line);
$content = get("$array[0]");
open(DB, ">/$array[1].db");
print DB "$content";
close(DB);
}

but I couldn't get it to work, and I don't want to have to have use $content = get("url"); 275 times to get the data off the other server. Frown

Thanks,

perlkid


TheGame+
Deleted

Jun 13, 2000, 4:18 AM

Post #2 of 6 (452 views)
Re: LWP Multiple Times [In reply to] Can't Post
You could use the mirror() function from the same module. It really helps to RTFM, you know.

<BLOCKQUOTE><font size="1" face="Arial,Helvetica,sans serif">code:</font><HR>

#!/usr/bin/perl -w
use LWP::Simple;
use strict;

# @all is filled somehow, assuming there are no trailing newlines and that file paths are correct
foreach my $line (@all) {
my @array = split(/\|/,$line);
my $rc = mirror($array[0],"$array[1].db");
if (is_error($rc)) {
print "Error retrieving or saving $array[0] : $rc ",status_message($rc),"\n";
}
else {
print "$array[0] : $rc ",status_message($rc),"\n";
}
}
</pre><HR></BLOCKQUOTE>

Note that you will also get a "500 Internal Server Error" if the file cannot be saved for some reason - e.g. if it has an invalid local path.


perlkid
stranger

Jun 17, 2000, 12:47 AM

Post #3 of 6 (452 views)
Re: LWP Multiple Times [In reply to] Can't Post
 
Thanks A lot TheGame+,

That code works great. I worked with it all day and I got it to do the 275 urls in 9 parts. Do you know a way that I can do the 9 parts/categories all at the same time. Parallel? It takes a long time to get everything now and I need it to be under 5 min. It does it in 25-30 min. right now.

It's ok If you can't spare the time. You have helped me so much already. Smile

Thanks TheGame+ Smile

perlkid


perlkid
stranger

Jun 17, 2000, 12:52 AM

Post #4 of 6 (452 views)
Re: LWP Multiple Times [In reply to] Can't Post
 
I forgot to mention that the nine parts are all in their own cgi's. Cat1.cgi , Cat2.cgi ... Cat9.cgi and the html file prints and I frame to the first cgi and when it's finished getting everything it prints the location (using javascript) to the next cat script. So if I could make a script to run all nine at the same time, It would be GREAT! Smile

Thanks,

perlkid


Kanji
User

Jun 17, 2000, 5:03 AM

Post #5 of 6 (452 views)
Re: LWP Multiple Times [In reply to] Can't Post
perldoc LWP::Parallel

... or is that too obvious a choice? Smile


perlkid
stranger

Jun 17, 2000, 8:06 AM

Post #6 of 6 (452 views)
Re: LWP Multiple Times [In reply to] Can't Post
 
That sounds like a good choice but after reading the doc I couldn't figure out what to do. Frown Must be my inexperience. Would you know how I can put the doc into action?

Thanks Smile

perlkid

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives