Perl Guru Forums: Perl Programming Help: Intermediate: I need a script to dig out certain information from a site.

CGI/Perl Guide | Learning Center | Forums | Advertise | Login

Site Search: in


	MAIN INDEX	SEARCH POSTS	WHO'S ONLINE	LOG IN

Home: Perl Programming Help: Intermediate:
I need a script to dig out certain information from a site.

falazar
Deleted

Feb 1, 2000, 11:02 AM

Post #1 of 8 (985 views)

I need a script to dig out certain information from a site.	Can't Post

I need a script to dig out certain information from a site.

Such as the temp in a certain city from a site or something like this... I want to work on a few little bot programs that are self sufficient and get information from a source and do thigns from tehre.... but first I have to start with an "easy" way of getting info from a page....
thanx... pls email me lettuce98@hotmail.com

Jasmine
Administrator

Feb 1, 2000, 3:48 PM

Post #2 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

If you want to fetch content from a page, you should get familiar with LWP. You can search for it at CPAN.

There's a few modules available for getting weather (search at CPAN)

If you want to see an example of fetching info, check out ABCNews Retriever

[This message has been edited by Jasmine (edited 02-01-2000).]

falazar
Deleted

Feb 3, 2000, 6:40 AM

Post #3 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

well .... is there any Simple way of doing something like this.... I would like to start off very easy, just be able to pull up the contents of a webpage and look thru it...

falazar
Deleted

Feb 5, 2000, 8:39 AM

Post #4 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

Ok, found a code piece that halfway works, you can call `lynx -dump $url`
and that will save a dumped html text file, which I can then search, only problem is this will only work from the command line, running the cgi off unix, not when I call it as a webpage.... is there any way to get around this?
... also simple question.. I want to then search the page for keywords, what is the easiest way to just make a nice @list of words?

Jasmine
Administrator

Feb 5, 2000, 1:08 PM

Post #5 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

If you want to simply nab a page you can use LWP.

<BLOCKQUOTE><font size="1" face="Arial,Helvetica,sans serif">code:</font><HR>

#!/usr/bin/perl -w
#
# Perl script to download a Web page.
#

use LWP::UserAgent;

# The agent helps you get web documents.
$webagent = new LWP::UserAgent;

print "Content-type: text/html\n\n";

$url = 'http://www.perlarchive.com/index.shtml';

$request = new HTTP::Request(GET => $url);

# Get web document.
$response = $webagent->request( $request );

if ( $response->is_success ) {
print $response->content, "\n";
}
</pre><HR></BLOCKQUOTE>

$response->content contains the page you've just fetched.

falazar
Deleted

Feb 5, 2000, 3:54 PM

Post #6 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

How do I go about loading the lwp stuff?
when i try to run above I get this...

Can't locate LWP/UserAgent.pm in @INC (@INC contains: /usr/local/lib/perl5/5.005
02/alpha-dec_osf /usr/local/lib/perl5/5.00502 /usr/local/lib/perl5/site_perl/5.0
05/alpha-dec_osf /usr/local/lib/perl5/site_perl/5.005 .) at temp.cgi line 6.
BEGIN failed--compilation aborted at temp.cgi line 6.

Cure
User

Feb 5, 2000, 4:34 PM

Post #7 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

Hi alazar:

That means you dont have the UserAgent.pm module. You need to d/l the module and install the module on your system.

Cure

Red Dragon
Deleted

Feb 11, 2000, 8:50 PM

Post #8 of 8 (985 views)

Re: I need a script to dig out certain information from a site. [In reply to]	Can't Post

If I may make a suggestion here. If you have root access to the machine (I'm assuming unix here because thats my forte`) you can run
perl -MCPAN -e shell
answer the questions that is asks you, then when finished type this and hit enter

install Bundle::LWP

This should do an "automated" install of the LWP and its depend modules on your system for you.

Search for (options)

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives