CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Beginner:
help with writing a web crawler with LWP

 



dilbert
User

Sep 8, 2018, 9:06 AM

Post #1 of 5 (1732 views)
help with writing a web crawler with LWP Can't Post

good day dear experts,

need some help with writing a web crawler with LWP

found a great tutorial with fairly nice and helpful explanations:
but unfortunatly there is no help - with some hints how to step further.

see the video here: https://www.youtube.com/watch?v=2-kU-mKrYjM

Dr. Rob Edwards from San Diego State University shows how to use Perl and LWP::Simple to write a simple web crawler.
Perl part 6: Writing a web crawler with LWP

unfortunatly we cannot follow the tutorial - since the code is not visible.

can you help out here a bit.


that would be a great pleasure.. and help in learing


Zhris
Enthusiast

Sep 11, 2018, 7:42 AM

Post #2 of 5 (1719 views)
Re: [dilbert] help with writing a web crawler with LWP [In reply to] Can't Post

Hi Dilbert,

You have been looking into LWP::* for a long time now, we have provided various examples throughout. The youtube example is a very basic script, I'd say for those completely new to LWP. I suggest you Google basic LWP::Simple examples, run them, and practice modifying until you are confident. If you get stuck, ask specific questions rather than general ones, as it stands I'm not sure what kind of help you need or want.

Regards,

Chris


(This post was edited by Zhris on Sep 11, 2018, 7:42 AM)


bulrush
User

Nov 7, 2018, 2:59 AM

Post #3 of 5 (795 views)
Re: [dilbert] help with writing a web crawler with LWP [In reply to] Can't Post

Are you wanting to learn how a web crawler works? Because there is already a great web crawler to download whole websites out there.

- HTTrack is one of the best as, by default, it does not travel down to URLs towards the root relative to the URL you give it. It only goes "deeper" from the URL you give it. It has a CLI and Windows version and is generally easier for the beginner to use. https://www.httrack.com/page/2/

- wget is another alternative but it's a CLI and harder to use.
-----


dilbert
User

Nov 7, 2018, 3:54 AM

Post #4 of 5 (792 views)
Re: [bulrush] help with writing a web crawler with LWP [In reply to] Can't Post

Hello zchris hello bulrush,

- all i want to do is diving into Perl - so to run httrack and to have a simple and useful tool is _ not _ my intention.




In Reply To
Are you wanting to learn how a web crawler works? Because there is already a great web crawler to download whole websites out there.

- HTTrack is one of the best as, by default, it does not travel down to URLs towards the root relative to the URL you give it. It only goes "deeper" from the URL you give it. It has a CLI and Windows version and is generally easier for the beginner to use. https://www.httrack.com/page/2/

- wget is another alternative but it's a CLI and harder to use.


i think that perl is pretty useful and capable to do lots of things.

To do some useful things and to learn how the perlcode works is all my intention.


so i just wanted to see this above mentioned youtubecode - which is a simple crawler that runs with a few perl lines...


greetings


dilbert
User

Nov 7, 2018, 7:55 AM

Post #5 of 5 (777 views)
Re: [dilbert] help with writing a web crawler with LWP [In reply to] Can't Post

it would b e a great great pleasure if someone could help out with the youtube-code that is shown above...

many thansk for any and all help

dilbert

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives