Home: Perl Programming Help: Beginner:
help with writing a web crawler with LWP



dilbert
User

Sep 8, 2018, 9:06 AM


Views: 1274
help with writing a web crawler with LWP

good day dear experts,

need some help with writing a web crawler with LWP

found a great tutorial with fairly nice and helpful explanations:
but unfortunatly there is no help - with some hints how to step further.

see the video here: https://www.youtube.com/watch?v=2-kU-mKrYjM

Dr. Rob Edwards from San Diego State University shows how to use Perl and LWP::Simple to write a simple web crawler.
Perl part 6: Writing a web crawler with LWP

unfortunatly we cannot follow the tutorial - since the code is not visible.

can you help out here a bit.


that would be a great pleasure.. and help in learing


Zhris
Enthusiast

Sep 11, 2018, 7:42 AM


Views: 1261
Re: [dilbert] help with writing a web crawler with LWP

Hi Dilbert,

You have been looking into LWP::* for a long time now, we have provided various examples throughout. The youtube example is a very basic script, I'd say for those completely new to LWP. I suggest you Google basic LWP::Simple examples, run them, and practice modifying until you are confident. If you get stuck, ask specific questions rather than general ones, as it stands I'm not sure what kind of help you need or want.

Regards,

Chris


(This post was edited by Zhris on Sep 11, 2018, 7:42 AM)


bulrush
User

Nov 7, 2018, 2:59 AM


Views: 337
Re: [dilbert] help with writing a web crawler with LWP

Are you wanting to learn how a web crawler works? Because there is already a great web crawler to download whole websites out there.

- HTTrack is one of the best as, by default, it does not travel down to URLs towards the root relative to the URL you give it. It only goes "deeper" from the URL you give it. It has a CLI and Windows version and is generally easier for the beginner to use. https://www.httrack.com/page/2/

- wget is another alternative but it's a CLI and harder to use.
-----


dilbert
User

Nov 7, 2018, 3:54 AM


Views: 334
Re: [bulrush] help with writing a web crawler with LWP

Hello zchris hello bulrush,

- all i want to do is diving into Perl - so to run httrack and to have a simple and useful tool is _ not _ my intention.




In Reply To
Are you wanting to learn how a web crawler works? Because there is already a great web crawler to download whole websites out there.

- HTTrack is one of the best as, by default, it does not travel down to URLs towards the root relative to the URL you give it. It only goes "deeper" from the URL you give it. It has a CLI and Windows version and is generally easier for the beginner to use. https://www.httrack.com/page/2/

- wget is another alternative but it's a CLI and harder to use.


i think that perl is pretty useful and capable to do lots of things.

To do some useful things and to learn how the perlcode works is all my intention.


so i just wanted to see this above mentioned youtubecode - which is a simple crawler that runs with a few perl lines...


greetings


dilbert
User

Nov 7, 2018, 7:55 AM


Views: 319
Re: [dilbert] help with writing a web crawler with LWP

it would b e a great great pleasure if someone could help out with the youtube-code that is shown above...

many thansk for any and all help

dilbert