Nov 2, 2009, 6:25 AM
Post #1 of 1
Perl + web2 crawler
I'm trying to write a web2 crawler :
Links including daynamic links which are generated by XML-HTTP-Request , Ajax + All the web2 fetures.
I know it's quite complicated to write this crawler, so i was thinking of using a "Browser Emulator" to do it,
The idea is to pipe the website URL to the "Browser Emulator" and by it's API to "Click" on each link on the website and collecting the results
does any one konw how to do it or a well known tool that do it (it must be in Perl) ?
thanks in advance for any help.