CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Perl Programming Help: Advanced:
Website Data Capture

 



jamescw0
New User

Jul 1, 2009, 8:09 AM

Post #1 of 2 (1331 views)
Website Data Capture Can't Post

Hey guys, Im looking for some help with my perl script. Let me give you some info first before I begin to explain my script. My goal is to capture data from a website. The website is designed in a tree format meaning that the main page only contains links and within those links contains the data I want to capture. Also, in the main page, there is an area for login/password info. I have completed 50% of this already as my script parses through the links to capture the data. My problem is that i found out that if you login to the website, there is extra info within the links that I need to capture. So, can anyone tell me or show me how I can pass through the login information. Also, the extra data I need to capture are all in drop down menus. I am having a hard time with both of these issues. Please help.


1arryb
User

Jul 5, 2009, 8:28 AM

Post #2 of 2 (1295 views)
Re: [jamescw0] Website Data Capture [In reply to] Can't Post

Hi James,

WWW::Mechanize can automate a login dialog.

If the information you want to scrape is in a dropdown menu, that implies javascript. You could try the WWW::Mechanize::Javascript plugin for that.

If this is your first web spider, please read http://mingo.info-science.uiowa.edu/eichmann/www-f94/ethics/ethics.html before unleashing it on the world.

Cheers,

Larry

 
 


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives