CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
Search Posts SEARCH
Who's Online WHO'S
Log in LOG

Home: Perl Programming Help: Beginner:
Mechanize Scraping Question


New User

Oct 6, 2011, 10:21 PM

Post #1 of 1 (559 views)
Mechanize Scraping Question Can't Post


I'm pretty new to Perl and have only used it for some data handling scripts.

I am having a bit of trouble with a scraping script. The script fills out form data, then submits the information. However, when I look at $mech->content{} after the form is submitted, I notice a lot of fields of the form:

<element>[text here]</element>

But if I submit the same form in my browser, and look at the markup, the bracketed text is replaced by values. Some places still contain square-bracketed text, but these are hidden.

The site is Travelocity. When you submit a flight search, a page comes up and for the first 5-10 seconds, it's "searching for flights" even though HTML has loaded.

Thus, my theory is that Mechanize is immediately grabbing the HTML because it thinks the page has loaded, where in reality, some JavaScript is probably loading content still.

Is there any way I can re-grab the content of the page after some specified time, or, more advanced, figure out when the JavaScript has finished its thing, then grab the content (Mechanize::Firefox??).

Any help would be appreciated. I apologize for my ignorance...


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives