1arryb
User
Jul 30, 2009, 7:13 AM
Post #5 of 8
(887 views)
|
|
Re: [sarthakganguly] Mechanize : parsing a html content
[In reply to]
|
Can't Post
|
|
Hi sarthakganguly,
There is no general answer to your question: Each page is different, and so will require a unique parsing strategy. The only good thing is that sites like blogs are usually written by a program, and so have a regular structure. You'll have to download one of the pages and study it. Once you understand the page structure, you can use the methods of your HTML parser package to zero in on the parts of the page you're interested in. In your particular case, you need to identify some HTML artifact (e.g. a tag name, CSS class, or text) that always marks the article. Then you search the parse tree (assuming you're working with HTML::TreeBuilder) for that artifact and extract the article text.
Sometimes you have to do 2D analysis of the tag hierarchy (e.g. Find the tables, get the css div called "123456" inside the 3rd inner table, get the table inside the "123456" div, get the text of the paragraphs inside the table cells). It can get tedious.
Good luck,
Larry
(This post was edited by 1arryb on Jul 30, 2009, 7:15 AM)
|
MAIN
SEARCH
WHO'S 
LOG 
Mechanize : parsing a html content
. So after that I get the html content from that page by the mechanize->content function I want to parse only the content part. And I want to exclude all other things on the page(like ads,blogroll,menu,header,footer etc.etc.). Can you suggest any generelized approach to solve the same. I don't want to write specific regex for only excluding blogroll and other things. 
. If you need robust, you have to talk to the site maintainer and see if you can negotiate some kind of structured data feed.