CGI/Perl Guide | Learning Center | Forums | Advertise | Login
Site Search: in

  Main Index MAIN
INDEX
Search Posts SEARCH
POSTS
Who's Online WHO'S
ONLINE
Log in LOG
IN

Home: Need a Custom or Prewritten Perl Program?: I Need a Programmer for Freelance Work:
Reply to: crawl & analyze generic web pages.
 
Subject:
Guest Username:
E-mail address:
Post Style:      Get Markup Help
Icon:
none









Post:




Replying to: crawl & analyze generic web pages. by mixxed
Post: hi, i'm new to this forum so bare with me a bit :)

i'm looking for a script/app that would perform the following actions in the first stage of the project:

Start from: a database with links and priorities.

1. read the list and determine the highest link to crawl
2. crawl the page and store locally.
3. create a database of such links
4. analyze the page to determine:
- identify primary content areas (exclude header/footer, banners, features)
- type of page (blog, forum,news,etc..),
- page last modified, commented, etc.
- various page-related or site-related parameters
- other various scores (based on formulas I provide)
5. analyze the primary content
- similarity to previously known contents (trained bayesian or smth similar)
- recency of the content
- retrieve the main keywords
6. store scores&values from #4 and #5 in a database.


I'm expecting this to be a 1-2 months project.
Is there anyone here that could get this done and provide a quote ?

M.


Search for (options) Powered by Gossamer Forum v.1.2.0

Web Applications & Managed Hosting Powered by Gossamer Threads
Visit our Mailing List Archives