Well, I haven't had the time to investigate it in depth but from the look of it, it looks pretty nice.
BTW, If you haven't read Danny's Blog Interview about it, then
do read here.
Quote:
Why are you doing this?
We want to index all publicly available information so we can offer better search results. However, currently web crawling is limited. Crawlers don't know all the pages at a website (e.g., dynamic pages), when those pages change, how often to recrawl pages, how much load to put on a website. So they try to guess. We want to work collaboratively with webmasters to get a big picture of all the URLs we should be crawling, and how often they should be recrawled. Ultimately this benefits our users by increasing the coverage and freshness of our index.
|
06-03-2005, 02:50 PM
Linear Mode
