View Single Post
  #2 (permalink)  
Old 05-26-2008, 01:46 AM
fatnewt fatnewt is offline
Entry Level
 
Join Date: Jul 2004
Location: Ottawa, Canada
Posts: 8
Re: prevent spider grabbing URLs....

Yep. You just need to create a text file called robots.txt and put it in the root folder of your website. This file allows you to exclude certain URLs from search engines.

Basically, you just identify which robots you want to exclude (* wildcard for all) and then list the directories or files you want them to stay away from. Example:

Code:
User-agent: *
Disallow: /hiddenfolder/
Wikipedia shows some more examples.
__________________
Colin Temple
Web Marketer, XAdvance
(follow me on twitter)
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote