| |||||||||||||||
![]() | ![]() | ![]() |
| |||
| prevent spider grabbing URLs.... Is there any way to prevent a spider from grabbing URLs that you want to keep off search engines? thanks...... _________________ outdoor fountain atlanta fha mortgage |
| |||
| Re: prevent spider grabbing URLs.... Yep. You just need to create a text file called robots.txt and put it in the root folder of your website. This file allows you to exclude certain URLs from search engines. Basically, you just identify which robots you want to exclude (* wildcard for all) and then list the directories or files you want them to stay away from. Example: Code: User-agent: * Disallow: /hiddenfolder/ |
| Thread Tools | |
| Display Modes | |
| |
| | ||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| baidu spider and referer spam | Fritz M | Internet Marketing | 3 | 06-08-2005 01:33 AM |