Search Engine Spiders Lost Without Guidance - Post This Sign!
To see the proper format for a somewhat standard robots.txtfile look directly below. That file should be at the root of the domain because thatis where the crawlers expect it to be, not in some secondary directory.Below is the proper format for a robots.txt file ----->User-agent: *Disallow: /cgi-bin/Disallow: