A text file placed in the root directory of a website that prohibits search engine crawlers from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See
crawler.