What it means
The file tells crawlers which sections they may request and which utility paths they should avoid.
Term in plain language
robots.txt
A text file at the website root containing crawl rules for selected paths and search crawlers.
Why it matters
An overly broad rule can block an important section, while missing rules may waste crawl activity on filters and utility URLs.
Example
The administration area is disallowed while CSS, images and public pages remain crawlable.
Related pageRelated terms