What it means
It is an automated visitor that follows links and checks which pages the website allows it to read.
Term in plain language
Search crawler
A search-engine program that opens accessible URLs and sends discovered data for processing.
Why it matters
If the crawler cannot open an important page, that page cannot enter search regardless of copy quality.
Example
The crawler finds a new service through the menu and adds its URL to the crawl queue.
Related pageRelated terms