What it means
The crawler opens a page, reads the server response and markup, then follows discovered links to further URLs.
Term in plain language
Crawling
Automated exploration of accessible URLs and links by a search crawler.
Why it matters
Without reliable crawling, search engines cannot discover new pages or notice changes on URLs they already know.
Example
A link from the blog hub leads the crawler to a new article, whose content is then sent for processing.
Related pageRelated terms
Continue exploring
SEO audit
A website review that connects an issue to a URL, priority and a repeatable acceptance check.
Indexing
Adding a processed page to the search-engine database used to build search results.
Search crawler
A search-engine program that opens accessible URLs and sends discovered data for processing.