What is measured
The audit requests /robots.txt, checks the response, and parses basic directives including the sitemap reference.
Technical access and indexing
robots.txt file
This check confirms that the crawler rules file exists and can be read.
Pass criterion
The file is available at the standard path, contains no accidental global block, and points to the current sitemap.
What to fix
Publish a simple root robots.txt and avoid rules that do not match the real route structure.
Automation limit
A complex robots.txt does not improve rankings by itself. Its job is to avoid blocking useful pages and to close utility URLs economically.
Reference