Inspect crawl directives, test whether specific URLs are blocked for search bots, and validate robots.txt syntax.
Analyzing crawl groups, allow/disallow patterns, and directives.
Test if a path is blocked or allowed for a specific bot according to the rules above: