20%
Discovery-crawler access
What we check
We parse Allow and Disallow rules for OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot against the checked page path. Matching follows the most-specific user-agent group and longest rule; an equal-length Allow wins. User-triggered fetchers, training crawlers and Google-Extended are classified separately and do not affect this score.
Why this weight
A declared block can stop a discovery crawler fetching the page. The five checks are equally weighted as technical coverage, not market share. A 404 or 410 means no declared restriction; 403, 429, server errors, timeouts and invalid HTML responses remain Unknown rather than being counted as Allowed.