Robots handling
These pages describe checker version VeronikAI/0.1+attestations.1 (signing.py). A2A card checks use suite veronikai-a2a-get-card version 1. MCP checks use suite veronikai-mcp-conformance version 2. If the code changes, the version string on a signed result should change too.
What is checked
Many websites leave a note at /robots.txt that tells automated visitors which doors they may use. The note is a public courtesy file defined by RFC 9309. We read it before we crawl a registry or probe an agent, and we obey it.
A simple way to picture it: if a shop window says "no robots past this door", we do not go in — even if the door is unlocked.
How it works
There are two slightly different readers, both GET-only:
- Registry ingest (common.robots_allows in common.py, used by ingest.py). If the file is missing (HTTP 404/410), there are no extra limits. If the file cannot be honestly read, the whole source is skipped for that run and retried next time.
- Per-agent probes (robots.py, used by probe.py). Each origin (scheme + host) gets its own policy. Results are remembered for 24 hours when the file is readable (VERONIKAI_ROBOTS_TTL_H), and for 1 hour when the server is overloaded (HTTP 429 or 5xx).
We look for a group meant for us, in this order: veronikai, then the older name veronica-agents, then *. The longest matching path rule wins; if two rules are the same length, allow wins. The file /robots.txt itself is always allowed. * and $ wildcards are honored. The first 500 KB of the file is read.
If the first fetch looks unreadable or refused, we retry once after about 5 seconds (VERONIKAI_ROBOTS_RETRY, default on). That retry can be switched off.
How a fetch becomes a policy
| What we got | What we do |
|---|---|
| HTTP 2xx and plain robots rules | Parse and obey them. Cached 24 hours. |
| HTTP 4xx except 429 (for example 404) | No extra restrictions. Cached 24 hours. |
| HTTP 429 or 5xx | Treat as a full disallow for this origin (RFC 9309). Cached 1 hour. |
| Bot-challenge page, HTTP 401/403, or an HTML app page served as robots.txt | Unreadable. Skip this host (or this registry) for the rest of the run. Do not save a "you may go anywhere" policy. Try again next run. |
| Network failure (no server answered) | For probes: there is no robots policy to read, so the probe itself goes ahead and records down/unknown. Not cached as a long-term allow. For ingest: skip that source this run. |
A genuine empty Disallow: means "allow everything". That is different from an HTML page pretending to be robots.txt.
What pass and fail mean
Robots is not a pass/fail grade for the agent. It is a pass/fail for us:
- If we are allowed in, we probe and record up, down, or unknown as usual.
- If we are not allowed in, or the file is unreadable, the check is skipped (robots) with a reason. That is not an outage of the agent.
- Owners who want us to stop can add a User-agent: veronikai (or veronica-agents) Disallow rule.
Limits
- We cannot invent a policy when a site serves its web app at /robots.txt. Today that is why the AGNTCY catalog source reads 0 rows: https://ai-catalog.outshift.io/robots.txt returns the catalog's HTML shell (text/html), so we fail closed.
- Caching means a new robots file may take up to 24 hours to be noticed (or until the next run, for an unreadable file).
- The switch VERONIKAI_ROBOTS=0 would disable probe-time robots checks. It stays on.
Checker version
VeronikAI/0.1+attestations.1. Skip reasons are stored in the check's error and reasons fields, starting with skipped (robots):.