Methods
VeronikAI is a search index of public AI agents: A2A agents and remote MCP servers. It records which agents answer, which follow their protocol, and how that changes over time. This page explains how we measure, adapted from the internal methods draft.
1. What we crawl
We read only public listing endpoints that need no account or key. Before crawling a site we check its robots.txt and terms. If robots.txt cannot be fetched, we skip that source for the run (fail closed).
| Source | What we read | How often |
|---|---|---|
| Official MCP Registry (registry.modelcontextprotocol.io) | Latest version of every server.json | Daily, incremental; full resync every 7 days |
| A2A Registry (api.a2a-registry.org) | Public agent list | Daily, full pass |
| Hugging Face Spaces tagged mcp-server | Space list via the public API | Daily, full pass with ETag. Only RUNNING Gradio Spaces get a probeable endpoint. |
| Neuronto (neuronto.com/agents) | Open agent feed | Rolling sweep of 400 pages a run |
| Smithery (registry.smithery.ai) | The first 10 pages an anonymous client can read | Daily |
We do not crawl Agent Galaxy (robots.txt disallows /api/), PulseMCP (key required; robots.txt disallows all), Glama (key required), NANDA Index, AGNTCY ADS and Localmark (no open bulk listing endpoint), or allagents.app (its terms forbid bulk extraction; stopped on 2026-10-31 and its rows removed).
Dedupe. One agent can appear in several sources. We merge listings into one agent using a key: A2A by the normalized agent-card URL with the well-known suffix removed (or the endpoint if there is no card), MCP by the normalized endpoint URL. Each agent keeps a record of which sources list it. An agent that no source lists any more is marked inactive. It is not deleted, so its history remains.
2. How probes work
- GET only. We never send MCP initialize, tools/call, A2A message/send or any POST. We never authenticate, never pay, and never call a tool.
- User-Agent: VeronikAI/0.1 (+mailto:[contact]; read-only directory crawler)
- A2A. We GET the card URL from the listing, then /.well-known/agent-card.json, then the legacy /.well-known/agent.json. Each card is read up to 512 KB.
- MCP. We send one GET to the declared remote URL with Accept: text/event-stream. We read at most 2 KB and then close the connection. On a 401 or 403 we also GET the OAuth protected-resource metadata (RFC 9728).
- Timeouts and retries. Each request times out after 10 s. Transient network errors are retried at most 3 times. After 3 consecutive network failures a host is skipped for the rest of the run.
- Our own failures. A canary checks whether our own connection is working. An outage on our side is never recorded as an outage on yours.
- Rate limits: at most 2 concurrent requests per host and 6 per site; at most 4,000 checks per site per run; pacing adapts after 429/503.
- robots.txt on every probe target. We honor the group for veronikai (legacy veronica-agents still honored). If robots.txt disallows a path, we record skipped (robots), never down.
An A2A card that returns HTTP 200 counts as up, even if the card fails conformance. 401/403 is up but fails conformance. 402 is up with conformance unknown. MCP treats 200, 202, 204, 3xx, 400, 401, 402, 403, 405, 406, 409, 415 and 421 as up: a server answered. 429, timeouts and unconfirmed resets are unknown. HTTP 402 (Payment Required) means a paid server is answering; we do not pay to look inside.
3. Conformance
Conformance is separate from uptime. An agent can be up and still fail.
A2A (agent card, v0.3, with v1.0 accepted): valid JSON with a non-empty name, description and version; an https URL (or v1.0 interface URLs); a capabilities object; non-empty default input and output modes; at least one skill with id, name, description and tags. A card that is not public (401/403) fails.
MCP v2 (introduced 2026-10-31) checks server.json quality and live transport with one GET. Servers listed only outside the official registry have no server.json to check; their conformance is n/a, and we report uptime only.
4. Revisit guarantee
Each daily run fills its budget with a uniform random sample (default 1,000), then every agent not checked in 7 days, then new and changed agents, then agents due under their own interval. Guarantee: every active agent is rechecked at least once every 7 days, as long as the daily budget is at least one seventh of the index. Every agent page shows the time of its last check.
5. Sampling
Ecosystem figures, such as the share of agents up, come from that run’s uniform random sample and are reported with 95% Wilson score intervals. Totals for each agent’s latest state are shown separately. We never present a partial run’s probe counts as the whole index. Headline numbers on this site are census counts from the snapshot, not sample estimates.
6. Retention
Daily rollups are kept indefinitely (this is the history). Individual check rows are kept 90 days once rolled up, and each agent’s latest check is always kept. We store no personal data beyond what owners publish in public listings. We do not store probe response bodies.
7. License
Listings and measurements are free to view and link to. Small excerpts are allowed with attribution (Source: VeronikAI, veronikai.com). Bulk copying, scraping, or redistributing the dataset needs written permission. Listings remain under their source registries’ and owners’ terms. See the data license page for the full wording. A future open protocol specification is planned under CC-BY 4.0.
8. Ranking
No one can pay to rank higher on VeronikAI. Ranking and ordering come only from our own checks and public data. Paid services, when they exist, never change position. See our no-paid-ranking pledge.
9. Disputes and opt-out
- Dispute a result. Send the agent URL and what you believe is wrong to [email protected]. We re-probe and log corrections.
- Opt out of probing. Add a robots.txt rule for veronikai (legacy veronica-agents still honored), or ask us, and we stop. Your listing stays visible as listed by its source but unchecked, unless you also ask for the listing to be hidden.