Agent test protocol (v1)

agent_test_report is a review type unique to Crail: instead of a human opinion, an AI agent runs a fixed, versioned task suite against the product and the full transcript is published alongside the result — auditable by anyone, not just summarized. Protocols are published before testing begins so they can't be adjusted after the fact to favor a result.

Status

The two protocols below are published and versioned; no test runs have been executed against them yet on Crail. We are not publishing placeholder or fabricated agent_test_report reviews — real reports will appear on vendor pages once test runs have actually been carried out and their transcripts recorded.

Protocol: AI Coding Agents (crail-atp-coding-v1)

• A fixed set of 10 real, closed GitHub issues (pulled from public repositories, mixed difficulty) is given to the agent under test.
• For each issue: the agent attempts a fix with no human intervention; pass/fail is determined by whether the resulting diff passes the repository's own existing test suite.
• Recorded per task: pass/fail, wall-clock time, token/API cost incurred, and the full raw transcript.
• Reproducibility: each task suite run is re-run once independently; a reproducibilityScore reflects how consistent results were across runs.
• Same task set, same repositories, same scoring rule used for every vendor tested under this protocol version.

Protocol: LLM & Agent Infrastructure (crail-atp-infra-v1)

• A fixed sample LLM application (a small RAG chatbot) is instrumented using the product under test, following only its own public quick-start docs.
• Recorded per task: whether traces/spans appear correctly, whether cost/latency are attributed accurately, whether eval scoring can be configured, and time-to-first-useful-dashboard.
• Recorded: pass/fail per sub-task, wall-clock time, and the full raw transcript (commands run, screenshots of the resulting dashboard).
• Same sample application and same setup steps used for every vendor tested under this protocol version.

Versioning

Protocol changes get a new version id (e.g. crail-atp-coding-v2) rather than silently editing v1 — old reports stay attributed to the protocol version they were actually run under.