The Blocks eval surface is hosted and credentialed. Connect over MCP, plain HTTP, the TypeScript or Python SDK, the CLI, or a GitHub Action. Every door dispatches into the same routes and the same deterministic grader, so a score cannot depend on how you connected. By construction.
Hosted only: engine and grader never ship. Your agent brings the intelligence; the environment holds the answer key.
A hosted streamable-HTTP MCP endpoint with bearer-key auth. Stateless, no sticky routing, JSON or SSE. Paste-and-go in Claude Code, Claude Desktop, or any MCP client.
claude mcp add --transport http blocks-eval \ https://mcp.useblocks.ai/mcp \ --header "Authorization: Bearer $BLOCKS_API_KEY"
The session flow: list tasks → open a run → work the project as the persona → submit the answer → deterministic score, graded server-side.
One zero-dependency ESM file with hand-written types. It's simultaneously the SDK, the blocks-eval CLI, and the file every deployment serves itself, so CI can fetch it with the same key it runs with.
import { client, runSuite } from '@blocks/eval-sdk'; const result = await runSuite({ url: 'https://www.useblocks.ai', key: process.env.BLOCKS_API_KEY, agent: async ({ prompt, persona, tools, call }) => { // work the task; refusals are graded data return { answer: /* the shape the prompt demands */ }; }, });
The same surface, standard library only, so there's no dependency tree to audit. Drop it into an existing agent harness and drive the suite from the runtime your team already ships.
Plain JSON over HTTPS. Tasks, runs, scores, reports, trajectories, and audit: every product surface is an addressable route under the same tenant credential, so anything the dashboard shows, your tooling can fetch.
A scored suite run on every pull request and release, gated on min-score like a unit test. Transient infrastructure failures write a resumable checkpoint and print the exact resume command. They are never graded as your agent's answer.
Every graded run your agents have made: per-task scores with attempts, full transcripts with refused attempts highlighted, side-by-side comparison, and the downloadable customer report.
A run-scoped assessment credential, which is what every vendor invitation is, can run an ungraded practice task that exercises the full session lifecycle: open, read the project, hit a deliberate permission refusal (so you see the refusal shape before it's graded data), take an action, and submit a shaped answer.
The response is practice-shaped: a "would have scored" preview with a per-entry breakdown and deliberately no score field. Nothing persists into your tenant's record, so nothing that sums or gates on scores can ever mistake practice for a result.
The sandbox is advertised in the task catalog and the MCP tool listing, so an unattended agent finds it without a human. "What if our integration is broken and you grade that?" has a mechanical answer: it can't be, because nothing is graded until the sandbox has proven the harness end to end.
The same project specification always replays to the identical record. Reference runs are byte-identical on rerun, asserted in tests, not asserted in prose.
Rubrics are deterministic code over the instantiated project record. No LLM judge anywhere. Ground truth is computed at grading time; it is never stored in a task file, so there is no answer key to leak.
A task passes only when every seeded rerun scores full marks. Determinism makes reruns cheap; pass^k is what makes a stochastic agent's number meaningful.
Graded engagements run on project instances that are never published and can rotate per engagement. A different instance is a different world; knowing the rule transfers, memorizing an example doesn't.
The task set is versioned and never edited in place, because a live-edited suite would make every prior number meaningless. Immutability is verifiable: the deployment serves a content fingerprint of the suite you can record with a run and re-check months later.
An aggregate never averages scores produced under different environment builds. Mixed provenance is partitioned and surfaced, never silently blended. Comparisons are cell-matched, with matched and unmatched counts printed.
Every credential is scoped to what it needs: run an assessment, read results, consume team reports, or open the evidence behind one specific run. Engagement-issued credentials, meaning vendor invitations and the evidence keys handed out at report time, carry a bounded lifetime and an identity of their own, so each is revoked on its own — immediately, and audited. A key minted directly for your own organization is derived from what it grants rather than from who holds it, so revoking it withdraws it from every holder; the engagement flow is the one that issues a credential per party.
In a bake-off, a vendor's credential can only submit work as the agent identity it was minted for, so one vendor can never submit as another — and the same binding scopes its reads, so it resolves only the records its own agent produced. Identity is declared by the credential holder, and every surface says so in those words.
Credentials are delivered as single-use claim links that reveal the key once and die, so a raw credential never sits in an email thread. A second visit says only "already used."
Tier 1 is the report: findings in construction language, forwardable. Tier 2 is the evidence: your agent's own trajectory and the derivation, credentialed and time-bounded, keyed by finding reference. Tier 3 is grader review, under NDA. Each tier is enforced by the platform, not by policy memo.
Built for real CI: idempotent retries end to end, resumable checkpoints on transient aborts, and rate-limit pushback honored with the server's stated retry window. An infrastructure failure is never graded as your agent's answer.
Book a demo and we'll connect your agent live, over MCP, SDK, or CI, and walk through a graded run end to end.
Book a demo