Eval framework for Claude Code, OpenCode,
and Codex that uses real tools against
real data. Evals are built from actual tasks — someone asked a question
in Slack, debugged a production issue, or traced a config problem through
logs and APIs. The agent solves the same task using the same MCP tools:
searching messages, querying service catalogs, reading docs, calling
internal APIs.
The problem is that a good searcher will find the exact thread the eval
was sourced from, or the post-mortem doc, or the incident report — and
the answer is right there. e2eval intercepts the LLM API the agent
talks to (Anthropic for claude/opencode, OpenAI Responses for codex) so
you can surgically scrub these sources from tool results, redact leaking
references, and observe exactly what evidence the agent used — without
disabling the tools it needs to actually solve the problem.
How it works
┌── beforeEval (rewrite prompt) │ agent ──────► proxy ────► LLM API (or self-hosted gateway) │ ├── onToolCall (Koa-style: short-circuit, mutate args, transform result) ├── toolCalls[] (one entry per call after the chain settles, in-process) │ └── afterEval (grade once agent finishes; receives toolCalls + answer)
The proxy sits between the agent and the LLM API. When the LLM emits
a tool_use block, the proxy pauses before forwarding it to the agent
and runs the middleware chain. Middleware can:
Short-circuit (return ToolResult without calling handler) —
the agent still runs the tool, but the LLM only ever sees the
middleware's synthetic result on the next request. Same shape an
exit 2 PreToolUse hook would produce in the conversation history.
Modify the request (await handler(mutatedArgs)) — the proxy
rewrites the tool_use.input before the agent reads it; the tool
runs with the new args.
Modify the response (await handler(args) then transform the
return) — tool runs with the original args; the LLM sees a
middleware-transformed result.
The chain suspends across HTTP cycles: when middleware calls handler,
the proxy emits the (possibly-mutated) tool_use to the agent, the
agent runs the tool locally, and on the next request the proxy resolves
handler's promise with the real tool_result.
Quick start
import { startRunner, claudeAdapter } from 'e2eval';// Or: opencodeAdapter, codexAdapter — same API.import { myGrader } from './middleware/my-grader.js';const runner = await startRunner({ adapter: claudeAdapter, middleware: [myGrader],});const ran = await runner.run({ name: 'deploy-rollback', question: `Service X had a bad deploy last Tuesday. What config flag was reverted to fix it?`, config: { expectedAnswer: 'enable_canary_bypass' },});console.log(ran.answer); // what the agent saidconsole.log(ran.results.myGrader); // whatever myGrader.afterEval returnedconsole.log
Middleware
Middleware is the extension point. A scrubber that redacts a source
thread from search results, a grader that checks correctness via LLM,
and a logger that prints tool calls are all middleware — they just
implement different lifecycle methods.
Phase
When
Use case
beforeEval
Before the agent runs
Modify the prompt
onToolCall
Wraps each tool call (Koa-style)
Block, mutate args, or transform results
afterEval
After the agent answers
Grade the answer; whatever you return lands in result.results[<name>]
onToolCall runs exactly once per call, in declaration order. Each
middleware receives handler — descend by calling await handler(args),
short-circuit by returning a ToolResult without calling it.
async onToolCall({ tool, input, handler }) { if (tool !== 'mcp__search__messages') return await handler(input); // Short-circuit: tool not run, LLM sees this synthetic result. if (input.query.includes('source-thread-id')) { return { content: [{ type: 'text', text: 'blocked' }], isError: true }; } // Modify request: tool runs with sanitized args. if (input.query.includes('PII')) { return await handler({ ...input, query: redact(input.query) }); } // Modify response: tool runs as-is, transform what LLM sees. const real = await handler
Every context exposes ctx.abort(reason) — any phase can kill the run
(rejects runner.run() with reason). If a middleware throws, the run
fails loudly instead of being silently swallowed. Each run also has a
shared ctx.data bag that flows through every phase (and every
middleware) — use it to pass values from beforeEval/onToolCall into
afterEval without going through module-level globals.
Both Config (read-only, set by the eval) and Data (read-write, scratch
during the run) are extensible via declare module:
runner.run(ev, { timeoutMs }) aborts the run when the budget elapses;
the agent is killed and runner.run() rejects. Default is no timeout
beyond the adapter's own per-spawn cap.
Block semantics
When middleware short-circuits (returns a ToolResult without calling
handler), the proxy redirects the call to a benign shell tool the
agent always has — Bash for claude, bash for opencode,
exec_command for codex — encoding the synthetic content as base64
args. The original tool — built-in, Read, MCP, anything — never runs.
On the next request the proxy restores the original tool name + input
in conversation history and replaces the tool_result content with the
exact ToolResult you returned, so the LLM sees a coherent transcript
with the original tool name. Middleware names must be unique within a
run; duplicates throw at startRunner.
callLLM
Middleware that needs an LLM (e.g., for grading) gets ctx.callLLM in
afterEval. It runs claude with all tools denied, so it's a single-shot
text generation.
afterEval returns whatever you want; the runner stores it under
result.results[<middleware-name>]. Returning undefined skips the
entry. Later middleware can read prior middleware's return via
ctx.results[<earlier-name>].
callLLM(prompt, { resume: true }) forks the eval session so the LLM
inherits the agent's context (tool calls, intermediate state) — useful
for graders that need to inspect long tool outputs claude truncated in
the transcript.
Adapters
import { claudeAdapter, opencodeAdapter, codexAdapter } from 'e2eval';
All three present the same AgentAdapter interface — swap them by
changing the adapter: passed to startRunner. Each adapter knows how
to spawn its CLI and point it at the proxy:
Adapter
CLI
Wire format
How the proxy URL is injected
claudeAdapter
claude
Anthropic
inline --settings env.ANTHROPIC_BASE_URL
opencodeAdapter
opencode
Anthropic
inline --config with the named provider's baseURL overridden
For claude specifically, we override via --settings because its
precedence puts --settings.env above ~/.claude/settings.json env
and process env — without it, a user with ANTHROPIC_BASE_URL already
in settings.json would bypass the proxy.
Upstream
The proxy forwards to whatever upstream: you pass to startRunner.
Each adapter has its own fallback chain when you don't pass one:
Adapter
Resolution order
claudeAdapter
upstream: → process.env.ANTHROPIC_BASE_URL → hook discovery (asks claude itself what it would resolve) → https://api.anthropic.com
Hook discovery (claude only): on first proxy start with no
upstream: and no env var, we run a one-shot
claude -p x --no-session-persistence --settings ... whose
UserPromptSubmit hook captures $ANTHROPIC_BASE_URL and exits
before any API call. That gives us whatever claude itself would
resolve from its full settings chain (managed → user → project →
env), without us mirroring the precedence rules. ~3-4s on first call,
cached after that. Override or skip it by passing upstream:
explicitly or setting ANTHROPIC_BASE_URL.
runner.run(ev, { record: true }) captures every upstream round-trip
(raw request + response bytes) and exposes them on
EvalResult.recording. Pass that recording to a later run via
replay: { recording, upTo } to reproduce the conversation
deterministically through the first upTo LLM exchanges, then
diverge: the proxy switches to the live upstream once upTo is hit,
and any tool calls inside the replayed prefix are auto-short-circuited
with the recorded results — so the agent sees the original transcript
byte-for-byte until the cutoff, then makes real LLM calls with whatever
new tools / MCP servers / config you've added since.
const baseline = await runner.run(ev, { record: true });// You add a new MCP server, fix a tool, etc. Now ask: would this// failure resolve at turn 5 with the new setup?const after = await runner.run(ev, { replay: { recording: baseline.recording!, upTo: 5 },});
Cleanup
By default the runner creates a deterministic per-eval scratch dir at
<cwd>/.eval_runs/<eval-name>/ (used as the agent's cwd, so its
session JSONL lives in a stable ~/.claude/projects/<encoded-cwd>/
that callLLM({ resume }) can find across processes). The directory
is removed on runner.close() and on SIGINT/SIGTERM/exit.
If you run multiple startRunner instances concurrently against the
same project tree (e.g. a parallel test runner), pass
cleanupOnClose: false so one runner's cleanup doesn't yank dirs out
from under another's in-flight evals — and clean up yourself once all
runners are done. The E2EVAL_NO_CLEANUP=1 env var flips the default
to off, useful for npm test:
Eval definitions — questions sourced from real tasks, with config
A run script that composes the middleware stack and drives execution
Testing your middleware
createClaudeTestAdapter({ respond }) (and createOpencodeTestAdapter,
createCodexTestAdapter for the other agents) swaps the upstream for an
in-process fake — the agent CLI itself still runs (so middleware that
targets MCP or native tools sees real tool dispatch), but the model's
responses are scripted. Use it for hermetic e2e tests with no API spend.
Pass an array (one response per turn) for canned scripts, or a callback
(req, { turnIndex }) => ScriptedResponse to assert on what reached the
LLM after middleware ran.
For unit-testing a single middleware without claude at all, drive
startChain directly with a mock backend handler.