Harness Platform M2: making evaluation evidence credible

01 / MILESTONE CONTRACT

M1 made a run inspectable. M2 makes the evidence more credible.

Stage 1, Milestone 2 adds a controlled target, one shared evaluation vocabulary, two evidence surfaces, and a real external protocol boundary—while keeping the default gate deterministic and free of live service dependencies.

The owner-supplied development conversation records a handoff after work with Pi, Ollama, and a local Qwen model stopped during M2. It does not contain the terminal failure or a reliable root cause. The repository and branch state place the work that Codex resumed in the final MCP stdio slice. I do not attribute the stop to the model, memory pressure, MCP, or any other unrecorded cause.

M2 — Eval credibility
├── one deliberately tiny golden HTTP repository
├── scenario YAML validated by @harness/sdk
├── read-only board over manifests + run reports
├── harness events projected into OpenTelemetry
└── initialize-era MCP client over stdio

boundary rules
├── deterministic test execution uses no external MCP server
├── malformed external data becomes a typed error
├── observability is optional and cannot kill a run
└── network compatibility lives in a separate lane

02 / FROM M1 TO M2

The operator loop had evidence, but little calibration or interoperability.

At the M1 pin, the harness could persist events, render them in a terminal, run one FakeModel scenario, compile process policy, and repeat the exit gate in CI. Its own note called that scenario a calibration seed rather than eval credibility. The web app was a placeholder, no telemetry connected the event stream to standard tooling, and packages/mcp exported wire schemas without a live client.

The important design choice is additive. M2 does not replace the event stream with traces, the report with a dashboard, or the deterministic fixture with a network dependency. It adds standard views and one explicit live-compatibility lane around the same local core.

03 / FIVE DOGFOODED TASKS

Each credibility claim arrived through its own manifest and linear commit.

M2 is a five-commit chain from the verified M1 baseline. Each slice has a committed task contract with goal, acceptance criteria, path scope, permissions, budget, and delivery shape. The task branches were fast-forwarded into main; there were no public pull requests in this sequence.

a596fc5  M1 operator-loop baseline
   │
ca9d7b7  golden hello-service
   │
01c8048  SDK-owned scenario DSL
   │
abcfded  read-only web task board
   │
4a61be4  event stream → OpenTelemetry
   │
8f18f6d  hardened MCP stdio client + live lane

1. ca9d7b7A zero-dependency golden HTTP repository, written contract, and seven loopback checks.
2. 01c8048Scenario YAML, typed invariants, and parse failures moved into the shared SDK.
3. abcfdedA GET-only local task board over validated manifests and generated reports.
4. 4a61be4One event bridge tested on rooted kernel runs, plus incomplete exit-gate CLI wiring.
5. 8f18f6dThe hardened stdio lifecycle, 19 offline MCP tests, and locked compatibility lane.

Git records 52 changed files, 5,949 insertions, and 261 deletions from M1 to M2. The development chat’s smaller file and line summary does not match that committed range, so this note uses the public Git diff. Generated tasks/runs/*.json files and the SQLite database remain intentionally ignored; the hosted gate uploads its own evidence artifact instead.

04 / GOLDEN REPOSITORY

The first calibration target is small enough to know exactly what “correct” means.

hello-service is a one-file node:http service with no runtime dependency. Its SPEC.md fixes the public behavior before an evaluator attempts a change: loopback binding, response codes, content type, JSON shape, URL decoding, and the distinction between an unknown path and a disallowed method.

GET /health
→ 200 application/json
→ { "status": "ok" }

GET /hello/:name
→ 200 application/json
→ { "greeting": "hello <decoded-and-trimmed-name>" }

unknown path → 404 { "error": "not found" }
wrong method on known path → 405 { "error": "method not allowed" }

runtime: Node 22 · node:http · 127.0.0.1 · zero dependencies
test: start on an ephemeral port → fetch real responses → close

Its standalone test starts the real server on an ephemeral loopback port and uses fetch to check seven outcomes. That is stronger than unit-testing the handler in isolation and intentionally weaker than pretending one toy service represents a production repository corpus.

05 / SHARED SCENARIO LANGUAGE

YAML becomes typed public invariants before the runner sees it.

M1’s scenario decoder lived beside the eval runner. M2 moves the vocabulary into @harness/sdk, next to task-manifest and run-report validation. A scenario can assert run status, step and tool counts, final text, budget warnings, ordered event subsequences, and an optional exit-gate report status.

const eventInvariant = z
  .object({ type: z.string().min(1) })
  .passthrough()
  .superRefine((value, context) => {
    if (!isEventType(value.type)) typedError(context)

    for (const [key, expected] of Object.entries(value)) {
      if (key === 'type') continue
      if (!key.startsWith('data.')) typedError(context)
      if (!isScalar(expected)) typedError(context)
    }
  })

export function loadScenario(yamlText: string): Scenario {
  return decodeScenario(parseYamlOrThrowTypedError(yamlText))
}

// The eval runner re-exports this SDK vocabulary instead of forking it.

Unknown event types, keys outside type and data.<path>, non-scalar expectations, empty expect blocks, and invalid YAML all become a ScenarioParseError. The old eval import surface survives as a re-export, so moving ownership does not fork the language or break callers.

06 / READ-ONLY TASK BOARD

The web surface exposes evidence without becoming a second source of truth.

apps/web is deliberately plain Node. Each request re-reads task YAML and run JSON from disk, validates both through the SDK, groups reports by task, and sorts each report list newest first. Invalid manifests and reports appear in dedicated typed arrays; they are never silently dropped from the operator’s view.

request
├── method !== GET
│   └── 405: board is read-only
├── /api/board
│   └── re-read + validate every manifest and report
├── /api/tasks/:id
│   └── task + newest-first reports, or typed 404
├── /api/reports/:file
│   ├── reject traversal-shaped names
│   ├── validate run-report/v1
│   └── invalid artifact → typed 422
└── anything else
    └── typed 404

refresh model: explicit browser refresh
not present: writes · polling · websocket · authentication

The HTML page and four JSON endpoints are GET-only. Report names cannot contain a slash or traversal segment, unknown objects return 404, malformed reports return 422, and any non-GET request returns 405. There is no framework, database, mutation endpoint, polling, or websocket. “Refresh” means pull the latest files again.

07 / OPENTELEMETRY

The typed event stream can now speak a standard observability dialect.

EventBridge is the only module that knows both harness events and OpenTelemetry. On a complete kernel stream, session.created starts the root span, model requests and tool calls become children, and agent.stopped closes the run. Kernel budget and error events attach to that root; policy, task, and run events do the same only when they arrive while a root is active. Four counters track model turns, model tokens, tool calls, and budget warnings.

session.created  → start harness.session root span
agent.started    → attach task, agent, and model attributes
model.request    → start harness.model.request child span
model.response   → close child + count turns and tokens
tool.call        → start harness.tool.call child span
tool.result      → close child + count calls and status
budget.warning   → counter + audit event on the root span
policy.decision  → audit event on the root span
error            → exception + audit event
agent.stopped    → status + close the run span

The eval runner feeds the complete kernel lifecycle into this bridge, so it creates the root and child spans above. The CLI wiring reaches the same bridge but its exit-gate sequence begins with task.updated and never emits agent.stopped. Without session.created, no active root exists, so that CLI sequence currently creates no spans or counters. The shared wiring is present; an end-to-end CLI trace is not.

This still avoids a telemetry-only event model and keeps the original serialized stream authoritative. Observer failures are swallowed deliberately: a broken exporter must not turn a valid harness run into a failed run.

# absent: no telemetry object, exporter, or network path
pnpm evals

# explicit local console trace
HARNESS_OTEL=1 pnpm evals

# explicit OTLP/HTTP trace + metric exporters
OTEL_EXPORT_OTLP_ENDPOINT=http://127.0.0.1:4318 pnpm evals

Telemetry is fully off unless the operator opts in. With no environment switch, no telemetry instance is created. HARNESS_OTEL=1 selects console spans; an OTLP endpoint selects HTTP exporters for traces and metrics; an injected exporter and reader keep tests in memory. The pinned collector is behind the otel Compose profile and can be started explicitly with docker compose -f infra/docker/docker-compose.yml --profile otel up otel-collector.

08 / MCP STDIO BOUNDARY

The final slice turns message shapes into a defensive subprocess protocol.

Before M2 finished, @harness/mcp could validate a few wire envelopes but could not launch an MCP server. McpStdioClient now owns process startup, initialize-era negotiation, newline-delimited JSON-RPC, request correlation, tool discovery and calls, notification delivery, timeouts, failure fan-out, and shutdown.

type McpClientState =
  | 'idle'
  | 'starting'
  | 'running'
  | 'initializing'
  | 'initialized'
  | 'closing'
  | 'closed'
  | 'failed'

supported revisions
├── 2025-11-25  advertised
├── 2025-06-18  accepted
└── 2025-03-26  accepted

unknown negotiated revision → MCP_UNSUPPORTED_PROTOCOL_VERSION

Startup uses shell: false, piped stdio, a dedicated POSIX process group, and a small allowlist of launch-safe environment variables unless the caller deliberately supplies a complete environment. That matters because a child tool server is a new trust boundary; it should not inherit arbitrary tokens merely because the parent process has them.

client                                  subprocess
  │  spawn(shell:false, pipes, restricted env)  │
  ├─────────────────────────────────────────────>│
  │  initialize { version, capabilities, info } │
  ├─────────────────────────────────────────────>│
  │<──────────────────────── result { version }  │
  │  notifications/initialized                  │
  ├─────────────────────────────────────────────>│
  │                                              │
  │  tools/list · tools/call · ping (IDs)        │
  ├─────────────────────────────────────────────>│
  │<──────────── out-of-order responses (IDs)    │
  │<──────────────────── server notifications    │
  │                                              │
  │  stdin.end → wait → SIGTERM → wait → SIGKILL│
  └─────────────────────────────────────────────>│

Each request receives a monotonic JSON-RPC identifier and a timeout. Responses may arrive out of order because the pending map owns correlation. Fragmented stdout waits for a newline; coalesced output becomes individual frames. Server notifications take a separate observer path, and observer exceptions cannot corrupt request handling. The client answers server ping requests and rejects other server-initiated methods with JSON-RPC -32601.

bounded input
├── 4 MiB maximum newline-delimited stdout frame
├── 16 KiB retained stderr tail
└── 1,000 remembered IDs for late timed-out responses

typed terminal failures
├── spawn / premature exit / stream / write
├── malformed or uncorrelated JSON-RPC
├── unsupported protocol revision
└── initialize timeout and close timeout

typed precondition failures
├── call before start → MCP_NOT_STARTED
└── call before initialize → MCP_NOT_INITIALIZED; retry is valid

recoverable request failures
├── JSON-RPC tool error
└── ordinary request timeout → cancel + ignore late reply

Ordinary request timeout is recoverable: the client drops the pending entry, sends a cancellation notification when possible, remembers that identifier, and ignores the late response. Initialize timeout is terminal because the negotiated state is unknowable. Closing is idempotent and escalates from stdin EOF to SIGTERM and then SIGKILL if the process group refuses to exit.

09 / TWO VERIFICATION LANES

Determinism and real compatibility no longer have to compromise each other.

Nineteen offline MCP tests, backed by a repository-local hostile fixture, cover envelope validation, fragmented and coalesced frames, concurrent out-of-order replies, server requests, asynchronous observer failure, JSON-RPC errors, late responses, initialize and close races, a broken stdin, secret-shaped environment variables, malformed messages, crashes, spawn errors, and an unsupported revision. Once dependencies are installed, test execution needs no registry or external MCP reference server.

name: mcp live stdio
on:
  workflow_dispatch:
  schedule:
    - cron: "17 3 * * 1"

default pull-request / push lane
└── 19 offline MCP tests · hostile fixture · no reference server

scheduled / manual compatibility lane
├── exact-pinned GitHub Actions, pnpm 10.34.5, Node 22.23.2
├── frozen dedicated lockfile; lifecycle scripts disabled
├── fetch first, then install offline
└── @modelcontextprotocol/server-everything@2026.8.18
    └── initialize → 13 tools → echo → ping

The separate workflow is manual or scheduled for Monday at 03:17 UTC. It never runs in the default pull-request or push lane. Its actions, Node, pnpm, and Everything reference-server version are exact-pinned. It fetches the locked graphs, switches to offline installation, disables lifecycle scripts, verifies the installed package identity, starts it in a temporary directory with a scrubbed environment, discovers tools, calls read-only echo, and pings.

This separation follows the 2025 MCP stdio transport: one client-launched subprocess, newline-delimited JSON-RPC over stdin/stdout, and logs on stderr. The repository supports the initialize family through 2025-11-25. MCP’s breaking 2026-07-28 stateless lifecycle removes that handshake, so the repository names it as a future compatibility adapter instead of silently approximating it here.

10 / WHAT BROKE

The handoff exposed the difference between a type surface and a process boundary.

The shared conversation says only that the local Pi/Qwen work stopped during M2 and asks Codex to complete it. Because no failing command or stack trace is present, there is no defensible story about a single original bug. The repository does show why the final slice was qualitatively harder: it crosses asynchronous process startup, bidirectional streams, protocol negotiation, concurrency, timeouts, process groups, and shutdown races.

1. 01

   A schema could not own lifecycle

   Valid JSON-RPC shapes did not say when a request was legal, whether initialization won a close race, or how a child crash rejects every outstanding operation.

2. 02

   A stream could not be treated as one message

   One stdout chunk may contain half a frame or several frames. The implementation needed a bounded newline buffer before schema validation and routing.

3. 03

   A timeout could not poison later work

   The client had to reject the caller, cancel when possible, remember the abandoned identifier, ignore its late response, and keep unrelated pending requests intact.

4. 04

   A passing fixture could not prove ecosystem compatibility

   That required one pinned official server—but in a lane whose network and supply chain cannot make every pull request nondeterministic.

The test suite became the failure ledger. Instead of claiming those races are impossible, it forces each one and asserts the public state or typed error that follows.

11 / VERIFIED RESULT

The public gate, a disposable clone, and the live fixture prove different things.

GitHub’s CI run at the pinned commit passed strict typecheck, 15 Vitest files with 123 tests, the one golden-kernel scenario, the canonical exit gate, and evidence upload. The parallel CodeQL run also passed.

$ pnpm install --frozen-lockfile --ignore-scripts
# 19 workspace projects · lockfile accepted

$ pnpm typecheck
# exit 0

$ pnpm test
Test Files  15 passed (15)
Tests       123 passed (123)

$ pnpm evals
1/1 scenarios passed

$ node --test  # inside hello-service
tests 7 · pass 7 · fail 0

$ for task in tasks/m2-*.yaml; do pnpm harness validate "$task"; done
# 5/5 M2 manifests valid

$ pnpm harness run tasks/m2-mcp-stdio.yaml --branch tasks/m2-mcp-stdio
# status passed · 123/123 · zero changed-path violations

$ pnpm --filter @harness/mcp test:live
protocol 2025-11-25 · 13 tools · echo + ping passed

I repeated typecheck, the full suite, the eval, every M2 manifest validation, and the M2 exit gate from a disposable clone of 8f18f6d. M2 adds 43 tests to the workspace suite—7 SDK, 12 web, 5 OpenTelemetry, and 19 MCP—and another 7 standalone hello-service checks. Hosted CI proves the 123 workspace tests, not a combined “130-test” lane.

12 / CURRENT TRUTH

M2 makes stronger evidence possible; it does not finish the platform.

The largest boundary is execution. MCP tool descriptions, annotations, inputs, and outputs remain untrusted protocol data. @harness/mcp is not imported by the kernel or internal tool registry, and no M3 sandbox or approval flow stands behind a discovered tool. M2 proves the transport can behave; it does not authorize that transport to act.

13 / FILE GUIDE

The milestone stays readable because each boundary has one obvious home.

14 / WHAT IS NEXT

M3 moves from credible local evidence to contained services.

1. 01

   Connect MCP through the harness boundary

   Adapt discovered tools into the internal registry, map every execution through policy and audit events, add reconnection and remote transports, and implement the 2026-07-28 lifecycle separately.

2. 02

   Make the golden repository participate

   Add task-to-repository scenarios that edit and verify the calibration target, then expand to representative tools, failures, and provider-backed runs.

3. 03

   Build the execution substrate

   M3 owns the ACP agent server, one isolated container per run, scoped mounts, default-deny networking, a provider adapter, and interactive approval flows.

4. 04

   Turn configured lanes into retained public evidence

   Run the scheduled MCP workflow, retain its artifact and logs, add the standalone golden checks to an explicit CI lane, and make merge protection enforce the gate.

15 / EVIDENCE LEDGER

Conversation for chronology; pinned source and reruns for claims.

• Harness Platform at 8f18f6d is the public source pin for every implementation statement.
• M1 → M2 comparison proves the five-commit, 52-file milestone range.
• Hosted CI run 33408149721 proves the default lane at the pin; its gate evidence artifact contains the generated report and SQLite file. GitHub reports that the artifact expires on November 29, 2026, so the durable proof is the run record and pinned source rather than that download alone.
• CodeQL run 33408152006 is the security-analysis result at the same commit.
• MCP 2025-11-25 transport specification defines the stdio framing and subprocess relationship implemented here.
• MCP 2026-07-28 release note explains the breaking stateless lifecycle that remains outside this adapter.

The owner-supplied shared conversation establishes who resumed the work and the order of the handoff. It is not treated as proof for file counts, test results, or protocol behavior. Those claims come from the pinned repository, hosted checks, and the independent verification recorded above.