Harness Platform M4: preserving agent-run state through failure

01 / MILESTONE CONTRACT

M3 made the agent loop permission-aware. M4 asks what survives a crash.

Harness Platform M4 turns one connection-owned run into a durable control-plane path: digest-bound task snapshots, fenced worker leases, PostgreSQL session replay, content-addressed artifacts, automatic audit export, and a fail-closed Kubernetes topology.

The important word is durable, not distributed. The release defines how state advances when workers retry, disappear, reconnect, or disagree. It refuses to call an uncertain tool outcome successful and refuses to make a stale worker authoritative again. That is the control-plane foundation needed before a real cluster can safely add scale.

M4 — Control plane & scale
├── services/control-plane
│   ├── digest-bound task snapshots + durable run state
│   ├── leases, heartbeats, fencing, and reconciliation
│   ├── transactional event outbox
│   └── artifact registry + automatic audit export
├── durable storage
│   ├── PostgreSQL: tasks, runs, sessions, events, checkpoints
│   └── S3-compatible objects: outputs, reports, audit JSONL
├── ACP session restore
│   ├── explicit last-seen sequence cursor
│   └── interrupted work closes without replaying effects
└── Kubernetes reference topology
    ├── control plane + agent server
    ├── Postgres + MinIO reference stores
    └── isolated, suspended sandbox Job template

default evidence lane
├── deterministic offline tests
└── injected PostgreSQL and S3 protocol fakes

not claimed
├── a live cluster deployment
├── exactly-once external side effects
└── load, soak, or capacity evidence

02 / FROM SERVICE TO CONTROL PLANE

The kernel stays small while ownership moves into durable infrastructure.

M3 wrapped the kernel in one permissioned WebSocket service. Its session belonged to one connection, its audit history lived locally, and its advertised replay capability was false. That was honest and sufficient for a single process. It was not enough for a worker pool in which a process can die after producing evidence but before updating its bookkeeping—or after beginning an external effect whose result is unknown.

M3                                      M4
────────────────────────────────────    ─────────────────────────────────────
one run owned by one connection          durable task and run ownership
local SQLite session log                 PostgreSQL or SQLite session store
sessions capability: false               ACP cursor restore when durable store exists
fire event observer                       awaited durable publication
local audit stream                        transactional outbox + JSONL export
files named by a process                  immutable object registry + SHA-256
Docker-per-run plan                       Kubernetes service topology contract
connection lifecycle is authority          graceful cancel + lease-expiry restore
no distributed run recovery                run lease expiry + operator reconciliation

M4 does not move policy or tool execution into the scheduler. The control plane owns task/run admission, state, leases, artifacts, audit progress, and run reconciliation. The agent server owns durable session ownership, ACP restore, and the permissioned kernel session. Both use the typed event stream as the evidence contract, and both can share PostgreSQL without collapsing into one service.

03 / CHECKS-GATED DELIVERY

One task contract became one checks-gated public merge.

The implementation was committed as fee081e, submitted through pull request #2, and merged into public main at d3b2859. The task contract intentionally denied network access and Git push during implementation; commit, hosted checks, and merge happened only in the explicit release follow-up.

M3 merge
defbf7bcf72fc72452b4adc81b099f3fc6c523cf
        ↓ one task contract · one implementation commit · one PR
fee081ecb8bc0f353f48c21dfc9e94aa53b8ab83
        ↓ pull request #2 · all required checks green
d3b2859a48cfb794472d30805ea91b47dc1086d0
M4 merge

authoritative public diff
102 files changed · 12,934 insertions · 484 deletions

These figures come from the public pull request and Git comparison. The development page's smaller editor summary measured a transient working session, not the committed M3-to-M4 range, so it is not used as release evidence.

04 / FENCED SCHEDULING

A worker holds a temporary capability, not permanent ownership.

Admission validates the complete task manifest, canonicalizes it, hashes it, and stores the immutable snapshot beside a caller-supplied idempotency key. Repeating the same admission returns the same record. Reusing the key for different bytes is a conflict. Scheduling applies the same rule to a run, so a lost HTTP response cannot create a second logical task or run by accident.

admit(manifest, admissionKey)
  → validate TaskManifest
  → canonical JSON + SHA-256 digest
  → insert immutable snapshot or return the identical retry

schedule(task, admissionKey)
  → queued
  → lease(workerId, leaseId, fencingToken, expiresAt)
  → running after an owner-checked start
  → passed | failed | blocked | canceled

expiry
  leased  → queued
  running → indeterminate

operator reconciliation
  indeterminate → queued | canceled

every worker mutation must match
runId + workerId + leaseId + fencingToken + unexpired storage-clock lease

A claim returns a worker ID, opaque lease ID, expiry, and monotonically increasing fencing token. Start, heartbeat, and completion require all four ownership values and a still-live lease. PostgreSQL evaluates expiry with its own clock. That prevents two machines with skewed clocks from independently deciding that they own the same run, while the fencing token prevents the old owner from committing after a takeover.

queued ──claim──▶ leased ──start──▶ running ──complete──▶ passed
  │                 │                         ├──────────▶ failed
  │                 └──lease expires──▶ queued├──────────▶ blocked
  └──cancel────────────────────────────▶ canceled
                                            │
                         running lease expires
                                            ▼
                                      indeterminate
                                       │         │
                                retry  │         │ cancel
                                       ▼         ▼
                                     queued   canceled

The most important transition is the least optimistic one. An expired leased run has not crossed the durable start boundary and is safe to requeue under the worker protocol. An expired running run may already have crossed an external boundary, so it becomes indeterminate. Only an operator, using the current row version, can retry or cancel it.

05 / DURABLE SESSIONS AND REPLAY

Recovery replays committed history and refuses to invent a successful turn.

The sessions package now exposes one store contract with SQLite and PostgreSQL implementations. Both validate events on write and read, append through the store contract, assign monotonically increasing per-session sequences, lease the active owner, and paginate by an explicit last-seen cursor. PostgreSQL additionally rejects event-row updates and deletes with a trigger. When a durable store is configured, the agent server can advertise ACP session replay without making its transport the source of truth.

session/restore {
  sessionId,
  afterSeq,   // last durable sequence the client has seen
  limit
}

response {
  status: completed | interrupted,
  replayedFromSeq,
  replayedThroughSeq,
  replayedEvents,
  hasMore
}

guarantee
├── replay only committed events where seq > afterSeq
├── ascending order, at-least-once delivery
├── active owner lease must expire before takeover
├── terminal agent.stopped closes bookkeeping without a fake event
└── nonterminal tail appends one interrupted marker and closes;
    it never repeats the uncertain model request, permission, or tool effect

If an active session's owner lease is still valid, another connection cannot restore it. After expiry, recovery inspects the durable tail. A terminal agent.stopped means only the row-close bookkeeping was lost, so the session closes as completed. For a nonterminal tail, recovery atomically appends one session.restored event with outcome interrupted and closes the session. The uncertain turn is never executed again during restore.

before model request
  await publish(model.request)
  → then call model

before tool side effect
  await publish(tool.call)
  await publish(policy.decision)
  await publish(permission.resolved when required)
  → then invoke tool

if durable publication fails
  abort the run
  do not cross the effect boundary

This orders evidence before an effect.
It does not make an external effect exactly once.

This required changing event observation from fire-and-forget to awaitable publication at model and tool boundaries. A persistence failure now stops the loop before the next side effect. Replayed committed events are delivered at least once, so clients still deduplicate by event ID and sequence; recovery is not an exactly-once claim.

06 / ARTIFACTS, OUTBOX, AND AUDIT

State commits with an ordered outbox; evidence publishes later.

Task, run, artifact, and event-producing audit-checkpoint domain mutations enqueue their typed event in the same PostgreSQL transaction. The empty-page audit bookkeeping case advances its checkpoint without creating a recursive event. The outbox assigns a stable event ID and commit-ordered outbox sequence. Its publisher later appends the event to the global session-event sequence used by audit export. If delivery succeeds but acknowledgement is lost, the sink may see that same event ID again and must treat it as an idempotent retry.

artifact bytes
  → bound size and content type
  → SHA-256 digest
  → conditional S3-compatible put
  → hash existing bytes on a conflict
  → immutable PostgreSQL metadata

audit export
  redacted canonical events
  → deterministic newline-delimited JSON
  → content-addressed object
  → upload object
  → commit immutable registry row + checkpoint in one DB transaction
  → for a non-empty export, enqueue artifact.registered + audit.exported
  → for bookkeeping-only input, emit neither recursive event

download
  → bounded SigV4 URL response
  → never store the URL in events, reports, registry, or logs

Artifact uploads are size-bounded and write-once through the service contract. A default-generated key carries the SHA-256 digest; a conditional-write conflict causes the registry to read and hash the existing object rather than trust object metadata. PostgreSQL rejects updates and deletes to artifact rows. Out-of-band object mutation remains an ACL and operator concern. The audit exporter reads the already-redacted canonical event stream, produces deterministic JSONL segments, uploads the object, and then commits the registry entry and stream checkpoint together.

07 / KUBERNETES TOPOLOGY

The base refuses readiness while critical environment choices remain placeholders.

Docker Compose remains a local-development aid. The M4 deployment contract is raw Kustomize: two application services, two persistent reference stores, separate service accounts, health probes, resource bounds, disruption budgets, topology spreading, two namespaces, Restricted Pod Security, and default-deny ingress and egress with only the required service paths reopened.

infra/kubernetes/
├── harness namespace
│   ├── control-plane Deployment ×2 + Service + PDB
│   ├── agent-server Deployment ×2 + Service + PDB + HPA
│   ├── Postgres StatefulSet ×1 + PVC
│   ├── MinIO StatefulSet ×1 + PVC
│   └── default-deny plus explicit service paths
└── harness-sandboxes namespace
    ├── Restricted Pod Security + quota + limits
    ├── service account with token automount disabled
    ├── deny-all networking
    └── ConfigMap containing a suspended Job template

fail-closed base
├── five example.invalid image sentinels
├── two REPLACE_STORAGE_CLASS placeholders
├── example Secrets excluded from rendered output
└── no executor, Job materialization RBAC, or workspace staging

Applying the base may create partial resources, but it cannot become a ready, working deployment as-is: five workload images point to example.invalid sentinels, both persistent stores require an unresolved storage class, and the four example Secret files are excluded from rendered output. The reference Postgres and MinIO StatefulSets each have one replica; their disruption budgets do not turn them into highly available services.

The sandbox path is even more explicit. Kustomize stores a suspended Job template inside a ConfigMap; it does not create a Job. No M4 process reads that ConfigMap, no Role or RoleBinding grants Job creation, and no trusted component stages workspace PVCs. A future executor must validate every image, command, mount, deadline, workspace, and network choice before it can materialize and unsuspend a run.

08 / WHAT BROKE

The review kept turning “durable” nouns into ordered failure behavior.

The shared development record is useful for chronology, not authority. The public commit, tests, public task-report comment, and pull request establish what shipped. The development record shows why the final design is more conservative than the first implementation.

1. 01

   Publishing an event did not initially block the side effect

   A synchronous observer shape could begin persistence and continue into a model or tool call. M4 made the boundary awaitable so a failed durable append stops execution before the effect. Cleanup must then leave the session active rather than falsely close it; lease expiry lets restore record the nonterminal turn as interrupted.

2. 02

   A dead running worker could not safely return to the queue

   Requeueing would imply nothing happened. Running lease expiry now produces indeterminate, preserving uncertainty until an operator examines the external system and chooses retry or cancel.

3. 03

   Worker clocks were not a safe ownership oracle

   Lease checks moved into storage-clock predicates and every active worker mutation gained a fencing token. A late heartbeat or completion from an old owner cannot advance the canonical row.

4. 04

   State and events could diverge between two writes

   An event emitted after committing state could be lost during a crash. The fix was a transactional outbox with stable IDs and commit order. A proposed homegrown database transport was replaced with the maintained pg driver and bounded pool behavior.

5. 05

   Audit retries could skip or mislabel evidence

   Object bytes, registry metadata, sequence range, and checkpoint now form one idempotent chain. The checkpoint moves only after object upload and the atomic registry/checkpoint commit, and a conflicting completion or artifact key must match exactly. The exporter filters out audit.exported and audit-kind artifact.registered events; other artifact registrations remain evidence, so export cannot recursively feed itself.

6. 06

   A green readiness endpoint could hide a dead background path

   Outbox publication and audit draining now mark readiness unhealthy immediately on failure and healthy only after recovery. Connection and request concurrency are bounded so HTTP pipelining cannot create an unbounded work queue.

09 / VERIFIED RESULT

The committed evidence validates deterministic contracts, not a deployed service level.

At the exact public merge, an isolated source verification passed all 421 tests in 32 files, strict TypeScript, and a complete Kustomize render. The render produced 32 objects and no Secret objects. The generated M4 JSON is intentionally Git-ignored, but its full contents were preserved in a public pull-request comment: 421 tests, all 102 changed paths inside policy, and zero violations.

$ git checkout d3b2859a48cfb794472d30805ea91b47dc1086d0

$ pnpm test
Test Files  32 passed (32)
Tests       421 passed (421)

$ pnpm typecheck
# exit 0

$ kubectl kustomize infra/kubernetes
# exit 0 · 32 rendered objects · 0 Secret objects

public M4 report comment
├── status: passed
├── changed paths: 102
├── path-policy violations: 0
└── tests: 421 / 421

Pull request #2 passed four checks: the exit-gate workflow, a separate CodeQL status check, and the Actions and JavaScript/TypeScript analysis jobs in the CodeQL workflow. The verified merge followed only after all required checks were green.

10 / CURRENT TRUTH

M4 is a durable control-plane contract with deliberately unproven operations.

The accurate description is production-shaped durable control plane. It is not a production Kubernetes deployment, a multi-tenant authorization system, an exactly-once executor, a highly available storage platform, or evidence of throughput at scale.

11 / FILE GUIDE

Each durable boundary has one inspectable home.

12 / WHAT IS NEXT

The next milestone is operational proof, not another architecture noun.

1. 01

   Build a real environment overlay

   Supply reviewed immutable images, external Secrets, encrypted storage classes, TLS, narrowly scoped egress, and target-cluster admission checks.

2. 02

   Separate migration power from runtime power

   Run one-shot migrations, give control plane and agent server independent least-privilege roles, and prove backup plus restore for PostgreSQL and artifacts.

3. 03

   Implement the Kubernetes executor

   Add narrowly scoped namespace RBAC, trusted workspace staging, exact template substitution, run-specific network policy, cleanup evidence, and denial tests.

4. 04

   Exercise real failure

   Run PostgreSQL and S3 integration tests, kill owners between every durable/effect boundary, force outbox and audit recovery, and test multi-replica takeover.

5. 05

   Make external effects reconcilable

   Carry provider idempotency keys where available, store effect receipts, and define operator playbooks for every indeterminate tool class.

6. 06

   Earn the word scale

   Only after security review, scoped service identity, observability, and restore drills should the project publish retained load, soak, chaos, and capacity results.

13 / EVIDENCE LEDGER

Every public claim resolves to a pinned source or named evidence class.

• M4 merge d3b2859 — the source pin for every implementation claim in this note.
• Pull request #2 — authoritative diff, checks, and merge chronology.
• M4 task contract — accepted scope, guarantees, offline test lane, permissions, and delivery boundary.
• Public run-report comment — the full generated report: 421 tests, 102 changed paths, zero policy violations, and task-session metadata. The corresponding local JSON is intentionally Git-ignored.
• Architecture, event ordering, security contract, and Kubernetes operations guide — repository-owned explanations of the boundaries shown here.
• Publication audit — an isolated verification of the pinned merge, used for the 421-test, typecheck, rendered-object, and no-Secret results. The private development conversation supplied chronology only and is intentionally not linked as public implementation evidence.