DevSprint Copilot: a tool-loop agent with human-approved writes and a bidirectional MCP story

The problem

I wanted a project that demonstrates agent architecture (tool design, human-in-the-loop approval, streaming UI, MCP interop) to reviewers who have seen a hundred thin chat-wrapper demos. DevSprint Copilot is a project/DevOps copilot that operates a mock software-delivery workspace through chat: ask how the sprint is going, close issues, draft release notes, kick off a simulated deploy, and watch the effects land on a board pane next to the conversation.

The workspace is deliberately fake and deliberately deterministic: one project, two sprints, 13 issues on a five-column board, four users, five deployments, all seeded from fixtures with a frozen demo clock (WORKSPACE_NOW = 2026-07-01T09:00:00Z). That is not a shortcut; it is the eval strategy. When the seed never changes, “DS-11 is blocked” and “8 of 31 points done” are exact assertions, and the same six scripted scenarios can serve as acceptance spec, trajectory-eval suite, and Playwright E2E spec without reinterpretation.

Architecture

The front end is a Vue 3.5 SPA on Vite 8 (Rolldown) with two panes: workspace board left, chat right. The backend is Hono on Cloudflare Workers running the AI SDK v6 ToolLoopAgent entirely server-side. Model keys live only in Worker env vars, and the browser never talks to a model provider. Shared types, route constants, and the fixture set live in a workspace package both sides import.

The agent gets exactly eleven tools, inventoried up front with Zod input schemas and a shared result envelope ({ ok: true, data } | { ok: false, error }). Tools are classified by risk: read/query tools run unattended; rich-data tools feed generative-UI cards (burndown chart, release-notes diff); every write is gated behind an in-stream approval card, one approval per write; an Approve all press answers a group of cards through the same per-card path (ADR-0023). The click the user makes is a UX affordance; the Worker re-validates the tool input and re-evaluates the approval policy before executing, because client-supplied message history could otherwise forge an approval that never happened.

The MCP story was staged deliberately (see the decision log below). Phases 3–5 ran all ten tools as plain function calls inside the API Worker; phase 6 moved the four issue tools (five since assign_issue joined) behind a custom MCP server on Workers, publicly reachable over Streamable HTTP for external hosts like Claude Desktop, and the backend consumes it as an MCP client itself. One system sits on both sides of the protocol, and that is the differentiator. The transport-agnostic result envelope is what made the move a refactor rather than a rewrite. The diagram shows the built topology: the deploys and analytics servers, drawn dashed, were never built, so those six tools stay plain function calls in the API Worker.

Evaluation design

No number is published yet; every row in the metrics table above is pending, and I would rather show an empty table than an invented one. What exists is the harness, and an honest account of what it can and cannot prove.

The eval unit is the trajectory, not the answer. A plausible paragraph about sprint health is worthless if the agent never called get_sprint_status; the interesting failures are wrong tool order, skipped gates, and hallucinated arguments, so each of the six demo scenarios ships with verbatim eval assertions. For scenario S-A, that means tools called in exactly the order get_sprint_status → list_issues → diff_release_notes → create_release_notes, and create_release_notes never executes without a prior approval response. Those blocks were transcribed 1:1 into eval checks in phase 10 and into Playwright expectations in phase 13, without reinterpretation; each browser test asserts the DOM, the UI stream and the Durable Object (ADR-0018).

The harness landed in phase 10 alongside observability: OTLP traces exported to Langfuse without the OpenTelemetry SDK. It runs every scenario against a scripted mock model, deterministic and free, which proves the application honours each trajectory, not that a live model produces it. It ran in CI until 2026-10-21 and runs locally with pnpm ci:local since. Live-model runs work since 2026-10-22: before the demo switched to a live model, the six scenarios ran against Groq and Gemini, and that surfaced six problems the mock had hidden, from both configured models having been retired to a tool description that contradicted its scenario (ADR-0021). The evals now have two tiers: every model must meet each scenario’s meaning, and the mock must also produce its exact sentences. A live run is a sample on free tiers, a quality signal on demand and never a CI gate. The deterministic seed keeps all of this cheap: no LLM-as-judge is needed when the expected tool calls and workspace states are exact.

Cost and latency as design inputs

The product brief sets a hard budget: running cost of at most 5 EUR/month, target 0. That is a design constraint, not a measurement. It shaped the model routing directly. Groq is the primary, now openai/gpt-oss-120b, whose free tier allows 8,000 tokens a minute while one agent turn spends about 2,000; gemini-3.8-flash is a fallback on a different provider; and a global daily token cap acts as a kill-switch that degrades the app into a designed “demo budget reached” state instead of surfacing provider errors mid-conversation. Since 2026-10-22 the deployed demo answers with these live models. The free tiers are its capacity limit: a busy minute falls through to Gemini, and Gemini itself sometimes answers “high demand”.

Latency is handled structurally rather than numerically at this stage: everything streams. The phase-2 milestone was token-by-token streaming from Worker to SPA before any model was involved, so the streaming path is proven plumbing by the time real tokens arrive. On the live site Lighthouse scored 100 in all four categories, on mobile and desktop (roadmap, phase 14); accessibility is gated by axe across five UI states in both themes instead, which found three WCAG AA violations the 100 had hidden (ADR-0017). The phase-10 tracing that would produce time-to-first-token and per-turn cost is built; the deployed Worker has had a live model since 2026-10-22 but sends no traces yet, so neither number is recorded.

What’s built, what’s next

Built today: phases 0–14 of the 16-phase roadmap. The agent runs: the AI SDK v6 ToolLoopAgent chains the eleven tools within a six-step budget, the six read and rich-data tools unattended and every write behind an in-stream approval that is signed, single-use and re-validated on the server, with an Approve all press for a group of cards and an Undo on each applied board write. The five issue tools are served by a public MCP server over Streamable HTTP, which the app consumes through its own MCP client. A long-running deploy tool streams its progress and can be cancelled mid-run; a daily alarm resets the board; every step can be traced to Langfuse over OTLP (no keys are set on the deployed demo); and two spend controls guard the demo, a per-IP rate limit and a daily token cap, next to a stated prompt-injection posture. Since 2026-10-21 three Workers run on Cloudflare: the SPA, the API and the public MCP server. Since 2026-10-22 the demo answers with Groq openai/gpt-oss-120b, falling back to Gemini gemini-3.8-flash.

What is not proven: the six demo scenarios’ assertions run as trajectory evals against a scripted mock model, and the browser tests drive the same mock. That proves the application honours each trajectory. The live-model runs are samples: in the production configuration 7 of 9 cases passed on 2026-10-22, each failure a provider refusing a request, so no pass rate is published. Model spend and latency on the live demo are not recorded. Next: the demo video, the one part of phase 15 left.