Designing agent_graph, an AI pipeline that QA-tests its own pull requests

Hand a coding agent a ticket and you usually get a diff back. Someone still has to check it against the ticket and run it on a device, and the reviewer has to take their word for it.

I built agent_graph to hand that part to agents as well. It’s a Dart CLI that takes a ticket, passes it through seventeen specialised agents, and finishes with a validator that drives the app and attaches evidence to the result. This post covers the design and the reasoning behind it. I’m not reporting production numbers here.

The scheduler is code

In a lot of agent frameworks, the model decides what happens next. Control flow becomes whatever the sampler produced on that run. You can’t unit-test that, and a run that dies halfway has no reliable way to resume.

In agent_graph the graph is a YAML file. Nodes declare their dependencies, and the scheduler that reads them is ordinary Dart with ordinary unit tests covering readiness, concurrency, conditions and the correction loop. A model’s job is to do one node’s work well and hand back a result that passes schema validation. None of them gets a vote on whether the security reviewer runs.

Agents never chat with each other. Each node receives an AgentInput and returns an AgentOutput, both typed and validated, and raw conversation never crosses from one agent to the next. One agent can’t talk another into anything, because all they exchange is records.

Run state is written to disk as JSON after every transition. When a long run dies near the end, resume picks it up and skips every node that already succeeded.

Everything an agent returns is treated as untrusted. It gets schema-checked and scrubbed for secrets, and nodes that write files can only touch the paths they declare:

implementer:
  writesFiles: true
  allowedPaths: [lib/, test/]
  concurrencyGroup: writers

Even the severity parser assumes the model might return garbage:

/// Parses a severity from its lowercase name, defaulting to [Severity.info]
/// for unknown values so that untrusted agent output can never crash the
/// engine.
static Severity parse(Object? value) { /* ... */ }

A model that answers "severity": "CATASTROPHIC" gets info, and the orchestrator keeps going.

The graph

Seventeen nodes in five phases:

task-intake             requirements and DoD become acceptance criteria
    ↓
context-retriever
    ↓
planner ──┬── architecture-researcher ─┐
          ├── codebase-researcher      │  parallel
          ├── product-domain           │
          └── test-strategist ─────────┘
              ↓
          plan-merger
              ↓
          implementer
              ↓
          test-agent
              ↓
    ┌── code-reviewer ────────┐
    ├── architecture-reviewer │         parallel
    └── security-reviewer ────┘
              ↓
        findings-aggregator
              ↓
        fixer ⟲  bounded correction loop
              ↓
        final-validator                 QA
              ↓
        knowledge-maintainer

Research fans out, because four agents reading the codebase from different angles don’t need each other’s output. Review fans out for the same reason. Anything that writes to the tree goes through a writers concurrency group one at a time, since two agents editing the same files at once will happily produce a merge conflict with themselves.

Requirements first

The first node, task-intake, normalises the request and derives acceptance criteria before anything else happens.

Tickets are prose, and plenty of them are vague. When the implementer and the validator each read the ticket on their own, you get two interpretations, and the validator ends up approving whatever got built. So the ticket’s requirements and its definition of done become a structured list of acceptance criteria once, before any code exists. The test strategist plans against that list. The final validator checks the finished work against the same one.

Fix loops that notice they’re stuck

Review-and-fix loops that run until the reviewer is happy can go on forever. Mine compares findings between cycles:

enum FindingDelta { added, resolved, rejected, unchanged }

class CycleAssessment {
  final bool shouldContinue;
  final String reason;
  final int added, resolved, unchanged;
}

Every cycle records which findings were added, resolved or left alone. The loop stops at a cycle cap, and it also stops when it isn’t getting anywhere. Two findings fixed and two new ones introduced means it isn’t converging, so it stops and escalates to a person. Only blocker and major findings count toward that decision. A minor finding can’t hold the loop open.

The AI as QA

The final validator’s objective is to verify the diff, the acceptance criteria, the tests and the critical user flows.

It’s gated:

finalValidation:
  agent: final-validator
  dependsOn: [aggregate]
  condition: findings.hasNoMajorOrBlocker
  approval: manual_approval_only_on_risk

While any blocker or major finding is still open it doesn’t run at all, since driving a simulator against code the reviewers already rejected would waste the time. The approval setting pulls a person in only when the change carries risk.

A QA step needs a verdict someone can act on, so the validator picks from a fixed set:

status:   passed | failed | blocked | passed_with_known_risks
decision: go | no-go

The rules come from the skill file that drives it. The validator can only say go once every blocker and major finding is resolved and every critical user-facing flow has passed end to end, and a critical flow that fails or can’t be validated at all means no-go. passed_with_known_risks is reserved for gaps that are documented and non-critical, the kind a reviewer can read about and accept on purpose. Every decision cites its evidence.

And a skipped critical flow never counts as a pass. That rule carries the most weight, because skipping is the easiest way for automated QA to lie. Maybe the simulator was busy, or a selector moved after a redesign. The flow never ran. A system that wants green checks reports green anyway, so here a skip is its own outcome.

blocked works the same way. A system that only knows pass and fail has to squeeze “couldn’t check” into one of the two, and it always ends up as pass. In agent_graph, blocked is a status with a required payload. It has to name the blocking reason, the commands it attempted, the environment, the affected flows, whatever alternative evidence exists, and the exact manual steps a person still needs to run. That last item makes a blocked verdict something a teammate can pick up and finish.

For user-facing mobile changes the validator runs the real app on a simulator and collects screenshots, screen recordings, the test runner’s debug output, app logs, device details, the step that failed, how long the run took and how many retries it needed. A reviewer who wasn’t watching gets a recording of the flow working, on a named device, at a specific commit, and can check the claim for themselves.

Failures get more useful too. A failed unit test tells you a function is wrong. A recording of checkout stalling on step three shows what a customer would have run into, and a product manager can watch it without an engineer translating.

The skill is also honest about its gaps. The Flutter app I built it against has no Maestro setup yet, since it tests flows with integration_test and SimDeck, and the skill says so. When Maestro is missing it returns blocked, names the missing tool, lists the commands it tried and the flows it skipped, and recommends a way forward: add Maestro, or run the equivalent integration_test flow on the iOS Simulator through SimDeck and validate again.

Running it

A run starts from the CLI:

dart run bin/agent_graph.dart run workflows/software-development.yaml \
  --task APP-284 --description "Fix empty-cart crash" \
  --repo /path/to/app --change-flag security

When the validator says go, one more node runs. knowledge-maintainer writes what the run learned into .agent-memory/ and docs/, and the next ticket starts from there.