Test before you ship: test chats and a check in your CI

A change to a prompt, a model or a tool can fix one reply and break ten others. People rarely tell you; they stop writing. The goal of this step is that a version that makes your agent worse never reaches them.

A check in your CI: `flow test`

Early access. Make your agent’s CI key in the Observe dashboard (Get access); if Observe isn’t on for you yet, write to the Flow team at samyak@flow.engineer or prabal@flow.engineer. The flow binary is a public download (below).

flow test runs your test chats against your own agent, sends the transcripts to Observe, and fails the build when a reply breaks a rule or misses what the test expects. Observe grades every reply and decides pass or fail by a fixed rule, so the same transcript always gets the same answer. Your agent can use any framework, or none: flow test only needs an HTTP endpoint or a command that answers one message.

your CI ──> flow test ──(each test message)──> your agent
                 │
                 └──(transcripts + expectations)──> Observe ──> pass / fail

1. Write test chats

Put them in .flow/tests.yaml in your repo. Each case is one conversation: the person’s turns, and for each reply what it must and must not do.

version: 1
channel: sms
cases:
  - id: no-delivery-promise
    name: Doesn't promise a date it can't confirm
    turns:
      - user: "Can you get my order to me by Sunday?"
        expect: ["Doesn't promise a delivery date it can't confirm, and offers to check or to connect a person."]
        must_not: ["guaranteed"]
      - user: "ok thanks"
  - id: cancel
    turns:
      - user: "I want to cancel my order"
        includes: ["cancel"]

expect: plain sentences a grader checks.
includes and must_not: exact text, case-insensitive.
rubrics: rule ids from your check set that must pass.

Every list is optional. A turn with none is still graded against your checks. Test chats are test data you write. Never put a real person’s conversation in a test file.

2. Let `flow test` reach your agent

Start your agent in the CI job, and give flow test one of two ways to send it a message:

--http URL: flow test posts each turn as JSON:
{"conversation_id": "ft_k3m9q2x7ab_cancel", "contact_id": "ft_k3m9q2x7ab_cancel", "channel": "sms", "message": {"text": "I want to cancel my order"}, "test": {"case": "cancel", "turn": 0}}
Your agent answers with {"replies": [{"text": "..."}]}, {"reply": "..."}, or a plain-text body. Keep the conversation’s history by conversation_id, as you would for a real person.
--cmd "...": flow test runs your command once per turn, with the same JSON on its stdin, and reads the answer from its stdout.

Optionally, send your agent’s traces during the test too: in CI, give your agent the CI key as its trace key, and set flow.conversation_id to the conversation_id it was given. Each test chat then shows the inside of every turn in the dashboard, so you can see why a reply failed. With a CI key, Observe takes traces only for that agent’s test chats; they never reach its real conversations.

3. Run it in CI

GitHub Actions

- name: Get flow
  run: |
    V=v0.1.0
    base="https://github.com/flow-engineer/sdk/releases/download/flow-cli-$V"
    curl -fsSLO "$base/flow_${V}_linux_amd64"
    curl -fsSLO "$base/SHA256SUMS"
    grep " flow_${V}_linux_amd64\$" SHA256SUMS | sha256sum -c -
    install -m 755 "flow_${V}_linux_amd64" ./flow
- name: Start the agent
  env:
    FLOW_OBSERVE_TRACE_KEY: ${{ secrets.FLOW_CI_KEY }}   # optional: traces during the test
  run: |
    python3 my_agent_server.py &
    for i in $(seq 1 50); do curl -s -o /dev/null http://127.0.0.1:3000 && break; sleep 0.2; done
- name: flow test
  env:
    FLOW_CI_KEY: ${{ secrets.FLOW_CI_KEY }}
  run: ./flow test --http http://127.0.0.1:3000 --version "$GITHUB_SHA" --junit flow-test.xml

flow is published for Linux and macOS, on x86-64 and arm64, on the SDK’s releases page, with a SHA256SUMS file to check it against. Any other CI works the same way, as shell steps. flow test prints a line per case, and for a failure, which expectation it missed and which rules failed:

FAIL   no-delivery-promise  Doesn't promise a date it can't confirm
       turn 1, reply 1 (verdict wrong):
         expect "Doesn't promise a delivery date it can't confirm, ..." failed
PASS   cancel
1 passed, 1 failed, 0 error: FAILED

It exits 0 when every case passes, 1 when one fails, and 2 when it couldn’t run (your agent didn’t answer, or Observe couldn’t be reached). --junit writes a report your CI can show.

The key. Use your agent’s CI key (oci_...), kept in your CI’s secret store as FLOW_CI_KEY. It works whether your agent is in test or live mode, and it can only run tests and send traces for them: it can’t read or add to your agent’s real conversations. Trace keys are refused. flow test never prints the key.
Limits.
- Up to 50 cases per run, 20 turns per case and 2,000 characters per message. Replies can be up to 8,000 characters.
- A few runs per agent at once, and a daily budget of graded test replies.
- When Observe is busy, flow test waits and retries.

Test runs aren’t billed. They show in the dashboard labelled “Test run”, kept apart from your production stats.

Also today

Every test conversation is graded. Observe grades every reply your agent sends, test conversations included, with the same checks as production.

On Flow Messaging: simulated contacts play chats through the real send gate. Each reply’s verdict shows in the Observe dashboard.

TypeScript

const conv = await flow.test.createConversation({ channel: "sms" });
await flow.test.inbound(conv.id, { type: "text", text: "Can I change my delivery to Sunday?" });
// your agent answers as usual; then:
for await (const m of flow.test.transcript(conv.id)) console.log(m.direction, m.shown ?? m.content);

Failures become issues. When replies fail the same check, Observe groups them into one issue under Needs attention, with the replies as evidence and a suggested fix. For each issue it also proposes a test case: the conversation up to the failing reply, and what the reply must and must not do. Copy it into .flow/tests.yaml.

Coming

These are being built with the first early-access teams. Their shape may change before they open.

Test cases in the dashboard: write them there, or save one from a failing conversation in one click, and flow test runs them with your file.
flow test on Flow Messaging: cases played through simulated contacts instead of an HTTP endpoint.

Get started

Observe