> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flow.engineer/llms.txt
> Use this file to discover all available pages before exploring further.

# Test your agent

> Check real conversation behavior with simulated people, simulated tools and a judge.

Write a situation and the outcome you expect. Flow's test cloud plays the person, simulates
your tools and judges the conversation, while your agent runs on your machine.

A good reply isn't enough if the agent ordered the wrong thing. Test the product outcome:
what was changed, what was confirmed and what happened when someone corrected themselves.
The scenario below checks that a second message changes the order, rather than creating another one.

> [!NOTE]
>
> Scenario tests use models and a test cloud. They are separate from `flow check`, which checks
> your tool, skill and CLI definitions locally without a model call.

## Before you start

You need a Flow project, a model key for its agent, and `FLOW_FRAMEWORK_CLOUD_KEY` for a test
cloud. [Set up a test cloud](#get-a-test-cloud) if this is your first run.

Use synthetic scenarios. `--synthetic` confirms that your test files contain no real customer data.

## Write a scenario

Create `tests/scenarios/latte-correction.yaml`:

```yaml tests/scenarios/latte-correction.yaml theme={null}
id: latte-correction
channels: [whatsapp]
person:
  persona: Sam, in a hurry; types in short bursts.
  goal: Get a large oat latte for pickup, under the name Sam.
opening:
  - say: "can i get a large latte"
  - say: "oat milk actually"
    after_ms: 1200
world: The point of sale is slow this morning, but orders go through.
expect:
  outcome:
    - Exactly one order is placed, for one large latte with oat milk.
  judge:
    - The agent asks Sam to confirm before placing the order.
```

The opening deliberately sends a correction after the first message. The outcome checks that
only one order is placed and that the agent confirms it before acting.

## Run it

From the project folder:

```sh Terminal theme={null}
npx @flow-engineer/flow test --synthetic --only latte-correction
```

**Check it worked:** the command opens a session and prints a result for each run, including the
scenario, channel, judge's reasons, cost and a link to the conversation. A file error is reported
before a session starts.

Your agent uses its own model and key. The cloud plays the person and tools and evaluates the run.

## Inspect and fix a failure

Use `show last` to read the checks, conversation and simulated tool calls. Then rerun only the
scenarios that did not pass:

```sh Terminal theme={null}
npx @flow-engineer/flow show last
# Make the change in your agent.
npx @flow-engineer/flow test --synthetic --failed
# Run the full suite before finishing.
npx @flow-engineer/flow test --synthetic
```

`show <run-id>` reads a particular run. `--failed` uses `.flow/last-session.json` and includes
failed, flaky, errored and unplayed scenarios. It cannot be combined with `--only`.

## Describe tools and time

`tests/tools/*.yaml` define how the cloud plays each tool, including slow responses and errors.
`expect.calls` and `expect.never` check the recorded tool calls by code.

For date-sensitive cases, set `now` and `timezone` and write dates relative to that clock, such
as `{{now + 1d | date}}`. Tests can also restart an agent with `restart: crash` or `restart: deploy`
to check recovery on the same store.

To test late in a long conversation without playing it, start the run from an earlier one with a
top-level `history`: what each side said, oldest first (1 to 200 items, each up to 4,000
characters, `{{now ...}}` allowed). It is written into the agent's store before the run, as if it
had been said, with no model call and nothing sent; the person model and the judge see it as the
conversation so far. It needs `@flow-engineer/flow` 0.5.0 or newer; an older client's run ends
`client_outdated`.

```yaml theme={null}
history:
  - person: "Book a table for four on Friday"
  - agent: "Done: Friday at 8 pm, for four."
opening:
  - say: "can we make it 9?"
```

The [scenario-writing skill](https://github.com/flow-engineer/flow/blob/main/coding-agents/test-skill.md)
covers the full file format.

## Get a test cloud

**1. Sign in**

```sh Terminal theme={null}
npx @flow-engineer/flow login
```

The command opens the browser and waits for sign-in. A coding agent can continue waiting
for the same code with `login --wait`.

**2. Set up the agent**

From the project folder:

```sh Terminal theme={null}
npx @flow-engineer/flow setup
```

This fills the cloud and trace keys in `.env` without overwriting your configured values.
Keep the file private; setup sets its permissions to 0600.

**3. Run a synthetic scenario**

```sh Terminal theme={null}
npx @flow-engineer/flow test --synthetic --only latte-correction
```

The default cloud is `https://flow.engineer/observe`. Use `--cloud` or
`FLOW_FRAMEWORK_CLOUD` to choose another one.

The cloud prints the workspace's concurrency and budget limits when a session opens. Test
records are retained for 30 days. Your agent's own model calls are billed through its provider.

> **Make a CI key by hand**
>
> Sign in to Flow, create an agent in the workspace, and make a CI key on its Developers page.
> Save it in the project's `.env` as `FLOW_FRAMEWORK_CLOUD_KEY`, never on the command line.
>
> The [coding-agent guide](https://docs.flow.engineer/agents.md) documents setup flags, exit codes and next steps.

## Test an agent you already have

An agent on another framework can use the same scenarios through an HTTP endpoint. With the
agent running locally and `tests/scenarios/*.yaml` in your current folder:

```sh Terminal theme={null}
npx @flow-engineer/flow test \
  --agent-url http://localhost:3000/flow \
  --synthetic
```

The CLI calls the endpoint from your machine, so it does not need to be publicly reachable.
Your real tools run on the agent's side; use judge checks for their effects.

> **HTTP request, reply and timeout contract**
>
> Before the session opens, the command sends one `HEAD` request to the URL (never a message), and stops with `Could not reach your agent at <url>` if nothing answers; any HTTP status counts as an answer.
>
> `flow test` runs on your machine and POSTs each of the person's messages to that URL from there, so your agent never has to be reachable from the internet (`http://localhost` is fine). The request is JSON, with `authorization: Bearer <FLOW_FRAMEWORK_AGENT_TOKEN>` when that variable is set:
>
> ```json theme={null}
> {
>   "conversationId": "whatsapp:ses_8f2a.opening-hours.whatsapp.1",
>   "messageId": "m1",
>   "channel": "whatsapp",
>   "message": { "text": "hi, what time are you open today?", "attachments": [{ "kind": "image", "mime": "image/jpeg", "name": "receipt.jpg", "data": "<base64>" }] }
> }
> ```
>
> Keep the conversation's history by `conversationId`, as you would per phone number. `channel` is the platform the run plays (`whatsapp`, `sms`, `imessage`, `telegram`, `web`), or the name you give with `--channel`. `message.text` is `""` when the person only sent a file; a tapped button or list row also comes as `message.choice` (`{ id, title }`, the button's or row's id), a quoted message as `message.replyTo`. Answer `200` with the replies to send, in order, in the shape a Flow Framework agent sends (`{ "type": "message", "text", "buttons"? (labels, or { id, label }), "list"?, "link"?, "image"?, "replyTo"? }`, `{ "type": "reaction", "emoji", "messageId" }`, `{ "type": "template", ... }`, ...); `[]` sends nothing:
>
> ```json theme={null}
> { "replies": [{ "type": "message", "text": "We're open 8am to 6pm today.", "buttons": ["Order ahead"] }] }
> ```
>
> A minimal server, with no dependencies (the whole file is [`examples/http-agent/server.mjs`](https://github.com/flow-engineer/flow/blob/main/examples/http-agent/server.mjs), with a test scenario beside it; it also ships in the package, in `node_modules/@flow-engineer/flow/examples/http-agent`):
>
> ```js theme={null}
> import { createServer } from "node:http";
>
> createServer(async (req, res) => {
>   let body = "";
>   for await (const chunk of req) body += chunk;
>   const { conversationId, channel, message } = JSON.parse(body);
>   const text = await myAgent.reply(conversationId, message.text); // your framework here
>   res.writeHead(200, { "content-type": "application/json" });
>   res.end(JSON.stringify({ replies: [{ type: "message", text }] }));
> }).listen(3000);
> ```
>
> Each reply is checked against what the run's platform can show, as a Flow Framework agent's is: one that does not fit (buttons on SMS, text over the limit) is not sent, and the run says why. A status other than 200, a body that is not `{ "replies": [...] }` or no answer within 60 seconds (`--agent-timeout <s>`) fails that message's turn, with the reason in the run; the session goes on. A conversation's messages are sent one at a time: the next waits for the last one's answer.
>
> Your agent's own tools run on its side, for real, so `tests/tools/*.yaml` are not used, and outcome, `calls` and `never` checks (which read simulated tool calls) see none: write what must be true as `judge` items. The agent's model cost shows as \$0, since Flow Framework does not see its model calls. Not yet supported: agents that answer later through a callback instead of in the response, and the person's reactions to the agent's messages.

## Any language

The endpoint's contract is written down in [`spec/`](https://github.com/flow-engineer/flow/blob/main/spec/): [`agent-endpoint.openapi.yaml`](https://github.com/flow-engineer/flow/blob/main/spec/agent-endpoint.openapi.yaml) (OpenAPI 3.1: the request, the answer, statuses, timeouts, the token, and what each platform shows) with JSON Schemas for the [request](https://github.com/flow-engineer/flow/blob/main/spec/request.schema.json) and the [answer](https://github.com/flow-engineer/flow/blob/main/spec/response.schema.json). A coding agent can implement it in any language from those files alone.

Check an endpoint against the contract before a test run, with no cloud, no key and no model call on Flow Framework's side:

```bash theme={null}
npx @flow-engineer/flow check-endpoint http://localhost:3000/flow [--token-env NAME] [--json]
```

It sends a fixed set of synthetic messages (a typed message, a double text, a photo, a tapped button quoting the agent, a message that may need no answer) and checks each answer: status 200, `{ "replies": [...] }`, every reply against the schema and what WhatsApp can show, and every id a reply names (`replyTo`, a reaction's `messageId`) one of the person's messages. One line per case; exit 0 when all pass, else 1. The token is the value of `NAME` (default `FLOW_FRAMEWORK_AGENT_TOKEN`).

The café's agent as an HTTP endpoint, with its two tools and one model through any OpenAI-compatible API (the same key variables as `openAICompatible`, `MODEL` for the model; `FAKE_MODEL=1` for a fixed reply and no model call): [`examples/http-agent-python`](https://github.com/flow-engineer/flow/blob/main/examples/http-agent-python) (standard library only) and [`examples/http-agent-go`](https://github.com/flow-engineer/flow/blob/main/examples/http-agent-go) (standard library only).

## Command output and reliability

> **Exit codes, flaky runs, retries and partial sessions**
>
> | Exit | Meaning |
> | - | - |
> | 0 | Every run passed, or was flaky (without `--strict`). |
> | 1 | A run failed (with `--strict`, or was flaky), or a message took a second send path (below). Or nothing was sent: a wrong flag, a test file that does not parse, no `FLOW_FRAMEWORK_CLOUD_KEY`; the message is printed on its own. |
> | 2 | The connection, the tunnel or the stream failed (before or after the session opened), the cloud answered 5xx or no JSON, the cloud ended the session for its own failure, or runs errored (the cloud could not play or judge them) or were not played (the session ended early), and none failed. |
> | 3 | The cloud refused: the open (a bad or wrong key, 401/403; an invalid request, 400; busy, 429 `rate_limited`; the spend cap, 429 `spend_cap`; 503 `capacity`), or it ended the session for the spend cap or capacity. |
> | 130 | Interrupted (Ctrl-C). |
>
> A busy cloud's refused open (429 other than the spend cap, or 503 `capacity`) with a Retry-After of 60 s or less is sent once more after it, with one line saying so; the spend cap never is. On a cloud that resumes, a dropped stream is opened again for up to 2 minutes, picking up after the last command. On a cloud that can re-open, a session the cloud lost is opened once more for only the runs with no result (the log names them, since they are played and paid for again), and the report covers both sessions with both dashboard links. Whenever a session ends before every run played (the tunnel, the stream or the connection failed, the cloud stopped it, or its summary left runs out), the runs that had already finished are printed, then `N of M runs finished before the session ended early (K not played): <reason>`, with the cloud's error as the reason when it sent one, then the dashboard link; it exits 2 (3 when the cloud stopped it for the spend cap or capacity). Ctrl-C prints that same partial report, says the cloud keeps playing until it is stopped from the dashboard, and exits 130.
>
> **Flaky runs.** The cloud plays a run that failed once more. When the second play passes the run is flaky: its line reads `FLAKY`, with `played again: passed <watch>` under it, and the summary lists it in its own section (`1 flaky (failed once, passed when played again):` with both watch links), apart from the failures. A flaky run does not fail the session (exit 0 when it is all that did not pass); `--strict` counts it as failed (exit 1). When the second play fails too, the run failed. The cloud also replays the person's, the tools' and the judge's stored answers for a scenario it has played unchanged, so runs vary less and `--runs 3` with a "2 of 3" bar should rarely be needed; `--fresh` plays every simulator anew. An older cloud does neither, and the output is as before.
>
> **Runs, pass rate and credit.** `--runs 3` plays each scenario three times on each of its channels. A run comes to one of four outcomes: `passed`, `failed`, `credit` or `infra`. Credit (the workspace's daily test budget, `spend_cap`, or the agent's model refused for credit, `agent_credit`, such as Flow's trial credit used up) and infra (the cloud could not play or judge it, such as `judge_failed`) say nothing about the agent: their lines read `CREDIT` and `ERROR`, the summary counts them apart from the failures (`4 of 6 runs passed. 1 failed. 1 stopped for credit (not the agent's fault, not counted as failed).`), and a session with only those and passes exits 2. When the cloud stops the session for the agent's credit, it ends with `agent_credit: ...` and exits 2. With `--runs` above 1 (or a run the cloud played again), the summary gives each scenario's pass rate on each channel over the runs that passed or failed, flaky when some passed and some failed (`  latte-correction (whatsapp): 2/3 passed, flaky`), naming any credit or infra runs as not counted. A failed run still fails the session (exit 1).
>
> **One send path.** Every message an agent sends in a test must go through its send (its channels), which announces it with a `message.out` event. On a Flow Framework agent the CLI counts, per conversation, the messages that reached the cloud and the ones announced; when they differ (an app's own wrapper or code sent one the agent never did, or the agent says it sent one that never arrived), it names the conversation and the session fails (exit 1).
>
> Each run's line carries its id (`FAIL  latte-correction #1 (whatsapp) trn_…: 6 turns, …`). `--dotenv <path>` reads the environment from that file instead of the folder's `.env` (variables already set still win; no value is printed), and `--verbose` also prints slow event uploads.

> **JSON output and saved session records**
>
> `--json` prints one JSON document on stdout when the command ends, on every exit once its flags are read; every other line goes to stderr:
>
> ```json theme={null}
> {
>   "version": 1, "outcome": "failed", "exitCode": 1,
>   "sessionId": "tse_…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…",
>   "error": null,
>   "counts": { "runs": 2, "passed": 1, "failed": 1, "flaky": 0, "errored": 0, "notPlayed": 0, "credit": 0, "infra": 0 },
>   "scenarios": [{ "scenario": "latte-correction", "path": "scenarios/latte-correction.yaml", "channel": "whatsapp", "runs": 2, "passedRuns": 1, "credit": 0, "infra": 0, "label": "flaky" }],
>   "sendPath": [],
>   "strict": false,
>   "costUsd": { "agent": 0.0123, "test": 0.0456 },
>   "runs": [
>     { "runId": "trn_…", "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "verdict": "failed", "outcome": "failed",
>       "error": null, "ended": "goal_met", "turns": 6, "watch": "https://…",
>       "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }],
>       "invalid": null, "rerun": { "runId": "trn_…", "passed": false, "watch": "https://…" }, "label": "failing",
>       "replayed": { "person": 3, "tools": 1, "judge": 0 } }
>   ],
>   "invalid": [],
>   "sessionInvalid": null
> }
> ```
>
> `outcome` is `passed` (exit 0), `failed` or `invalid` (1), `errored` or `transport` (2), `refused` (3) or `interrupted` (130). A run's `verdict` is `errored` whenever the cloud names an `error` for it (such as `judge_failed`, `simulator_failed` or `spend_cap`), whatever it `ended` as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads `ERROR` instead of `PASS` or `FAIL`, and the run's `error` is that code (else `null`). A run's `verdict` is `flaky` when it failed and passed when the cloud played it again; `rerun` is that second play (`{ "runId", "passed", "watch" }`, else `null`), `label` is `flaky` or `failing` (it failed again) on a run played again (else `null`), and `replayed` counts the stored simulator answers the run replayed (else `null`). The second play is never a run of its own in `runs` or `counts`. `counts.failed` leaves flaky runs out unless `--strict` (`strict: true`); `counts.flaky` counts them. A run's `outcome` is `passed`, `failed`, `credit` or `infra` (`null` for one not played; a flaky run's is `failed`, its first play), and `counts.credit` and `counts.infra` count those runs, which `counts.errored` also holds. `scenarios` is each scenario (by its `id`, with its file as `path`) on each channel: `runs` the runs that passed or failed, `passedRuns` how many passed, `credit` and `infra` the runs not counted, and `label` `flaky` when some but not all passed, or a run failed and passed when played again. `sendPath` lists each conversation whose messages did not all go through the agent's send, as `{ "conversationId", "delivered", "announced" }`. `error` is `{ "message", "status", "type", "retryAfterSeconds" }` when the session did not finish (`type` is the cloud's code, such as `spend_cap`; the others are `null` when it did not say).
>
> A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are `errored` with `error` `scenario_invalid` and `invalid` `{ "path", "reason" }` (the reason starts `unsupported:` when the cloud does not know a key), its line reads `INVALID` with the file and what is wrong, and the top-level `invalid` lists each such scenario once as `{ "scenario", "path", "reason" }`. When the cloud ends the whole session `scenario_invalid` (a tools or checks file it cannot read, or an older cloud), `sessionInvalid` is `{ "reason", "olderCloud", "sent" }`: what the cloud said after the code (else `null`), whether that suggests a cloud older than this client, and the scenarios sent as `{ "scenario", "path" }`; the same lines are printed.
>
> `--only latte-correction,order-fails` runs only those scenarios (file names without `.yaml`); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes `.flow/last-session.json` in the agent's folder and prints its path; `flow new` git-ignores `.flow/`. It holds what each run ended as and the judge's reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by `--synthetic`):
>
> ```json theme={null}
> {
>   "sessionId": "…", "agent": "agt_…", "dashboard": "https://…", "suiteHash": "…", "endedAt": "2026-10-16T09:12:03.000Z",
>   "ended": "finished",
>   "only": null,
>   "runs": [
>     { "scenario": "latte-correction", "channel": "whatsapp", "run": 1, "runId": "trn_…", "verdict": "failed",
>       "failedChecks": [{ "kind": "judge", "text": "The agent asks Sam to confirm before placing the order.", "reason": "…" }] }
>   ]
> }
> ```
>
> A run of a scenario the cloud could not play also has `invalid` (`{ "path", "reason" }`), and a session the cloud ended `scenario_invalid` has `sessionInvalid`, as in `--json`.
>
> `ended` is `finished`, `stopped` (the session broke off or the cloud stopped it) or `interrupted`. `agent` is the cloud's id for the agent (a hosted cloud names it), which `flow show` reads. `only` is the scenarios the session was limited to, `null` for the whole suite. `verdict` is `passed`, `failed`, `flaky` (failed, then passed when the cloud played it again), `errored` (the cloud could not play or judge the run) or `not_played` (the session ended first; its `runId` is `null`), and `outcome` is as in `--json`.

> **[Try a case by hand](https://docs.flow.engineer/playground.md)**
>
> Inspect the agent and tools before writing a scenario.

> **[Deploy a checked version](https://docs.flow.engineer/hosting.md)**
>
> Exercise staging, inspect logs and promote.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.