Scenario tests use models and a test cloud. They are separate from
flow check, which checks
your tool, skill and CLI definitions locally without a model call.Before you start
You need a Flow project, a model key for its agent, andFLOW_FRAMEWORK_CLOUD_KEY for a test
cloud. Set up a test cloud if this is your first run.
Use synthetic scenarios. --synthetic confirms that your test files contain no real customer data.
Write a scenario
Createtests/scenarios/latte-correction.yaml:
tests/scenarios/latte-correction.yaml
Run it
From the project folder:Terminal
Inspect and fix a failure
Useshow last to read the checks, conversation and simulated tool calls. Then rerun only the
scenarios that did not pass:
Terminal
show <run-id> reads a particular run. --failed uses .flow/last-session.json and includes
failed, flaky, errored and unplayed scenarios. It cannot be combined with --only.
Describe tools and time
tests/tools/*.yaml define how the cloud plays each tool, including slow responses and errors.
expect.calls and expect.never check the recorded tool calls by code.
For date-sensitive cases, set now and timezone and write dates relative to that clock, such
as {{now + 1d | date}}. Tests can also restart an agent with restart: crash or restart: deploy
to check recovery on the same store.
To test late in a long conversation without playing it, start the run from an earlier one with a
top-level history: what each side said, oldest first (1 to 200 items, each up to 4,000
characters, {{now ...}} allowed). It is written into the agent’s store before the run, as if it
had been said, with no model call and nothing sent; the person model and the judge see it as the
conversation so far. It needs @flow-engineer/flow 0.5.0 or newer; an older client’s run ends
client_outdated.
Get a test cloud
1
Sign in
Terminal
login --wait.2
Set up the agent
From the project folder:This fills the cloud and trace keys in
Terminal
.env without overwriting your configured values.
Keep the file private; setup sets its permissions to 0600.3
Run a synthetic scenario
Terminal
https://flow.engineer/observe. Use --cloud or
FLOW_FRAMEWORK_CLOUD to choose another one.Make a CI key by hand
Make a CI key by hand
Sign in to Flow, create an agent in the workspace, and make a CI key on its Developers page.
Save it in the project’s
.env as FLOW_FRAMEWORK_CLOUD_KEY, never on the command line.The coding-agent guide documents setup flags, exit codes and next steps.Test an agent you already have
An agent on another framework can use the same scenarios through an HTTP endpoint. With the agent running locally andtests/scenarios/*.yaml in your current folder:
Terminal
HTTP request, reply and timeout contract
HTTP request, reply and timeout contract
Before the session opens, the command sends one Keep the conversation’s history by A minimal server, with no dependencies (the whole file is Each reply is checked against what the run’s platform can show, as a Flow Framework agent’s is: one that does not fit (buttons on SMS, text over the limit) is not sent, and the run says why. A status other than 200, a body that is not
HEAD request to the URL (never a message), and stops with Could not reach your agent at <url> if nothing answers; any HTTP status counts as an answer.flow test runs on your machine and POSTs each of the person’s messages to that URL from there, so your agent never has to be reachable from the internet (http://localhost is fine). The request is JSON, with authorization: Bearer <FLOW_FRAMEWORK_AGENT_TOKEN> when that variable is set:conversationId, as you would per phone number. channel is the platform the run plays (whatsapp, sms, imessage, telegram, web), or the name you give with --channel. message.text is "" when the person only sent a file; a tapped button or list row also comes as message.choice ({ id, title }, the button’s or row’s id), a quoted message as message.replyTo. Answer 200 with the replies to send, in order, in the shape a Flow Framework agent sends ({ "type": "message", "text", "buttons"? (labels, or { id, label }), "list"?, "link"?, "image"?, "replyTo"? }, { "type": "reaction", "emoji", "messageId" }, { "type": "template", ... }, …); [] sends nothing:examples/http-agent/server.mjs, with a test scenario beside it; it also ships in the package, in node_modules/@flow-engineer/flow/examples/http-agent):{ "replies": [...] } or no answer within 60 seconds (--agent-timeout <s>) fails that message’s turn, with the reason in the run; the session goes on. A conversation’s messages are sent one at a time: the next waits for the last one’s answer.Your agent’s own tools run on its side, for real, so tests/tools/*.yaml are not used, and outcome, calls and never checks (which read simulated tool calls) see none: write what must be true as judge items. The agent’s model cost shows as $0, since Flow Framework does not see its model calls. Not yet supported: agents that answer later through a callback instead of in the response, and the person’s reactions to the agent’s messages.Any language
The endpoint’s contract is written down inspec/: agent-endpoint.openapi.yaml (OpenAPI 3.1: the request, the answer, statuses, timeouts, the token, and what each platform shows) with JSON Schemas for the request and the answer. A coding agent can implement it in any language from those files alone.
Check an endpoint against the contract before a test run, with no cloud, no key and no model call on Flow Framework’s side:
{ "replies": [...] }, every reply against the schema and what WhatsApp can show, and every id a reply names (replyTo, a reaction’s messageId) one of the person’s messages. One line per case; exit 0 when all pass, else 1. The token is the value of NAME (default FLOW_FRAMEWORK_AGENT_TOKEN).
The café’s agent as an HTTP endpoint, with its two tools and one model through any OpenAI-compatible API (the same key variables as openAICompatible, MODEL for the model; FAKE_MODEL=1 for a fixed reply and no model call): examples/http-agent-python (standard library only) and examples/http-agent-go (standard library only).
Command output and reliability
Exit codes, flaky runs, retries and partial sessions
Exit codes, flaky runs, retries and partial sessions
A busy cloud’s refused open (429 other than the spend cap, or 503
capacity) with a Retry-After of 60 s or less is sent once more after it, with one line saying so; the spend cap never is. On a cloud that resumes, a dropped stream is opened again for up to 2 minutes, picking up after the last command. On a cloud that can re-open, a session the cloud lost is opened once more for only the runs with no result (the log names them, since they are played and paid for again), and the report covers both sessions with both dashboard links. Whenever a session ends before every run played (the tunnel, the stream or the connection failed, the cloud stopped it, or its summary left runs out), the runs that had already finished are printed, then N of M runs finished before the session ended early (K not played): <reason>, with the cloud’s error as the reason when it sent one, then the dashboard link; it exits 2 (3 when the cloud stopped it for the spend cap or capacity). Ctrl-C prints that same partial report, says the cloud keeps playing until it is stopped from the dashboard, and exits 130.Flaky runs. The cloud plays a run that failed once more. When the second play passes the run is flaky: its line reads FLAKY, with played again: passed <watch> under it, and the summary lists it in its own section (1 flaky (failed once, passed when played again): with both watch links), apart from the failures. A flaky run does not fail the session (exit 0 when it is all that did not pass); --strict counts it as failed (exit 1). When the second play fails too, the run failed. The cloud also replays the person’s, the tools’ and the judge’s stored answers for a scenario it has played unchanged, so runs vary less and --runs 3 with a “2 of 3” bar should rarely be needed; --fresh plays every simulator anew. An older cloud does neither, and the output is as before.Runs, pass rate and credit. --runs 3 plays each scenario three times on each of its channels. A run comes to one of four outcomes: passed, failed, credit or infra. Credit (the workspace’s daily test budget, spend_cap, or the agent’s model refused for credit, agent_credit, such as Flow’s trial credit used up) and infra (the cloud could not play or judge it, such as judge_failed) say nothing about the agent: their lines read CREDIT and ERROR, the summary counts them apart from the failures (4 of 6 runs passed. 1 failed. 1 stopped for credit (not the agent's fault, not counted as failed).), and a session with only those and passes exits 2. When the cloud stops the session for the agent’s credit, it ends with agent_credit: ... and exits 2. With --runs above 1 (or a run the cloud played again), the summary gives each scenario’s pass rate on each channel over the runs that passed or failed, flaky when some passed and some failed ( latte-correction (whatsapp): 2/3 passed, flaky), naming any credit or infra runs as not counted. A failed run still fails the session (exit 1).One send path. Every message an agent sends in a test must go through its send (its channels), which announces it with a message.out event. On a Flow Framework agent the CLI counts, per conversation, the messages that reached the cloud and the ones announced; when they differ (an app’s own wrapper or code sent one the agent never did, or the agent says it sent one that never arrived), it names the conversation and the session fails (exit 1).Each run’s line carries its id (FAIL latte-correction #1 (whatsapp) trn_…: 6 turns, …). --dotenv <path> reads the environment from that file instead of the folder’s .env (variables already set still win; no value is printed), and --verbose also prints slow event uploads.JSON output and saved session records
JSON output and saved session records
--json prints one JSON document on stdout when the command ends, on every exit once its flags are read; every other line goes to stderr:outcome is passed (exit 0), failed or invalid (1), errored or transport (2), refused (3) or interrupted (130). A run’s verdict is errored whenever the cloud names an error for it (such as judge_failed, simulator_failed or spend_cap), whatever it ended as: the cloud could not play or judge it, so it says nothing about the agent. Its line reads ERROR instead of PASS or FAIL, and the run’s error is that code (else null). A run’s verdict is flaky when it failed and passed when the cloud played it again; rerun is that second play ({ "runId", "passed", "watch" }, else null), label is flaky or failing (it failed again) on a run played again (else null), and replayed counts the stored simulator answers the run replayed (else null). The second play is never a run of its own in runs or counts. counts.failed leaves flaky runs out unless --strict (strict: true); counts.flaky counts them. A run’s outcome is passed, failed, credit or infra (null for one not played; a flaky run’s is failed, its first play), and counts.credit and counts.infra count those runs, which counts.errored also holds. scenarios is each scenario (by its id, with its file as path) on each channel: runs the runs that passed or failed, passedRuns how many passed, credit and infra the runs not counted, and label flaky when some but not all passed, or a run failed and passed when played again. sendPath lists each conversation whose messages did not all go through the agent’s send, as { "conversationId", "delivered", "announced" }. error is { "message", "status", "type", "retryAfterSeconds" } when the session did not finish (type is the cloud’s code, such as spend_cap; the others are null when it did not say).A scenario the cloud cannot play (a key it does not know, a value it cannot read) does not stop the others: its runs are errored with error scenario_invalid and invalid { "path", "reason" } (the reason starts unsupported: when the cloud does not know a key), its line reads INVALID with the file and what is wrong, and the top-level invalid lists each such scenario once as { "scenario", "path", "reason" }. When the cloud ends the whole session scenario_invalid (a tools or checks file it cannot read, or an older cloud), sessionInvalid is { "reason", "olderCloud", "sent" }: what the cloud said after the code (else null), whether that suggests a cloud older than this client, and the scenarios sent as { "scenario", "path" }; the same lines are printed.--only latte-correction,order-fails runs only those scenarios (file names without .yaml); every tool file still goes. Once the session has opened, every exit (0, 1, 2, 3 or 130) writes .flow/last-session.json in the agent’s folder and prints its path; flow new git-ignores .flow/. It holds what each run ended as and the judge’s reasons, never the messages (a reason can quote one, which is why the file stays on your machine; the suite is synthetic by --synthetic):invalid ({ "path", "reason" }), and a session the cloud ended scenario_invalid has sessionInvalid, as in --json.ended is finished, stopped (the session broke off or the cloud stopped it) or interrupted. agent is the cloud’s id for the agent (a hosted cloud names it), which flow show reads. only is the scenarios the session was limited to, null for the whole suite. verdict is passed, failed, flaky (failed, then passed when the cloud played it again), errored (the cloud could not play or judge the run) or not_played (the session ended first; its runId is null), and outcome is as in --json.Try a case by hand
Inspect the agent and tools before writing a scenario.
Deploy a checked version
Exercise staging, inspect logs and promote.