How a test run works

Sessions

start_session opens a run. It must come first, before any account is created: an account made outside a session has nothing to clean it up. The agent gives it:

• a project name (2–30 lowercase letters, digits and hyphens, like acme or acme-staging). It’s created the first time it’s used, and it appears in every test address, so use your app’s real name;
• a label for the run (“signup + reset, new onboarding”), shown on the report;
• the flows it plans to test, so the report can flag any it said it would test but didn’t.

Before that, the agent calls whats_next. It needs no memory of earlier runs: it says whether a run is still open, whether cleanup is connected, which emails are left to test and how each can be triggered (by using your app, or needing a test hook, a test checkout or an outside event), and what to do next.

start_session won’t open a second run while one on the project is still open: the agent finishes that one first, unless you ask for two at once. And if no cleanup is connected, it asks you first: the test accounts the run creates would stay in your app for you to delete by hand. It only runs on your yes.

A session lasts 24 hours by default (the agent can set anything from a minute to 7 days). When it runs out, its addresses stop receiving mail and Flowproof closes it and cleans up automatically. Its report then says it was never closed, and covers only what arrived before it ran out.

Test addresses

new_address makes a fresh address for the run, laid out as project.session.name@flowproof.dev:

acme.s7k2ab.alice@flowproof.dev

It can come with a made-up person: a name, a strong password that meets any sensible password rule, and a phone number from a range reserved for fiction. Ask for an edge-case person and the name will carry an apostrophe, an accent, a right-to-left script or an overlong surname: the users that break real signup forms.

Steps and latency

Just before each action that should send an email (submitting the signup form, clicking “forgot password”) the agent calls mark_step with a name like “submitted signup form”. The mark is a timestamp. The gap between it and the email’s arrival is the email’s latency, and it’s graded:

• up to 30 seconds is fine;
• over 30 seconds is flagged as slow;
• a verification, password reset, magic link or login code that takes over 60 seconds is critical, because people give up waiting.

In a chat app like claude.ai or ChatGPT, the AI often can’t click the button or type a password, so it asks you to. It then marks the step with by: "person", and that email shows as “person-driven step, not timed”: the gap would measure you reading and typing, not your app, so it isn’t graded.

Waiting for email

wait_for_email watches a test address. It answers within about 25 seconds, so no connection ever times out, with one of three results:

Some emails should not be sent: a failed signup mustn’t send a welcome email. assert_no_email waits out a window and passes if nothing arrives (silent) or fails if something does.

Links and codes

extract pulls the verify link, magic link, reset link, unsubscribe link or one-time code out of an email. Then the agent uses it itself, in your app.

Recording what the app did

Flowproof sees every email and nothing of your app. Only the agent knows whether the link actually verified the account, so it tells Flowproof with record_outcome. Without a record, the report marks that behaviour not tested, never “passed”.

“Passed” always means your app did the right thing, which for the unhappy paths means it refused:

A reset link that works twice is a real security bug, and this is the only way the report can catch it. Each record is read back in plain words (“the app ACCEPTED a reused link”) so a mix-up is easy to spot.

Ending the run

end_session does two things: it runs cleanup on every system you’ve connected and armed, then builds the final report and returns its summary. After that, the run accepts no more evidence, so the agent records every outcome first.

Want to see what cleanup would delete before it does? Ask for a dry run: nothing is deleted, and the run stays open so it can clean up for real afterwards.

Before you end the Flowproof session, call end_session with dry_run: true and show me exactly what cleanup would delete.

The agent should end the run even when testing went badly. A run left open keeps its test accounts in your app until the session runs out (24 hours by default); then Flowproof closes and cleans it up anyway.

The report

The agent can ask for a provisional report at any time with get_report, which is a good way to see what’s still untested. The final one is built when the run ends.

The grade

Runs are graded A+ to F. A few rules keep the letter honest:

• Any critical problem (a reset link that works twice, a verification email that takes minutes, a staging URL in a live email) holds the grade at C+ or below.
• If a security behaviour was never tested (reusing a link, an expired link, a wrong code), the best possible grade is B+. An untested flow can’t earn an A. A behaviour an earlier run proved still counts: see carried forward.
• Steps marked but no email at all is an F: the app failed to send.
• If nothing was tested, or mail only reached addresses the run never made (usually a typo), there is no grade at all.

Carried forward from earlier runs

Some tests don’t fit in one run. Checking that a reset link expires means waiting out the app’s timer, which can be an hour. So you don’t have to prove everything at once: what a finished run proves is remembered for its project, and a later run that doesn’t repeat a test can still count it.

A behaviour is carried forward only when all of these are true:

• An earlier run of the same project passed it. A failure is never carried forward.
• It was the same email: same sender, same subject, same link route. If you edited the template or moved the link since, the report says “re-test: this email changed since it was proven” and it doesn’t count.
• It was in the last 30 days, and the latest result on that email was a pass. A newer failure wins over an older pass.
• This run didn’t test it itself. If it did, this run’s own result is the one that counts.

The report always says what it carried and from where, for example “Reset link expiry: proven 28 Sep in run gm4v7n”, in the grade’s reasons, the summary your agent reads, and on the run page. Carried behaviours are never shown as passes of this run. Each project’s page lists what its runs have proved and until when each one carries.

Since your last run

When you run a flow again, the report compares it with the last finished run of the same project that tested the same flows: the grade then and now, what was fixed, what is newly broken, emails that arrived last time and not this time (or the other way round), and emails whose subject, sender or link changed. An email that got much slower is flagged too, once it is at least twice as slow as before and over 30 seconds. Your agent gets it as one line under the grade, for example “Since run abc234 (2 days ago): B → B+, 1 fixed, 1 change: the password reset email’s subject changed”, and the run page shows it in full.

Only your agent’s runs are compared, only within one project, and only with a run whose report was final. Flow Watch runs, pull request checks and the free grader are never part of it. When there is no earlier run to compare with, the report says nothing about it.

What is checked

• Speed: latency per email, measured from the step before it.
• Authentication: SPF, DKIM and DMARC, and whether the sender addresses line up.
• Links: broken links, plain http links, and staging or localhost hosts in live emails.
• Content: leaked merge tags (“Hi ,” or {{first_name}}), missing plain-text versions, oversized HTML, images without alt text.
• Provider defaults: the stock Supabase, Firebase, Cognito or WordPress email nobody replaced, and mail sent through the provider’s built-in, rate-limited mailer instead of your own. A default that never names your product counts as a serious problem: it looks like phishing.
• Duplicates: the same email sent twice.
• Missing emails: emails a flow should send but didn’t, like the “your password was changed” notice.
• Token behaviour: from what the agent recorded (above).
• Spam score: how a common spam filter rates each kind of email (below).

Fixes, and what wasn’t tested

Problems become fixes, most serious first, each with how to fix it. Checks that couldn’t run are listed as not tested, with the reason. They’re never counted as passes.

Spam score

When the run ends, the first email of each kind (up to 5) is scored by SpamAssassin, a spam filter many mail servers run, through Postmark’s free spam-check service. Links, codes and your test addresses are blanked before the copy is sent. Lower is better, and each email’s score is shown as, for example, “Spam score 1.2 / 5”:

• Under 2.5: good. The email is unlikely to be flagged.
• 2.5 to 4.9: some filters may flag it. A small problem, worth one point of the grade.
• 5 or more: likely to land in spam. SpamAssassin’s own default treats it as spam.

The fix names the three things that added the most, in plain words: for example, “the email has only an HTML version, send a plain-text version alongside it”. The score is one filter’s opinion; Gmail and Outlook use their own, and they lean heavily on SPF, DKIM and DMARC, which are checked separately. The spam check shares the project’s AI review switch, and when it doesn’t run (an open run, the switch off, or the service not answering in time) it is listed as not tested and never changes the grade.

Wording review

An AI reads each email the way a new user would and suggests clearer subject lines and buttons. The suggestions appear next to each email and never change the grade. Plans include a number of wording reviews (the Free plan: your first 3 runs); when they run out, the report is still built in full, without this part.

The timeline

The run page’s “What happened, step by step” lists, in order, each step the agent marked, each email that arrived (with its delay from the step before it), and each thing the agent recorded, marked Passed or Problem. Times count from the first thing that happened in the run.