Customer outcome as acceptance. Natural-language scenarios that prove the product still works for the people who use it — even when code is shipped by machines.
AI made writing code cheap. What did not get cheaper is proof that a real customer can still succeed.
Unit tests stay green while signup confuses a first-time user. End-to-end scripts pass while checkout feels hostile. A one-shot crawl finds broken links and still says nothing about whether a skeptic can recover a password or a novice can reach first value.
In the vibecoding era, we advocate VDD (Vibetest-driven-development) — letting agents drive development while bringing clients into the process.
A customer outcome in plain language becomes acceptance criteria, agent task, and regression contract at once.
Encode customer outcomes as natural-language scenarios; let an independent agent act as the customer on a real product surface; ship only when the acceptance result is acceptable — and keep those scenarios as living contracts, not one-off demos.
For most of software's history the scarce resource was the ability to change the system. Verification scaled roughly with how much humans could implement. That balance is gone.
Coding agents multiply surface area: more files touched, more edge paths, more UI states that never appear in a ticket. Manual checks do not scale with agent throughput — they become a queue of tired humans replaying happy paths while risk hides in the scenarios nobody re-runs.
Brittle end-to-end scripts couple to selectors and layout. A redesign that improves the experience can fail a suite for the wrong reason; a regression that only confuses humans can leave it green. The suite proves the machinery; it does not prove engagement.
Crawls and swarm probes surface broken links and blank pages without a hand-written script. But a probe without a named customer outcome cannot tell you whether this goal still works, and a one-shot run never becomes a regression contract.
There is a subtler failure than slow checking: premature completion. An agent working alone rarely runs out of time or budget — it stops because, by its own assessment, it is finished. Research from Factory makes the size of this gap measurable: rebuilding large programs from scratch, a frontier model reproduced 36 percent of gdal's behavior when it judged its own completion — and 90 percent when a separate role authored an executable standard of completion before implementation began and the work was held to that standard. The model's skill did not change; the judgment of completion did. Additional compute does not help an agent that will not spend it.
The lesson transfers directly to product work: the bottleneck is not only how fast checks run after a ship, but whether the definition of done was ever independent of the implementer at all.
The time and uncertainty between "code is in" and "we know a real user can succeed." AI compresses the left side of that interval and leaves the right side — or worse, moves risk into production analytics and support tickets. Escaped defects of this kind are not always crashes. They are scenarios that fail quietly: the invite that never lands, the recovery flow that loops, the first-value path that requires tribal knowledge.
Teams need a durable, re-runnable definition of "a real user can succeed" — expressed in language product, engineering, and success already use; verified independently of the implementer; backed by evidence a human can trust; and decidable enough to accept or block a ship.
An outcome-first practice: you encode what a customer must be able to do in natural language, an independent agent attempts that scenario on a real product surface, and you treat the resulting evidence-backed acceptance result as the bar for ship — then keep the scenario as a living contract for every meaningful change that follows.
| Object | Role |
|---|---|
| Scenario | Natural-language contract: who the customer is (implicitly or by lens), what outcome they pursue, what success looks like, and what is out of scope. The durable asset. |
| Run | One independent attempt of a scenario against a specific product surface — environment, build, credentials, fixtures. Ephemeral. |
| Evidence | Durable artifacts from the run: what the agent did, saw, and recorded. |
| Acceptance result | The decidable summary — status, score or summary of evidence, issues — consumable by humans and by automation. |
Customer engagement lives inside the scenario wording — who is trying, what "good" feels like — not as a second methodology layered on top.
The difference is not the model in the browser. It is whether customer truth is a durable contract.§2 — What VDD is
These keep the practice about customer truth rather than tooling fashion. None of them require a particular vendor or harness.
Write what success means for a customer before — or at least alongside — the change meant to deliver it. The order is not dogma; the discipline is that "done" is not defined only by the implementer's knowledge of the UI.
The scenario text is the contract. Selectors and page objects may exist as substrate; they are not what product and success can own.
Attempt the goal the way a person would: reading labels, following affordances, recovering from confusion. An agent that only clicks pre-baked coordinates is running a script in disguise.
A wall of green checkmarks without artifacts is weak acceptance. Prefer fewer scenarios with durable evidence over hundreds of brittle asserts nobody trusts.
"Looks fine" is not a ship decision. Soft judgment about friction still belongs in the run — it must land as a decidable acceptance result, not an open-ended chat log.
A scenario that passes once and is discarded was a demo. Scenarios accumulate as regression of engagement, guarding first-value, recovery, invite, and checkout long after the author has moved on.
Scenarios may expand or get stricter as the team learns. They must never quietly collapse around what was already built. When a run fails, fix the product — or change the contract deliberately and in the open.
Engineering, product, and customer success already share one medium for describing a good experience: plain language. None of those sentences mention selectors or page objects.
For the team shipping the change.
For an independent verifier acting as a customer.
Re-run whenever the product surface moves.
When those three jobs collapse onto one natural-language artifact, product outcomes stop drifting away from what automation actually exercises.
First visit, limited product vocabulary, low tolerance for jargon and empty states that assume prior knowledge.
Already has an account or prior context; expects continuity, saved state, and recovery that respects that history.
Will abandon if trust, pricing, or exit options are opaque; needs clear value before commitment.
You do not invent a character backstory. You write sentences like "as a returning member whose session expired, recover access and land where you left off." The lens tightens acceptance criteria and friction signals; it does not require role-play.
Locator-based scripts encode how the UI was wired last week. Natural language encodes what customer outcome must still be achievable. Layouts and copy change constantly — especially when agents generate the UI. Outcomes change more slowly, and when they do, product and engineering edit the same paragraph instead of reverse-engineering a brittle chain of clicks.
A named, natural-language contract for one customer outcome — specific enough that an independent agent can attempt it on a real product surface and produce a decidable acceptance result.
Avoid "test the app," "make sure nothing is broken," or crawl briefs that ask an agent to wander and report vibes — those are probes, not contracts. Prefer customer-recognizable outcomes over implementation steps ("click the blue button in the third card") unless the path itself is the point. If you cannot tell from the text whether a run should pass or fail, rewrite before anyone ships against it.
VDD separates the systems that build a change from the process that judges whether a customer can still succeed. The maker is not the checker.
The implementer knows the intended path and skips the dead end a stranger would hit. They start already authenticated, already on the right URL, already holding fixture knowledge the customer does not have. They treat partial success as "good enough" because they remember yesterday's hard fixes. None of this requires bad faith — familiarity is enough. A coding agent can generate both the UI and a cheerful narrative that the flow "works," while a real browser session for a novice still fails on the first form.
There is a structural problem deeper than familiarity: checks inherit the scope of the work that produced them. As an implementer decomposes a feature, they decide at each piece what evidence would count and whether it is sufficient. Those checks can establish everything the implementer thought to build while excluding outcomes, interactions, and customers that were never represented. A scenario derived from the build verifies the implementation's memory of itself.
Independent verification means an agent attempts the scenario as a customer would: on the actual product surface, through the real UI, under the context the scenario specifies. It is not a locator script replaying known selectors. It pursues the natural-language outcome, discovers affordances as a user would, and stops when acceptance criteria are met or the attempt clearly fails. Layout this week is the agent's problem for that run — not something product rewrites every time a button moves.
Sharing tests with the implementer is sometimes said to invite teaching to the test. The risk is real, but it lives at the case level, not the outcome level. A fixed script — exact inputs, pinned selectors, a known click path — is a sparse sample of the behavior a customer actually needs. Once that sample is visible, it becomes the target: an implementer can patch until it greens without the outcome ever becoming true.
The outcome-level contract is deliberately shared. Teaching to "a novice completes signup without external docs" is simply building the product.
Session, probing, discovered affordances — regenerated each run, never a fixed artifact handed to the implementer.
The outcome crosses the wall; the cases do not. This is also why a locator script handed to the implementer is a weaker instrument than a goal-seeking agent, even when both count as "automation."
Distinct from implementation, with its own session and environment binding.
Written for reviewers and machines, not only a chat reply that scrolls away.
Screenshots, traces, step logs, and a structured result that outlive the run.
Without durable evidence, "it passed" is a rumor. With evidence, a failing scenario is actionable: you see where the customer would have stalled, not only that a boolean flipped false.
VDD does not replace your pipeline design. It supplies a contract (the scenario) and an acceptance result that any stage can re-run. The cycle is intentionally boring.
Goal, acceptance criteria, context, constraints, out of scope, lens if it matters. The test: could a stranger decide pass or fail from the text?
Humans or coding agents implement against that outcome. Branch, preview deploy, or local surface — whatever exposes a real UI.
An agent acting as customer on the real surface, under the scenario's context. Capture a structured result and durable evidence.
Merge or promote, fix and re-run, or open work when the scenario itself is wrong. Acting includes saying no to ship.
Do not delete it after launch. It becomes regression of engagement — the smoke of "a real user can still succeed."
Often product with engineering. Decide what must stay true.
Usually the team that shipped: repair product or fixtures, re-run until the result is acceptable.
Protect the practice from vague scenarios and evidence-free scores.
Always finding a way to green without changing code or contract is not real acceptance.
A scenario without a decidable result is a prompt, not acceptance. The point is not a magic number — it is a result a pull request or release checklist can act on without replaying the session live.
Account created, invite accepted, first report generated, payment confirmed. If the path is blocked — error page, missing control, broken auth — the run fails on reliability regardless of how polished the surface feels.
The outcome may have been reached after dead ends, opaque copy, or optional steps a novice would abandon. Both signals ride the same run so the team does not ship "it worked for the bot" while customers still bounce.
Reliability can be green while engagement is poor; engagement can look smooth while a silent failure means the outcome never completed.
Thresholds should be explicit and stable. Changing them every week teaches people to ignore acceptance.
A score is a compact summary of scored criteria and observed issues — not a free-floating vibe. It should be reconstructible from the evidence pack: which acceptance criteria held, which constraints were violated, which friction points were recorded. If two reviewers cannot explain why the score moved, the score is not ready to drive merge or release.
Less tooling theater, more choosing a few paths that define whether the product still works for the people who use it — and treating those paths as durable contracts.
Start with real first-value or revenue-critical outcomes: signup to first success, core checkout, invite accept, password recovery. Resist covering every screen. Breadth without decidability produces noise and trains the team to ignore results. Product or success often drafts who and what good feels like; engineering owns environment, fixtures, and actionable failures.
Author a single critical scenario, run it independently on a real surface, and review the evidence pack until humans trust pass and fail.
Attach the same scenario to meaningful ships. Start advisory if needed; promote to hard block once flakiness is under control.
Grow a living set as features ship. Retire scenarios only when product outcomes die — not because UI class names changed.
Catch these in the first month; they reappear later as formal failure modes.
On main or release candidates; split reliability vs engagement if results allow.
After a scenario fails, how long until outcome and product agree again.
Support tickets, funnel drops, and P0s on paths a scenario should have caught.
If pass rate is always 100%, thresholds may be too soft or scenarios too shallow. If nothing ever greens, criteria may be undecidable or the environment is lying. Adjust contracts and fixtures before adding more scenarios.
A new engineer can find the critical scenarios, read an acceptance result, and know whether shipping is allowed — without watching a browser live.
One feature through VDD. The shape is what matters: outcome first, independent verify, engagement-aware fail, fix, green, scenario kept.
After a user connects a data source, the product produces a short summary. Engineering can build the pipeline and UI quickly. The open question is whether a novice can reach that first valuable digest without a guide on a call.
Account creation works; the sample source connects; a digest page loads. The evidence pack shows why the customer still lost, without a live watch:
Sensible defaults for the sample source, plain-language labels, and an empty state that shows a partial digest or a single "Generate sample digest" action. A second independent run completes the outcome with no major engagement issues. The score rises because evidence improved — not because someone argued for a higher vibe.
The same contract fails with comparable evidence. The fix is obvious because the customer outcome never moved. Code and agents can iterate freely underneath; customer-proof of first value remains the acceptance bar.
VDD fails less from missing tools than from soft contracts. These seven look productive and still leave you without reliable customer acceptance.
"Test signup" is not an outcome. Without goal, criteria, context, and bounds, two runs cannot disagree usefully — and acceptance cannot decide.
A pass/fail that depends on a human rereading a novel of logs is not usable acceptance. Merge, fix, or hold must follow without a meeting.
A number with no screenshots, steps, or failure notes is a vibe in numeric clothing. If nobody can reconstruct why it moved, the run is incomplete.
A single impressive walkthrough proves little about the next release. Archive demos; keep contracts.
Site-wide probes surface broken links and surprises but encode no named customer outcome. Discovery can seed scenarios; it cannot replace them.
When the actor that implemented the change grades its own result, maker and checker collapse. Self-check is a useful draft; it is not acceptance.
Writing the scenario from what the UI happens to do inherits the implementer's scope. Everything they thought to build is covered; everything they never represented stays unmeasured.
Code is no longer the scarce input. Agents can draft, refactor, and iterate faster than any review queue can watch every path by hand. What remains scarce is a durable account of what must stay true for a real customer.
VDD is that account made operational: customer outcomes written in natural language as scenarios, verified independently on a real product surface, and kept as living contracts.
Who owns customer truth becomes the advantage as autonomous iteration increases. Own it by writing it so it can be re-run — not by hoping the next human QA pass catches what the last one missed.
The practice does not depend on any single product. It depends on treating customer outcomes as acceptance — and keeping that bar honest.
Author one critical scenario for a path that would embarrass you if it broke.
Make the result decidable — status, threshold, evidence a stranger can read.
Re-run it on every meaningful ship. Expand only after that cycle is boringly reliable.
Conceptual — adapt field names and storage to your stack. The methodology requires scenario, independent run, evidence, and decidable acceptance result; not a particular schema or vendor.
| Term | Meaning |
|---|---|
| Customer outcome | What a real customer must still be able to achieve. |
| Scenario | Natural-language acceptance contract for one outcome. |
| Acceptance criteria | Observable conditions that mean the outcome was met. |
| Run | One independent attempt of a scenario on a real surface. |
| Evidence | Durable artifacts (acceptance materials) from a run. |
| Acceptance result | Status, score, issues, and pointers to evidence. |
| Engagement lens | Optional wording: novice, returning, skeptical. |
{
"scenarioId": "string",
"runId": "string",
"status": "pass | fail | inconclusive",
"score": 0,
"threshold": 0,
"summary": "string",
"reliability": "pass | fail | inconclusive",
"engagement": "pass | fail | inconclusive",
"issues": [{
"severity": "blocker | major | minor",
"title": "string",
"detail": "string"
}],
"evidence": {
"steps": ["string"],
"artifacts": ["uri-or-path"]
},
"startedAt": "ISO-8601",
"finishedAt": "ISO-8601"
}
score summarizes evidence against the scenario; never ship on score alone without artifacts. inconclusive means re-run policy applies before treating status as decisive — environment, timeout, or fixture failure, not a silent pass.