How to run a Wizard of Oz test before you build

Every product team that has tried to validate an ambitious feature (an AI assistant, a routing engine, a smart triage inbox, a recommendation surface) has run into the same wall: the feature is too expensive to build for a validation study and too underspecified to describe in a survey. The team ends up interviewing users about a feature that does not exist yet, gets a room full of polite affirmations, and ships something the audience never asked for. The Wizard of Oz test is the method that removes this wall, and how to run a Wizard of Oz test correctly is one of the highest-leverage skills a product team can pick up before they commit engineering time to a feature they have not yet earned the right to build.

This is a working playbook on how to run a Wizard of Oz test in 2026: what the method is, the failure modes that turn it into a demo instead of a finding, the six steps that produce evidence about the interaction the team is thinking about building, and where the same instrument goes to work again after the first round.

What a Wizard of Oz test is

A Wizard of Oz test is a research method in which a participant interacts with what appears to be a working system, while a human operator (the "wizard") is secretly generating the responses the system pretends to produce. The method was named and formalized by John F. Kelley in his 1984 paper "An iterative design methodology for user-friendly natural language office information applications" at IBM, though the technique had been in informal use in HCI research earlier. The clearest short reference in current practice is Nielsen Norman Group's article on the Wizard of Oz method, which frames it as a way to test complex or expensive technologies (chatbots, voice assistants, agent-shaped features) before the underlying model or logic is built.

The unit of analysis is the interaction, not the interface. The participant is not evaluating a screen. They are moving through a workflow, making a request of the system, receiving a response, and reacting to it. The wizard, hidden from the participant, is producing that response by hand, in real time or in a scripted branch. The output of the study is a set of transcripts and recordings in which the participant thinks aloud about what the system did, what they expected, and what they would try next.

The method sits early in the design-validation pipeline, upstream of any prototype that actually runs code end-to-end. It runs after a rough interaction is designed on paper and before the engineering team commits weeks to implementation. It is one of the few tools that produces real user reactions to a system that is still a hypothesis.

Why most Wizard of Oz tests return a demo, not a finding

Three failure modes show up across most Wizard of Oz tests that returned nothing beyond a video reel of the wizard performing well. Each one is structural, and they tend to appear together.

The first is treating the session as a demo, not a study. When the wizard's job is to make the interaction feel magical, the participant ends up smiling politely at the show and the team goes home with a pull-quote about how impressive the future feature will be. The interaction has not been tested. The wizard has been rehearsed. The fix is to script the wizard's responses to include the failures the real system will actually have, not just its successes, and to instrument the study to catch what the participant does when the response is wrong.

The second is the wizard improvising without a decision rule. If the wizard decides in the moment what a good response looks like, each session is a different system. The participant in session one is talking to a wizard playing a helpful assistant. The participant in session five is talking to a wizard playing a slightly bored analyst. There is no way to compare the sessions, because the "system" was different in each. The fix is to write the wizard's decision rules in advance, treat them as the design under test, and revise them between sessions as findings emerge.

The third is capturing only what the participant did, not what the participant thought. A recording of the participant clicking around while the wizard generates responses tells you the participant kept engaging. It does not tell you what mental model they were building of the system, whether they trusted the responses, or what they would have tried next if the wizard had failed. The interaction data is the surface. The reasoning is where the design decision lives.

How to run a Wizard of Oz test, step by step

Six steps. The first three set up an experiment the team can learn from. The last three are where the dataset either supports a decision or evaporates into anecdotes.

01 · Pick the interaction that has to be simulated before code

A Wizard of Oz test validates a specific interaction, not a whole feature. Write the interaction as a single sentence in the team's own voice ("the user asks the assistant to draft a customer-response email, and the assistant returns a draft the user can send, edit, or reject"). The wizard's job is to produce responses to that specific loop, and everything outside the loop stays in placeholder state.

Interactions worth simulating share three traits. They are expensive to build in the fully-functional version (an AI model, a real-time routing engine, an integration that would need weeks of work). They are load-bearing on the feature's value proposition (if this interaction does not land, the whole feature does not). And they are ambiguous enough on user reaction that the team cannot predict the outcome from a spec review alone. If any one of the three is missing, cheaper methods will usually get you a faster answer: concept testing on a written or storyboarded description, or a static prototype validated in usability testing.

02 · Script the wizard's decision rules

The wizard is not free. The wizard is running a specific design. Write the decision rules as a short branching document: for each class of user request, what is the response the "system" should produce. Include the good responses (this is what a working version would return) and the failure responses (this is what a version that misunderstands the request would return). The failures matter more than the successes, because they are what tell you whether the participant recovers, escalates, or abandons.

The wizard needs at least three response tiers scripted. A correct response, so the interaction has a happy path to test. A partially-correct response, so the study can observe repair strategies. A wrong response, so the team learns the failure mode the participant would tolerate versus the one that ends the session. Without the failure tiers, the study only observes what happens when the system works, which is the least interesting question a Wizard of Oz test can answer.

Between sessions, the decision rules get revised. The Wizard of Oz test is iterative in the strict sense: the "system" the second participant meets is the version corrected against what the first participant surfaced. The wizard's script is the design artifact, and the version-history on that script is one of the study's outputs.

03 · Recruit participants doing the underlying work

Twelve to twenty participants across two or three sessions each is the working range. Below twelve, the interaction patterns are too participant-idiosyncratic to generalize. Above twenty, the wizard's script has usually stabilized by session fifteen and marginal returns drop sharply.

The screener filters on behavior, not stated interest. The audience is people who currently do the work the simulated system would automate, expedite, or replace. If the "system" is a customer-response drafter, the participants are people who write customer responses in their real jobs today. The operational side of recruiting is covered in how to recruit user research participants. The rule specific to Wizard of Oz: participants who do not do the underlying work in their real jobs will engage politely with the simulated system and produce reactions that are theatrical, not diagnostic.

04 · Run the session, live or async

The classic Wizard of Oz session is live. A moderator sits with the participant, the wizard sits in a separate room, the participant types or speaks a request, the wizard produces a response, and the participant thinks aloud as they read it. This shape still works and is the default when the interaction is fast (sub-30-second turnaround) and the wizard's script is short.

The async version works when the interaction is naturally slower (a report the "system" would generate in a few minutes, a research plan the "system" would draft overnight, a routing decision that would normally take a workday). The participant sends a request through a form or a voice prompt, the wizard produces a response within an agreed turnaround, the participant receives the response and records their reaction. The async shape trades the moderator's real-time observation for a wider pool of participants who cannot spare an hour for a live session, and it usually preserves more of the participant's honest reaction, because they are not performing for anybody.

Whichever shape, the participant thinks aloud. The recording is the artifact. The interaction data (what they typed, what the wizard sent back, what they did next) is the frame, and the participant's own words about what they thought the "system" was doing are the payload.

05 · Capture the participant's reasoning alongside the interaction

This is the step most Wizard of Oz tests skip. The default instrumentation captures what happened: the request the participant made, the response the wizard generated, the click the participant made afterward. It captures nothing about why the participant did what they did.

The fix is to probe the moments that matter, in the modality that fits the moment. A well-run Wizard of Oz session has four kinds of probes, one per modality:

• On the participant's first reaction to a wizard response, ask voice: "walk me through what you thought when you read that." The open reasoning arrives in a sequence of partial thoughts and revisions that compresses badly into a text field, and the voice recording preserves the pause and the hedge alongside the words.
• On a confident acceptance of a wizard response, ask text: "in one sentence, what made you accept it." The short clarifier is enough, and a full voice probe on an easy answer adds friction the study does not need.
• On a tone read of the interaction, ask rating: "on a scale of 1 to 5, how much did that response feel like it understood you." A number is exactly what the synthesis wants for trend detection across sessions.
• On a hesitation or a low-confidence acceptance, ask choice: "were you accepting because you agreed, because you were tired, or because you did not know what else to try." The dichotomy is more honest than a free-form follow-up, and the option list forces the participant past a polite default.

Adaptive follow-up probing earns its keep on the open reasoning moments. Treat probing depth as a per-prompt setting, owned by the researcher: medium depth on the participant's open reasoning about a wizard response, shallow on the rating and choice probes, and expert depth when the participant contradicts themselves or names a mental model the team did not anticipate. The longer treatment of how to set follow-up depth is in how AI follow-up questions work in user research. The short version: depth is a methodology decision, not a global toggle.

"I thought it was going to escalate that to a supervisor. When it just drafted the reply itself, I... I felt like I had already given up my say. I did not know it could do that."

06 · Synthesize the failure modes, not the demos

A common mistake at synthesis is to collect the sessions in which the wizard performed well and present them as proof the interaction lands. The findings live in the sessions where the wizard's response missed, and in what the participant did next.

Cluster the sessions by the failure mode the wizard triggered. Which wrong responses did participants recover from without help, and which caused them to abandon the interaction. Which right responses did participants trust immediately, and which did they second-guess even though the answer was correct. Which requests did the wizard have no script for, and how did the participant frame the request they wanted to make. The last cluster is often the most valuable output of the study, because it tells the team what interactions the real system will need to support that the design did not anticipate.

The general thematic pass we describe in how to analyze user interview transcripts still applies, with one substitution: the atomic unit being coded is a wizard-response / participant-reaction pair, not a quote about the product. Code by the participant's mental model of what the system was doing, not by the surface content of what they said.

How to run a Wizard of Oz test on an AI feature

The Wizard of Oz method has always fit AI features and it fits them more now than at any point in the method's history. Kelley's original 1984 use was a natural-language office application, precisely because building the real language model was more expensive than simulating one. The situation in 2026 is structurally the same: an LLM-powered feature is cheap to prompt and expensive to build into a production product, and the interaction design (what the feature offers, when, in what tone, at what confidence) is where the value lives, not in the model choice.

Three adjustments matter when the "system" the wizard is impersonating is an AI feature. First, script the wizard's failures to match the failures a real model would have, not the failures a human operator would have. Real model failures are confidently wrong, off-topic in specific ways, and prone to over-explaining. Human failures are hesitant, apologetic, and self-correcting. If the wizard's failure mode is the human kind, the participant will react to a system that will never exist. Second, script latency into the wizard's responses. Real inference has visible delay; instant responses in a Wizard of Oz session read as suspicious to the participant and change how they trust the "system". Third, script the same request across two or three different response styles, and vary which style the wizard uses across sessions, so the study returns a read on how the response tone (confident, hedged, structured, conversational) shifts what the participant does with the answer. The wider treatment of research on AI features is in how to run user research on AI features.

When to run a Wizard of Oz test internally first

A pattern that under-uses the Wizard of Oz method: running it only externally. The same instrument works inside the company, and running it internally first almost always saves a round of external sessions.

Before the first external participant sees the simulated system, share the same study inside the team: engineering (what interactions did they think they were building), design (what mental model did they intend the participant to hold), support (what interactions do customers currently need help with in the shape this feature would replace), operations (what edge cases does the current process handle that the simulated system would inherit). Each internal respondent runs through the wizard-simulated interaction and answers in whichever modality the question wants: voice on the open reasoning about the simulation, text on the specific fix suggestion, choice on the edge case they know matters, rating on how confident they are the feature will land.

The output is a synthesized view of every stakeholder's reaction to the simulated feature before the external test loads any cost. Engineering surfaces the interactions that would be hard to build behind the ones the participant expected. Support surfaces the edge cases the wizard's script has not covered. Design surfaces the tone drift between the intended interaction and the one the wizard is actually producing. The external study starts on ground the internal team already agrees with, instead of on assumptions the first external participant will hit cold.

Where to keep the study link live after the first round

A Wizard of Oz test is usually treated as a one-shot study: run it before a feature is built, ship the interaction design, close the link. The version that scales is a standing instrument for interaction-level signal.

Four placements that work for the Wizard of Oz shape specifically:

• Inside a private beta of the built feature. Once the real system is live for a small cohort, keep a Wizard of Oz variant running alongside it on interactions the real system does not yet handle. Beta users try the not-yet-real interaction against the wizard, and the responses tell the team which of the wizard's edge cases to build next.
• In a customer-support surface where the real feature is being considered. If the feature would eventually replace a support workflow, keep the wizard-simulated version live for a subset of tickets. The participants are real users doing real work, the wizard is a support team member trained on the scripted decision rules, and the tickets are the interaction transcripts.
• In internal channels for stakeholder review. The same wizard-simulated interaction, shared as a study link inside Slack, returns a continuous read on how the shipped design is aging against internal reviewers. Engineering, design, support, and sales each run through the current interaction, respond in their own words, and surface drift the external testing would take weeks to catch.
• In a research repository as a reusable artifact. The wizard's decision rules, the participant transcripts, and the synthesized failure modes are worth carrying into the next study on the same feature. A Wizard of Oz session six months old is a benchmark for how the shipped system's interaction has moved.

The framing to use: a Wizard of Oz study link is a standing instrument for capturing reactions to an interaction that has not yet stabilized, not a survey campaign with a start and end date. The interaction changes as the feature is built, and the standing instrument catches whether the built version is drifting from the design that tested well.

When a Wizard of Oz test is the wrong tool

Three cases where a Wizard of Oz test returns theatre instead of a finding.

The interaction is already cheap to build. If the real version of the feature can be prototyped in a day, prototype it. The Wizard of Oz method exists to buy interaction-level evidence about expensive systems. On cheap systems, the real prototype is a better artifact for the participant to react to, because it forces the design constraints the simulated version can dodge.

The team does not yet know what interaction to script. A Wizard of Oz test validates a specific loop. If the team has not yet decided what the loop is, the study will design the loop under pressure, in front of participants, and return sessions that are as much about the team's confusion as about the participant's reaction. Run a discovery pass first: a customer discovery interview or a jobs-to-be-done interview to surface which interaction the audience actually wants, then script the wizard against the answer.

The feasibility question is unresolved. The Wizard of Oz method is a desirability and interaction-design instrument. It cannot tell the team whether the underlying model or engine can be built at all, only whether the interaction with a working version would land. If the constraint on the feature is technical (can the model do this, at this latency, at this accuracy), the answer lives in a feasibility spike, not in a Wizard of Oz session.

How a Wizard of Oz test fits into a wider research practice

The Wizard of Oz test is one tool in a design-validation practice and it pairs with three others at different stages of the build.

• Concept testing sits upstream, on the proposition the interaction expresses. The concept testing playbook covers the pre-interaction validation: does the audience want this feature at all. Run concept testing before the wizard is scripted.
• Usability testing sits downstream, on the built interface. The usability testing playbook covers the post-build validation: can the participant actually complete the task with the real system. Run usability testing once the wizard's decision rules have been implemented in real code.
• Continuous discovery sits around all of them. The continuous discovery interview playbook covers the weekly cadence that keeps the interaction the team is building against the interaction the audience is asking for. A Wizard of Oz session on an interaction the team has stopped hearing about in discovery is a precise read on a stale design.

All four sit inside the wider practice covered in the voice user research guide. The shorthand: hear the problem, script the interaction, run the wizard, build the system, test the build, keep listening. Skip any step and the next one is validating something the previous step has not earned.

FAQ

What is a Wizard of Oz test in user research?

A Wizard of Oz test is a research method in which a participant interacts with what appears to be a working system while a human operator (the "wizard") secretly generates the responses the system pretends to produce. The method was named by John F. Kelley in a 1984 ACM paper on natural-language office applications and has been standard in HCI research since. It is used to test the interaction design of complex or expensive systems (chatbots, voice assistants, AI features, agent-shaped interfaces) before the underlying model or logic is built, so the team can learn what interaction lands with the audience without committing engineering time to a design that would fail in production.

When should you use a Wizard of Oz test?

Use a Wizard of Oz test when the interaction is expensive to build in a fully-functional version, the interaction is load-bearing on the feature's value proposition, and the team cannot predict the audience's reaction from a spec review alone. If any one of the three is missing, cheaper methods will usually get a faster answer. On a static screen, 5-second testing and usability testing are better tools. On a proposition the team has not yet validated, concept testing is upstream of the wizard.

How many participants do you need for a Wizard of Oz test?

Twelve to twenty participants across the study, screened on the behaviour the simulated system would automate or expedite in real life. Below twelve, the interaction patterns are too participant-idiosyncratic to generalize. Above twenty, the wizard's script has usually stabilized and marginal returns drop sharply. Because the wizard's script is revised between sessions, the study is not a static instrument; the "system" the last participant meets is a version corrected against what the earlier participants surfaced.

How is a Wizard of Oz test different from usability testing?

A usability test validates a built interface against a specific task and returns evidence on whether the participant can complete the task with the real system. A Wizard of Oz test validates an interaction design against the participant's mental model of the system and returns evidence on whether the interaction is worth building at all. The two are sequential, not interchangeable. Run the Wizard of Oz session before you commit engineering time to the feature, and run the usability test once the wizard's decision rules have been implemented in real code. A Wizard of Oz session on a shipped feature is a benchmark; a usability test on a not-yet-built feature is impossible.

Can Wizard of Oz tests be run async?

Yes, and the async version is now the working default when the simulated interaction is naturally slower than a real-time conversation (a report the "system" would generate in a few minutes, a routing decision that would take a workday, a research plan the "system" would draft overnight). The participant sends a request through a form or a voice prompt, the wizard produces a response within an agreed turnaround, the participant records their reaction, and the AI interviewer probes the open reasoning in the modality that fits (voice on the mental model, text on the confident clarifier, rating on the trust read, choice on the hesitation). The async shape trades in-the-moment moderator observation for a wider participant pool and usually preserves more of the participant's honest reaction, because they are not performing for a camera.

---

A Wizard of Oz test fails when the wizard is rehearsed, the failures are missing from the script, and the team synthesizes the demos instead of the misreads. It works when the interaction is written down as a specific loop, the wizard's decision rules include the failures the real system will actually have, the participant's reasoning is captured alongside the interaction, and the synthesis clusters by failure mode and by the requests the wizard had no script for. Talkful is built for the second shape: a Wizard of Oz study link goes out, participants react to the wizard-simulated interaction and answer in voice, text, choice, or rating on their own time, the AI interviewer probes the open reasoning and the low-confidence acceptance into honest mental-model data at the depth the moment deserves, and the synthesis engine streams failure-mode clusters and edge-case requests alongside the participants' own words, ready for the team to ship from or for the agents you build with to act on. The wider voice user research guide covers where the Wizard of Oz test sits inside a continuous research practice; the downstream usability testing playbook covers the step that validates the interaction once the design has earned the build.