LIVE EXPERIMENT - PART 2 of 4

Four ways to build a synthetic user (I tried all of them)

Building our synthetic user in public, Part 2. Part 1 mapped the ways to build one. This time I built each one and put it through its paces.

Follow the experiment→
Tania Clarke
Tania
PMM · Great Question
June 2026~12 min read

Four parts · One live experiment

We're building our synthetic user in public,
start to finish.

01

The map

The vocabulary, the ways to build a synthetic version of your customer, and the priors going in.

YOU ARE HERE
02

Four ways to build one

I built each of the four workflows, and worked out what each is good for and where each falls apart.

03

Robot vs human

An experiment putting synthetic users and real humans head to head. Where do they diverge?

04

The recap & the skill

Concludes the live experiment. The synthetic user skill ships, and everyone on the list gets it.

Data scientists have spent years trying to predict what a user will do next from what they've done before. With the availability of AI, it's hard not to feel optimistic about how we could blend past user behaviour to predict user behaviour and feedback, at a much greater level. I like to think of synthetic users as data science on steroids.

What data science & synthetic users have in common

When it comes to building a synthetic user, we're essentially blending behavioral prediction (product usage), and giving it much better raw material (interview transcripts).

The old way mostly had the numbers to work with, what people clicked and how often. A grounded synthetic user adds the why behind the numbers: the actual words customers used in interviews, and the context that only qualitative data can collect.

Caitlin Sullivan framed it for me that way in Part 1, and it reset how I thought about the whole project. It's the same goal the data teams have always chased, just with far richer input.

The ground rules (they apply to all four): A synthetic user is only as honest as the evidence under it, so the same non-negotiables went into every version before I picked a workflow:

Evidence-backed claims only. No source, no claim.

Cite every claim inline, with the quote attached.

Flag the gaps. When the evidence is thin, the synthetic user says so instead of inventing a pattern.

Qualitative confidence threshold. When I run a qualitative study, I always want to know how many people actually said something before I trust it as a pattern. The skill does the same thing. It tells me how many interviews are behind every claim it makes, and it won't call something a pattern unless enough of them back it up. I set the bar at 8: hit 8 interviews and it states the point plainly, anything below that and it categories the theme as medium or low confidence.

I've built all of these requirements into a synthetic user skill, that I'll be testing for the duration of this live experiment. We'll make it available to you when part 4 lands, and concludes our live experiment.

Why I'm building the synthetic user from a research repository, not a pile of transcripts

The first question I asked Jack, an AI Product Manager from the Great Question team, is why can't I just query a whole bunch of transcripts from github or Google Drive?

The DIY version is to drop a folder of transcripts into Claude and start asking questions. It works for about three transcripts. Past that you hit "lost in the middle," where the model skims the middle of a long document and quietly fills the gaps with things that sound right. You won't catch it, because the invented parts read exactly like the real ones. Jack, who built our repository retrieval system, said this:

"If you don't build a RAG pipeline that knows what it's doing? It's going to be hallucinating left and right. And you won't know."

Jack · AI Product Manager, Great Question

A repository earns its place by doing the unglamorous work that keeps that from happening:

Hybrid search. Keyword and semantic together. Pure semantic search feels clever but loses the exact-string matches that let you anchor a claim to the precise sentence a customer said. You want both running.

Server-side filtering. Rather than shipping a 90-minute transcript to the model and hoping, the repo narrows to the relevant chunks first, so the model only ever reasons over material it can actually hold in context.

Structured metadata. Studies, segments, dates, participants. You can scope a query to "B2B researchers, last 18 months" instead of praying the right transcripts surface on their own.

A curated layer. Insights and highlights you've already validated sit on top of the raw transcripts, so the synthetic user draws on evidence that's been checked, not just whatever the search happened to return.

Citations that resolve. Every claim links back to the session it came from, which is the whole difference between a synthetic user you can audit and one you have to trust blind.

The DIY route can get there, but only by building your own version of all this. Anything you'd actually rely on, and especially anything high-stakes, means building your own RAG: server-side filtering, citation plumbing, metadata, the lot. That's a real engineering project before you've even started on the synthetic user. A repo is that project already finished, which is why all four workflows below run on top of one.

By the way, we did experiment with building a lightweight synthetic persona in the past, which was a collection of 8-10 interview transcripts from a previous study. This felt lightweight to me? My intention with this series is to build something meatier, with MUCH more data available to you than 8-10 raw transcripts.

Jargon, decoded

Two terms worth getting straight.

RAG (retrieval-augmented generation)

RAG

Instead of relying on what the model already knows, you pull the relevant pieces of your own data (your interviews and notes) and feed them in alongside the question, so the answer is built on your evidence.

Lost in the middle

Lost in the middle

Language models read the start and end of a long document closely and skim the middle. Hand one a 90-minute transcript and the middle is exactly where it's most likely to miss something or quietly make it up.

The build

4 ways to build a synthetic version of your customer

Here are the four ways I experimented with below, and their pros and cons.

Workflow 1

Digital twin

How I built it

Take one real, named user you have deep data on. Strip the PII (any personal details that identify them), store what's left as a synthetic-user document, and tell the agent to answer as that person. It's the highest-fidelity option because it's grounded in one real human rather than an average.

Best data to ground it

This one goes deep on a single person, not wide. You want everything you have on them: their interview transcripts, their product-usage history (feature adoption, drop-offs, the Mixpanel trail), their support tickets, and their CRM and sales-call notes. The richer the single-person record, the more convincing the twin.

The trade-off

Its strength is fidelity. Nothing gets you closer to a specific person's perspective, which makes it ideal when a key account or a design partner needs a seat in a roadmap conversation. Its limit is that it's exactly one person, quirks and all, so it can't speak for a segment.

Workflow 2

Segment-based synthetic user

How I built it

Aggregate eight to ten or more real users across a bunch of different segments into one synthetic user. You're building an archetype from a cluster of evidence, then writing it up as a single coherent person whose every trait traces back to the underlying interviews.

Best data to ground it

Breadth is the whole game here. Transcripts across the segment for language and goals, the insights and highlights you've already curated for themes that are validated, candidate and demographic data so no single sub-group dominates the mix, and product-usage data so the behavior is real and not self-reported.

Step zero is an audit: do you actually have eight or more solid sessions on the segment you want? If not, your next step would be to fill that research gap, so you have solid foundations to build upon.

Types of segment-based synthetic users:

Power users: gather the data on your 'best' customers, how they're using the product, what they say, what they love, what feedback they've given in the past.

Casual users: gather product usage data on a segment of your less frequent users.

Churned users: pull out churn surveys, customer interview transcripts or closed-lost interviews.

The reason I like segment-based synthetic users is because you can then run a PRD or an artifact, or a concept past all 3, and then compare the insights. I love experimenting with Perplexity's model council for this reason. If you haven't used it yet, Perplexity's model council runs any query through 3 models so you can triangulate and sharpen your point-of-view based on the models arguing against each other. Fun stuff.

The trade-off

Because it's built on a few dimensions of data, the patterns are stable enough to trust…with a grain of salt of course.

Its limit is that you sand off the sharp edges of any one person, and it's only as good as your data's coverage of that segment. Thin segment = thin synthetic user.

Again, at best this could be used internally to drive customer empathy, and provide some directional feedback on new concepts. We know from Part 1, that nuance is where synthetic users unfortunately fall down…today.

Workflow 3

Synthetic panel

How I built it

Sample several synthetic users from a segment (built on survey data), and run them through the same study together, the synthetic version of a recruited panel. Instead of one voice, you get a spread of them answering the same questions.

Best data to ground it

This one is built purely off a large survey, where you'd have statistical significance, plus enough demographic and behavioral variation in the source data to make the panel genuinely diverse.

The trade-off

Its strength is distribution. You get a range of responses rather than a single point estimate, which is exactly what you want for dry-running a survey or catching a broken question before a human ever sees it. Its limit is that the diversity is capped by your data, and a panel that looks varied but isn't will hand you false confidence.

Workflow 4

Live retrieval

How I built it

No stored user at all. A skill queries the whole repo live, contextualized to whatever artifact you feed it, a PRD or a design spec, and assembles the relevant customer evidence on the spot. This is the one Ned demoed on our recent synthetic user webinar.

Watch Ned demo it on our synthetic user webinar →

He pasted in a PRD, the skill built synthetic users from the matching screener and interview data, and each one reacted to the parts of the PRD he put through.

Best data to ground it

This leans on the whole repo rather than one stored doc, so it lives or dies on the indexing from the section above, the hybrid search and server-side filtering especially. Whatever's freshest in the repo is what it pulls, which is the point. You want it reacting to your latest evidence, not a stored persona document from last year.

The trade-off

Its strength is that it's always current and contextual. You point it at the artifact in front of you and it reacts to that, with no document to maintain. Its limit is consistency. We experimented with it a few times while prepping the webinar, using the same repository and a similar prompt, and the outputs came back consistent-ish but not the same. The themes held every run; the exact wording and the examples it reached for moved around. Useful for a directional gut-check, risky the moment two people quote their own separate runs as the source of truth.

So which one do you reach for?

It comes down to what you're holding when you start. A specific person's perspective points to a digital twin. A whole segment points to the segment-based build. A need for spread, like dry-running a survey, points to a synthetic panel. A live artifact you want reactions to points to live retrieval. And for any high-stakes go/no-go, real users still win. Synthetic users is where you can start, but not use as a final decision point.

The consistency question

Does it matter if everyone gets a slightly different answer?

This is the part I keep going back and forth on. Live retrieval gives a slightly different answer each time you run it. Does that actually matter? Especially if that data is the most recent?

I saw a small version of this in our own testing. We ran live retrieval a few times while prepping the webinar, same repo, similar prompt, and the answers came back close but not identical. The big themes held every time, which was at least comforting. The exact wording and the examples moved around. That's fine if you just want a quick gut-check. It's a problem if two people each run it and quote their own version as the truth.

So my answer is yes, consistency matters, but not all the time. For a quick directional read, a slightly different answer each time is fine. For anything the whole team is building around, you want one version everyone trusts. That's why I think we'll keep one saved synthetic user as the official reference, and use live retrieval for one-off questions on top of it.

From the webinar

The questions the webinar raised

When Ned ran the live demo, the chat filled with sharp and thoughtful questions that we wanted to outline here:

On the data behind a synthetic user

How much cleanup did it take before the output felt decent?

Less than you'd expect. Coverage matters more than polish.

Does it use the repo's transcripts? What about usability-test videos?

Yes to transcripts, they're the backbone. Video works through its transcript today, so the spoken content is in but the on-screen behavior isn't yet.

On trust and accountability

Isn't it easy to over-trust this?

Yes, which is exactly why the skill we've built cites every claim and flags every gap.

Who's accountable when the output informs a real decision?

A genuinely tough question. I think where we're landing is that people are, not AI. Its job is to make the evidence legible enough that a researcher and a PM can share that accountability.

How do you coach people to read it?

Get them looking at the citations and the gaps before the conclusions.

On replacing humans

Is this only for low-stakes calls?

Early discovery isn't always low-stakes, it can set product strategy. So the rule holds: synthetic is the floor, not the ceiling, and the higher the stakes, the faster you go to real humans.

How do I stop budget-holders swapping humans for synthetic to save money, without being called "anti-AI"?

Lean on synthetic where it's strong, like dry runs and gap-finding, and say plainly where it isn't, like anything you'd bet the roadmap on.

The hard ones I can't fully answer yet

Hyper-rationalization:

real people reason messily, synthetic ones don't.

Web access:

does letting it browse inflate what a synthetic user "knows"?

Model temperature:

how do you get realistic variety instead of ten identical voices?

Brand knowledge:

how much should a synthetic user know about you?

Language bias:

AI tends to reward concrete, action-oriented wording over abstract ideas.

WEIRD bias:

model training data skews Western, Educated, Industrialized, Rich and Democratic, and aggregation alone won't fix it.

I don't have clean answers to that last group. What I have is repo-only retrieval, so the model isn't pulling in web knowledge a real customer wouldn't have, plus a gap flag that fires when the evidence isn't there.

Under the hood

The skill behind all this

Everything above runs on a skill I built. Here's how it works below:

Point it at the repo, and it builds profiles

It surveys what's in the Great Question repo, groups sessions into clusters by role, workflow and shared pains, and counts the evidence behind each one:

sessions: a high-confidence profile, stated plainly

a profile, but hedged

a signal, not yet a finding

a gap, flagged for you to go research

Each profile card covers who they are, the world they work in, their pains in their own words, the phrases they use, what they want, what it's safe to use them for, and what it still can't tell you.

Hand it a PRD or design, and a profile reacts

It pulls the relevant evidence with hybrid search (keyword and semantic at once), then responds in the first person as that customer, pushing back where the evidence contradicts the artifact and flagging anything the repo can't speak to.

The rules, every time

Every claim cites the real session it came from, by anonymous speaker handle, never a name. The repo is the only source. If the evidence isn't there, it says so instead of filling the gap.

It's not public yet, on purpose

I'm testing it in the open across this series first, so you can see where it holds up and where it doesn't before you run it yourself.

What's coming in Part 3

Part 3 is about the robot vs human experiment I'm running! I'm going to design an experiment and put both synthetic users and humans head to head. It should be interesting!

What I honestly don't know going in is where they'll diverge.

The build continues.
Follow along.

Part 3 lands next: the robot vs human experiment, putting synthetic users and real humans head to head.

01
The map
Vocabulary, the ways to build one, priors going in.
02
Now · Four ways to build one
Built each workflow. What each is good for, where each falls apart.
03
Coming · Robot vs human
Synthetic users and real humans, head to head.
04
Coming · Recap & the skill
The full guide, and the synthetic user skill you can run yourself.
Live experiment · Part 4 of 4

We built a synthetic user skill.
It's now yours.

Four editions of building and testing, plus one experiment we killed halfway through.

This week we open up the skill, give you a decision tree and wrap up our findings.
Get the skill
Tania
PMM · Great Question
The Synthetic User Series
01
Catch up · Part 1, 2 & 3
Synthetic users: we're building in public
The vocabulary, the ways to build one, and the priors I took into this experiment.
Four editions · One live experiment

We built in public, start to finish.

01
Terminology & hypothesis
The vocabulary, the ways to build a synthetic version of your customer, and the priors going in.
02
The build
The data audit, the rigor framework, and the step-by-step build from 726 transcripts.
You are here
03
The experiment that broke
We tried to grade synthetic against real, couldn't, and it changed what we think they're for.
You are here
You are here
04
The skill & wrap-up
Our top findings on synthetic users, when to use them, plus the Claude skill you can run yourself.

Four editions ago, I said most of the internet was arguing about whether synthetic users should exist while almost nobody was building one in public and telling you what they found. That was the whole point of this series: build one out in the open and show you the parts that didn't go to plan.

So before I open up the skill, I want to be honest about where the experiment actually landed.

Where it landed

Part 3 was meant to be the head-to-head: a synthetic panel against a real one, with a clean accuracy number at the end. We killed it. You can't grade a synthetic user's accuracy when you can't grade a human's either, and I wrote up the whole detour in Part 3.

The takeaway was: stop asking how accurate a synthetic user is, and start asking what it's good for and when to trust it. That question shaped the skill I'm shipping today.

My top findings

Between building this on our own data and a steady stream of calls with product and research teams building the same thing, here's what's valuable to know before you start your own experimentation.

01
It lives or dies on your own data
A synthetic user built off a generic "market" model knows nothing about your product or your customers. It's a statistical impression of a demographic an LLM read about online, and it answers like one. The whole value is in grounding it on your own transcripts. If the evidence on a segment isn't there, you've found your next research project, and no amount of clever prompting fills that in.
02
Build from verbatim, never summaries
One team I spoke to built theirs off the study summaries in their repo, and it made things up. You want the skill pulling exact verbatim from the transcripts. The real quotes are what make it echo your customer's language back to you instead of something generic and "clean."
03
A repo beats a pile of transcripts, every time
Dumping 20 or 30 transcripts will somewhat work, but not very well: it loses the middle of a long context and fills the gaps with things that sound right. Point it at an indexed repo where the retrieval is already done, then supplement with a handful of sales calls or support transcripts if you want more dimensions to your synthetic user.
04
Treat it as a sparring partner, not a final insight
Every team I talk to that has experimented with synthetic users, use it to react to concepts and rough prototypes as a fast first pass, before they hand it to research. Nobody's presenting "insights from a synthetic user" as the final finding. It's the dry run, and the real interviews still come after.
05
Watch out for 'clean' answers
Real people say surprising things. A synthetic user, even one built on real verbatims, hands you a smooth, low-variation answer. It's strongest on directional, yes-or-no, good-or-bad questions and weakest on nuance: the emotional, identity-laden ones, and brand-new concept areas where your repo has no current research on. That last case is exactly where it'll confidently invent, which is what the confidence meter in our synthetic user skill is there to catch.
06
Make it argue with you
Because it has so much context on you, it's agreeable to a fault. The sycophancy problem again. So build an adversarial version: instruct it to critique, or ground it in churned and closed-lost interviews so it plays skeptic instead of fan. A synthetic user that pushes back is worth ten that nod along.
07
It goes stale, so let the gaps drive you
The data ages. Refresh it through the MCP when you have new transcripts, and when it flags something it can't answer with confidence, treat that as your next research brief. Go fill it with humans, then fold the new evidence back in to your repo and update your synthetic user.
08
And the one that surprised me most: it's fast
With the MCP wired up, I've built a grounded synthetic user in about 5 minutes. Ask it which customers you have the most data on, get a shortlist, sanity-check it isn't hallucinating, and go.
The decision tree

So which workflow, when?

Across Parts 1 and 2 I built four ways to make a synthetic version of your customer. People keep asking which one is "best." They do different jobs and can be used based on what you need to know.
Which synthetic user? A live artifact to test A classic persona One key account A survey dry-run Live retrieval Reacts to your artifactin the customer's voice.Strongest at critique. Segment-based Built once from acluster; the team'sshared reference. Digital twin One customer, deepestfidelity. Needs 8+sessions on them. Synthetic panel A spread of voices fora dry run. Needs realsurvey scale. THEN RUN THREE GATES BEFORE YOU TRUST IT GATE 01 Grounded? Every claim traces to8+ cited sessions. GATE 02 How high are the stakes? Low, cheap-to-be-wrong:go. Go/no-go decisions:humans. GATE 03 Rational or emotional? Trust the logical calls;be cautious of anythingmore nuanced. lower stakes high stakes ✓ USE IT Run it as your first pass,then aim research. → HUMANS Skip synthetic and gostraight to real people.
One more the tree bakes in: if consistency across your team matters, and in enterprise it almost always does, keep one saved synthetic user as the official reference and use live retrieval for one-off questions on top of it. Live retrieval causes drift across synthetic users.

Inside the skill

I've been referencing "the skill" for four editions. It runs entirely on the Great Question MCP, the only source it's allowed to touch, and it does two jobs.
Mode 01 · Default
Point it at the repo, it builds profiles
No artifact needed. It surveys the sessions, groups them into clusters by role, workflow and shared pain, counts the evidence behind each, and hands back reusable profile cards you can summon later.
Mode 02 · On demand
Hand it a PRD, a profile reacts
It pulls the relevant evidence with hybrid search, then answers in the first person as that customer, pushing back where the evidence contradicts the artifact and flagging anything the repo can't speak to.
The design principle behind both, in the skill's own words:
You are not a generic AI persona. You do not extrapolate beyond what the evidence supports. When the evidence isn't there, you say so and flag it as a research opportunity rather than fill the gap.
Everything else is enforcement of that one idea. Five non-negotiable rules run on every response, whatever you ask.
Rule 01
Evidence-backed claims only
Every claim, reaction, concern, or pushback you return must be backed by retrieved evidence from the Great Question MCP. No source, no claim.
Before it includes any claim, it runs a self-check: does the retrieved quote explicitly support this specific point, or am I making an inferential leap? Am I confusing a study title with what a participant actually said? If the answer is "unsure," the claim gets dropped and logged as a gap.
Rule 02
A confidence threshold baked in
The 8-interview bar I've mentioned since Part 2, is hard-coded into the skill. It mirrors the pattern-matching threshold our own research team uses, and it always shows the count, e.g. "Mentioned in 12 of 14 interviews across 3 studies."
High
8+
sessions → stated plainly
Medium
3–7
sessions → okay confidence
Low
1–2
sessions → "a signal," not a claim
Gap
0
sessions → surfaced as a gap
That count is the confidence flag Part 3 argued was the most useful thing the tool can give you, the error bar you otherwise can't feel.
Rule 03
Cite every claim, and make the citations resolve
Every point traces back to the session it came from, quoted verbatim, by anonymous speaker handle. The rules are strict on purpose: quotes must be at least three words and copied exactly from the transcript search, badge text has to match the quote, and it's forbidden from ever citing a study summary as if it were something a participant said. No quotation marks, no invented paraphrase dressed up as evidence. Of course, I do encourage you to do your own checks here...
Rule 04
Surface gaps
Every response ends with a Research Gaps section: the topics where the evidence ran out, the claims it would have made with more data, and suggested questions to close the hole. The line in the skill file is blunt:
The gap is valuable, flag them.
Filling these gaps in future will also help you build more robust synthetic users in the future.
Rule 05
Stay in character, stay honest
It speaks in the first person as the customer ("this is the part I'd push back on…") because that's what makes it useful to react to. But voice never overrides evidence. If it can't back something up, it doesn't say it, not even in character. A reaction is still a perspective grounded in a real quote.
Two more things. Anonymisation runs by default: speakers only ever appear as handles like Speaker_abc123_1, never real names, which is what makes the aggregated workflow safe.

‍And it's honest about non-matches: if it finds nothing relevant, it says so and recommends the research to run, rather than dressing up tangential quotes to look thorough.

That's the whole skill. Its only job is to refuse to say more than the evidence allows, which after Part 3 is exactly what I want it doing.
Live experiment · Part 4 of 4

Free synthetic user skill - now yours!

The synthetic-user skill is live. Point it at your Great Question repo and it'll build profiles on day one. Hand it a PRD and one of them will react.

And I want testers. Building this in public only works if the last step is public too. Run it against your own repo and tell me where it breaks, especially where a citation doesn't hold or where it should have flagged a gap and glossed over it instead.

A full synthetic user guide is coming soon: the terminology, when to use it, and more.

Sign up to our newsletter and we'll send it the day it lands.
Get the skill
Sign up to hear more from us
If you've followed along since Edition 1, thank you.
Tania
PMM · Great Question
The Synthetic User Series
Get great UX research reads & resources once a month.
Thank you! Please check your email to confirm your subscription.
Oops! Something went wrong while submitting the form.
© 2026 Great Question, Inc. All rights reserved.