How accurate are synthetic users? Our focus group study

Why we ran this study

The first question every UX researcher asks us is a fair one: how do you know this is true? Synthetic users, also called AI personas or digital twins, browse a website like real customers. If they only told you what an AI thinks a customer would say, they would be a creative writing tool, not a research tool.

So we tested it the way you would test any new research method. We put it next to the method people already trust, a moderated focus group, and compared the output.

How we set it up

A customer's agency ran three moderated focus groups with the site's target audience as part of a website redesign, and the customer shared the full transcripts with us. That gave us something rare: an answer key from real people about a live website we could test ourselves. We can't name the site, but the setup is simple to repeat.

1. One benchmark group, two held back. The three groups were recruited from the site's target audience with the same criteria. We used the largest one, with 5 people, as the benchmark, and kept the other two, with 4 and 2 people, aside to check our extra findings later.
2. Personas built from introductions only. We rebuilt each of the 5 benchmark participants as 3 personas, 15 in all, varied by city, age and type of company. Each persona combines demographics, psychographics and beliefs, all drawn only from what people said when they introduced themselves, such as their role, background and history with the brand. Everything they said about the website was withheld.
3. An answer key frozen before the test. We turned the benchmark group's website findings into 10 core findings and 28 single claims. That list, the task wording and copies of the website were locked in version control before the first persona session started. An AI judge proposed matches and a person checked every one. A match only counts if it is the same problem on the same part of the website, and a missed finding counts against us.
4. Tasks that never name a problem. The personas got tasks that mirror the moderator's assignments, such as exploring the offer, getting a free trial and sharing first impressions. No task mentioned clarity, navigation or any other outcome. In all, the 15 personas ran 60 browser sessions. Topics a website visit can't reveal, such as paperwork or events, were left out of the comparison in writing.
5. Validating the extras. Issues that only the personas raised were checked against the transcripts of the two held-back groups, which we never used to build or score the personas. If those people had raised the same issue, it counted as a confirmed new finding. We also checked each one on the live site.

As I like to put it: "We need to follow a control and experiment workflow, to make sure we can interpret result accuracy the right way." The two held-back groups are that control for the extra findings.

What we found

80% overlap, no contradictions

The personas independently found 8 of the benchmark group's 10 core findings (80%). The other 2 were partly covered, and nothing the personas said contradicted the group. In practice, most of the issues you would hear in a focus group room show up in a synthetic run.

40% extra findings that held up

The synthetic runs also raised 11 issues the benchmark group never mentioned. To count one as real, other people had to confirm it: 4 of the 11 also came up in the two held-back groups. That is 40% more findings than the benchmark group produced. Counting the issues we could confirm by checking the live site, it is 70%. This is the part that surprised us most.

That makes sense once you think about scale. Five people can only walk so many paths through a site in one session. Fifteen personas in 60 sessions, each with their own background and priorities, cover more ground.

What didn't hold up: wishes for the future

Not every synthetic insight was confirmed. The ones that weren't had one thing in common: they were about the future. They described what people would like to see in a new design, not what blocks them on the current site.

That gives a clear rule for using synthetic users:

• Lean on them for diagnosing the site as it is: friction, confusion, missing information, trust gaps.
• Be careful with what they say about features or designs that don't exist yet. Treat those as hypotheses, not findings.

The second check we run with every customer

A study is one thing. Every project gets a second, simpler test: we compare our findings with what the customer's team already knows.

When we present results, we ask the team which findings are already on their radar. Typically 15 to 30% of the findings are already in their A/B testing pipeline or match a hypothesis they had. That is a good sign in two ways. It shows the personas pick up what experienced CRO and UX teams have seen in their own data. And it means the other 70 to 85% is new input for the backlog.

ENGIE's review is a good example of what happens next. Their team went through 42 recommendations, implemented 9 right away and sent 20 to further research and A/B testing.

How this compares with other research

We are not the only ones measuring this. Stanford researchers built AI agents from two-hour interviews with 1,052 people. The agents matched the participants' survey answers 83% as well as the participants matched their own answers two weeks later, against 74% for agents built from demographics alone (Park et al., 2024, revised 2026).

Research on surveys points the same way. Across 57 personal care product surveys with 9,300 human responses, AI respondents reached 90% of human test-retest reliability, once they first answered in their own words instead of picking a number on a scale (Maier et al., 2025).

Closer to website testing, researchers built UXAgent, a system in which AI personas browse a real website so a usability study can be tried out before it runs with people. The 16 UX researchers who evaluated it praised the idea and also voiced concerns about how far LLM agents should go in UX research (Lu et al., 2025). That matches how we use synthetic users: for pre-testing, repeat testing and new hypotheses, and as a substitute when there is no time for real users, not as a full replacement.

Critics are right that synthetic respondents can drift toward average answers. NN/g's 2026 study makes that point. That risk is highest for personas made up from a one-line prompt. It shrinks when personas are built from real customer research and act on real pages. We explain how we do that in How we build synthetic users, and the research behind it.

What this means for your team

• Use synthetic users to get most of what a focus group finds, and then some.
• Use them to pre-test, to re-test after every change and to generate hypotheses worth testing. They can run before real users and A/B tests, or stand in when there is no time for real users, but they never fully replace them.
• Trust them most on the site as it is today. Treat future wishes as hypotheses.
• Check findings against what your team already knows. The overlap is your own accuracy check.

Limits of this study

• It was one website and one redesign project. We keep repeating the comparison on new projects.
• The focus groups were small, with 5, 4 and 2 people, and the benchmark came down to 10 core findings. That is normal for qualitative research, but it is not a statistical sample.
• Matching involves judgment. We split findings into single claims, froze them before the test, and had a person check every match the AI judge proposed.