Goodbye A/B Testing: LLMs Simulate Real Consumer Responses

Directly asking LLMs for numerical ratings (like purchase intent on a 1–5 scale) usually produces unrealistic, overly positive, or narrowly distributed answers that don’t match real human data.

This paper introduces Semantic Similarity Rating (SSR)— a simple method that first lets the LLM generate free-text reasoning about a product, then maps that text to a realistic Likert distribution using embedding similarity to anchor statements.

Validated on real data from a major personal care company (57 surveys and 9,300 human responses), SSR achieved:

•⁠ ⁠Strong distributional similarity to actual human responses

•⁠ ⁠~85–90% of human test-retest reliability in mean purchase intent

•⁠ ⁠Rich qualitative explanations alongside the ratings

The method requires no fine-tuning and significantly outperforms both direct numeric prompting and follow-up Likert mapping.

This Ledger Entry expands how readers think about using LLMs for synthetic data and consumer research by showing that simple prompting for numerical ratings produces poor, unrealistic distributions, while eliciting free-text responses and mapping them via embedding similarity to Likert anchors yields significantly more realistic and useful synthetic consumer insights — without any fine-tuning.