Section 01
The Problem Simile Solved — and the Constraint We Faced
In their confidence model post, Simile describes the core problem of simulated research perfectly: a simulation can be accurate on average while being wrong on any individual question. Their answer was to train a model — probing hidden states, fine-tuning on error labels — to predict the error of each simulation before you see the ground truth.
That approach requires something most teams don't have: thousands of labeled questions with known human distributions to train against. We're a small team shipping weekly. So we asked a different question: how much can you learn about a simulation's reliability from the simulation itself, with zero training data?
The answer turns out to be: quite a lot. Not as much as a trained confidence model — we're honest about that — but enough to catch the failures that actually burn founders: unstable panels, degenerate distributions, and questions where the persona sample fell apart.
Section 02
The Math: Split-Half TVD, Explained Simply
Total Variation Distance measures how different two probability distributions are. A TVD of 0 means identical; 1 means completely disjoint. Simile uses it to compare a simulation against real human data. We use it differently — to compare the panel against itself.
TVD(P, Q) = ½ Σi |Pi − Qi|
- Split the panel in two, randomly. 100 personas become two groups of 50.
- Aggregate each half into its own opinion distribution. What did group A say? What did group B say?
- Measure TVD between the halves. If random halves of the same panel produce nearly identical distributions, the panel is stable.
- Repeat 20 times with seeded shuffles and average. Deterministic seeding means Inngest retries produce identical scores.
The intuition: if you ran your survey twice with a fresh random draw of personas, would you get the same answer? Split-half TVD approximates exactly that — without running anything twice.
Input
- Panel
- 200 personas, one question
- Method
- 20 seeded random half-splits
- Measure
- TVD between halves' aggregated distributions
- Runtime
- Milliseconds — no LLM calls
Output
Illustrative stability levels. A split-half TVD of 0.05 means random halves of your panel agree within 5 percentage points — the sampling noise is tiny relative to the 0.16 decision-grade threshold.
Section 03
What the Score Captures — and What It Honestly Doesn't
We ship four signals per question, computed in milliseconds inside the analytics pipeline:
| Signal | What it tells you | Red flag threshold |
|---|---|---|
| Split-half TVD | Sampling noise — would a re-run agree? | > 0.16 |
| Bootstrap TVD | Resample spread — a second stability view | Compared against split-half |
| Missing ratings | Did every persona actually answer? | > 10% missing |
| Degenerate distribution | Is one answer option at 95%+? (suspicious unanimity) | > 95% single bucket |
| Low-quality reasoning | Empty answers or near-empty reasoning | > 20% of responses |
Section 04
How This Compares to Simile's Approach
Simile's confidence model is the state of the art: AUROC 0.74, correlation 0.57 between predicted and actual error, trained on ~8,600 held-out questions. We're not going to pretend a heuristic matches that.
Simile's published numbers vs. our zero-training approach. Their method needs thousands of labeled questions; ours ships today on every survey. Our benchmark program is how we close the gap.
But there's a trade Simile doesn't have to make and we do: their confidence model runs on their own foundation model's hidden states. Our pipeline runs on multiple LLMs (Kimi, DeepSeek, GPT-OSS via Cerebras and BaseTen) — a hidden-state probe doesn't transfer across providers. Split-half TVD is model-agnostic by construction. It works the same whether the response came from Kimi K2.5 or DeepSeek-V4.
FAQ