Methodology · v0.1
How we build
the common world.
We ask the same questions to the leading LLMs, measure how they answer, and surface where they agree and where they diverge.
- ROUNDS
- —
- RESPONSES
- —
The journey
From a question to consensus.
- 01
Fact
A question that divides how the world is told
- 02
Round
We ask every model the same prompts
- 03
Response
Responses are collected in English
- 04
Metric
We extract evasion, framing, tone
- 05
Consensus
We measure how much models agree
Concept · 01
Responses
The current corpus is English-only: each round, we ask every model the same prompts in English. We store the raw text exactly as the model returned it — no edits, no cleanup.
Concept · 02
Metrics
We extract traits from each response: evasion type, framing, tone, and token counts.
Concept · 03
Pair consensus
We measure semantic similarity between every pair of responses using embeddings and cosine similarity.
Concept · 04
Fact consensus
When models broadly agree, we show a ranking around the common centre. When they split into camps, we show factions instead.
Ranking mode (cohesion ≥ 0.70)
Factions mode (cohesion ≤ 0.40)
Concept · 05
Alignment
We calculate how close each model sits to the common centre (or to its faction).
Dots represent models; centre circle is the common centroid
The formulas
Formulas & definitions
Pairwise similarity
similarity = (cosine + 1) / 2Raw cosine similarity in [-1, 1] is remapped to [0, 1] for display.
Calibration
calibrated = (raw − baseline) / (ceiling − baseline)Linear remap to [0, 1] based on empirical corpus percentiles. Current values: baseline = 0.45, ceiling = 0.92, embedding model = openai/text-embedding-3-small:2024-02.
Clustering
average-linkage agglomerative clusteringGroups responses into factions using hierarchical clustering with a configured threshold of 0.45.
Consensus score
consensusScore = agreementRatio × cohesionMeanAgreement ratio is the largest faction size divided by total responses; cohesion is the mean intra-faction similarity.
Polarization
polarization = separationMean × (1 − agreementRatio)Measures how distinct factions are when models split into camps.
View mode
ranking | mixed | factionsDetermined by cohesion thresholds: ranking if cohesion ≥ 0.7, factions if cohesion < 0.4, mixed in between.
Global centroid
globalCentroid = median(modelValues)The median of all model values when view mode is not factions; null otherwise.
Alignment band
aligned | edge | minority | unaffiliatedClassifies a model's position relative to the dominant faction centroid. Aligned if distance ≤ 0.05, otherwise edge.
Cross-fact similarity
mean of per-fact pair similaritiesAveraged across facts where both models responded; coverage ratio indicates shared fact proportion. Low-coverage threshold: 0.5.
Model similarity
How models compare
We compute pairwise similarity for each fact from the models' answer embeddings, then average across the facts where both models responded. Coverage ratio indicates the proportion of shared facts, and pairs below 50% coverage are treated as unreliable.
Every pair of models on one interactive heatmap — grouped into clusters, with the most similar and most divergent pairs called out.
Open the matrix →Glossary
Every term, one place.
- Fact
A question that divides how the world is told.
- Round
A batch where the same prompts are asked to every model.
- Response
What a single model returned, stored verbatim in its original language.
- Metrics
Traits extracted from each response: evasion, framing, tone, token counts.
- Pair consensus
Semantic similarity between two responses (cosine similarity on embeddings).
- Fact consensus
How much all models agree on one fact.
- Alignment
How close each model sits to the common centre (or to its faction).
- View mode
When cohesion is high we rank around a centre; when low we show factions.
- Round consensus
One-number summary (0..1) of how much the system agreed in a round.