The problem
The Recommendation–Reality Gap
Generative engines recommend products, restaurants, services, and decisions constantly. Users act on those recommendations. Sometimes it works. Sometimes it doesn't. No one is systematically measuring the gap.
Current GEO tooling focuses on citation share and LLM visibility - whether a given entity appears in a model's output. What it doesn't measure is whether the recommendation actually held up when someone acted on it.
The RRG project documents that gap. The core insight that makes it tractable: the behavior we want to measure - frustration when misled, vindication when right - already occurs constantly across social platforms, unprompted. We only need to detect it and convert it into a calibrated signal.
Design principle
Real-world recommendations
We focus on restaurants, travel, products, and services. Coding, debugging, and writing examples are kept in a separate archive. A request such as restaurants near Gion, Kyoto is a category; the actual restaurant name is the entity.
Reports record the AI product and the exact model version only when the original post supplies it. Unknown versions stay unknown. Published submissions are labeled unverified until their source has been checked. Illustrative seeds are excluded from analysis.
The system is self-contained: no incentive scheme or task design is required to generate the underlying behavior. People report on AI recommendations against reality because the experience provokes a reaction - not because a researcher asked them to.
This makes the ledger evaluable (detection precision and recall can be measured against hand-labeled samples), honest (no causal claim about GEO, only a documented gap), and resistant to the failure mode that kills community efforts: the manufactured task.
Self-contained
No incentive scheme needed. The behavior exists.
Evaluable
F1 can be measured against hand-labeled samples.
Honest
Documents a gap. Makes no causal claim about GEO.
What we build
The Reliability Ledger
The core output is a per-entity, per-model reliability ledger - not a single global score. The framework covers four dimensions. The pipeline includes Phase 2 calibration; this public view currently shows individual reports and raw counts.
| Dimension | Measures |
|---|---|
| Recommendation share | Frequency an entity is recommended for a query category |
| Outcome polarity | Self-reported result when someone acted on the recommendation |
| Calibrated gap score | Weighted difference between recommendation share and outcome polarity |
| Coverage confidence | Sample size and platform diversity backing a given score |
Phase 2 implements confidence and popularity weighting, Bayesian smoothing, and bootstrap confidence intervals. Platform base-rate correction is not yet implemented. These adjustments do not eliminate selection bias or establish causal model reliability.
Research roadmap
Phases
HN pilot reported relevance F1 0.88. The 91.2% agreement is single-reviewer agreement, not blind inter-annotator agreement. Reddit and Xiaohongshu validation remains open.
Target: Classifier F1 > 0.75 on held-out annotation
Confidence and popularity weighting, Bayesian smoothing, and bootstrap intervals implemented. Platform base-rate correction remains open.
Target: First defensible per-entity, per-model scores for a small category set
Ledger interface implemented. Collect forwarded, existing recommendation posts across platforms; no new scraping work planned.
Target: Ledger live; cross-platform coverage confirmed
Taxonomy, comparative, and cross-lingual analyses. Paper + dataset.
Target: Paper submitted to NLP/ML venue; dataset release
Open questions
What this doesn't resolve
Causal confound
Gap scores reflect staleness, genuine quality mismatch, and GEO manipulation together. The system cannot disambiguate these on its own.
Platform sampling bias
Which communities vent on which platforms is non-random and should be acknowledged, not silently corrected.
Cold start and stickiness
The registry needs enough credible coverage early to become a destination before momentum stalls. This is the project's central execution risk.
Dispute and moderation
Required before any entity-level score is surfaced publicly, to manage legal exposure.
Platform terms of service
A sustainable crawling strategy for Xiaohongshu and WeChat without violating platform terms.
Positioning
Academic, community-framed
The project is positioned as an academic research program rather than a commercial GEO product. It is intended to draw researchers and curious contributors rather than competing GEO vendors. The competitive break is shifting from a profit-driven optimization product to a credible, community-driven behavioral study.
The durable advantage is not the technical pipeline, which a well-resourced competitor could replicate. It is being first to accumulate a large, credible, structured dataset. The project's central risk and its central asset are the same dynamic.
Target output: a system-and-empirical-contribution paper for an NLP/ML venue, with the calibration framework as the core technical claim. Research at the DREAM Lab, UIUC, under Prof. Haohan Wang.