Platonic Representations for Poverty Mapping: Do Images and Language Share a <Wealth Code>?
TL;DR. We paired ~60k African DHS clusters with satellite imagery and two flavors of place‑specific text—LLM‑only spatiotemporal descriptions (“neural memory”) and web text fetched by an AI search agent—and asked whether these independent channels converge on the same latent signal of material well‑being.
Fusing vision with LLM text explains about 77% of held‑out International Wealth Index (IWI) variance on random splits (vs. ~63% for vision alone). The LLM‑only text is stronger than agent‑retrieved text; adding the agent’s material to the fused model yields only a small, inconsistent bump. Embeddings align moderately across vision and language (median cosine ≈ 0.60 after alignment), which looks like a shared code of wealth, with each modality contributing residual detail.
Why this problem, why now
Most poverty mapping has been limited to what satellites can see: roads, roofs, fields, and the morphology of settlements. That view is powerful but incomplete. Places also generate linguistic residue—administrative descriptions, historical notes, development reports, stray facts on the open web. Modern language models compress a great deal of that prior into their weights; retrieval agents can try to bring in timely context. Our study asks whether these two worlds—pixels from orbit and narratives about place—gravitate toward the same representation of material well‑being, and whether an active agent adds meaningfully novel structure beyond what an LLM already “knows”.
Specifically, for each DHS cluster (Africa, 1990–2020; ≈60k), we assembled a cloud‑free Landsat composite, an NMR description from an LLM prompted with the cluster’s year and coordinates (no tools), and an ASA trace from a LangGraph‑based agent using the same LLM core to search, extract, and summarize web text. We embedded text either with a frozen 1536‑d model or a fine‑tuned MPNet, encoded imagery with a Landsat‑pretrained ViT (frozen), and trained ridge regressions on individual and concatenated embeddings. We evaluated three regimes—random, out‑of‑country, out‑of‑time—using bootstraps for frozen setups and 5‑fold CV for fine‑tuned ones, reporting R² and RMSE on held‑out splits.
Short version of the results
Fusing modalities is reliably better than betting on just one. On random train/test splits, a simple ridge on top of frozen image and text embeddings reaches R² = 0.765 for NMR+CV; adding the agent’s text yields 0.772. The vision‑only baseline sits at 0.634, while NMR (LLM‑only text, from year+coordinates) reaches 0.668, reflecting strong internal priors. Generalization across borders is hardest: in an out‑of‑country regime, CV drops to 0.446, NMR to 0.440, and NMR+CV to 0.527 (the agent nudges this to 0.529). Out‑of‑time shifts are gentler, with NMR+CV at 0.700 and the full NMR+ASA+CV at 0.694. Text embedding choice matters in practice: a frozen, high‑capacity embedding (OpenAI’s text‑embed‑3‑small) generally does better than our fine‑tuned MPNet on random and OOT splits; OOC is tighter and occasionally flips the ordering by a hair, but the fused picture remains the same.
On random splits, CV alone explains 63.4% of IWI variance, NMR explains 66.8%, and ASA (CleanedTraces) reaches 60.6%. Fusing NMR+CV reaches 76.5%, and NMR+ASA+CV edges up to 77.2%. Crossing borders is the stress test: CV falls to 44.6%, NMR to 44.0%, NMR+CV to 52.7%, and NMR+ASA+CV to 52.9%. Time shifts are kinder: CV at 56.9%, NMR at 60.9%, NMR+CV at 70.0%, NMR+ASA+CV at 69.4%. Confidence intervals are tight on random splits and widen on OOC, as expected. Full tables with RMSEs and CIs are attached.
What changed our minds
We expected the search agent to shine by injecting fresh, local detail. In reality, retrieval added a sliver of signal only some of the time. The LLM’s internal spatiotemporal priors—conditioned solely on location and year—consistently outperformed the best agent‑only setup, and most of the performance gain came from fusing that LLM signal with imagery rather than bolting on more text. It’s not that the web is useless; it’s that noisy retrieval and bloated contexts don’t automatically beat the distilled regularities already baked into the weights.
If a single latent variable—material well‑being—drives both the physical imprint of neighborhoods and the way they’re talked about, independently‑trained encoders should rhyme. After alignment, NMR↔CV embedding pairs for the same cluster show a median cosine similarity around 0.60 (a large, highly significant shift from null), suggesting a shared “wealth code”. The fact that fusion still beats either stream alone implies that this code doesn’t collapse modalities into each other; each keeps some unique signal, which is exactly what we want in a practical predictor.
Practical implications
If you need a fast, scalable baseline, start with NMR: one LLM pass per location‑year already captures much of the action and delivers R² ≈ 0.67. If your goal is robustness, especially under temporal shift, add imagery; NMR+CV delivers the biggest and most consistent gains across splits. Treat retrieval as optional seasoning rather than the main course—use it when you can keep the summaries focused and time‑stamped, not as an excuse to paste in more text.
Borders remain the difficult case. The OOC penalty hints that country‑specific asset patterns, roof materials, street layouts, and institutional context are not fully interchangeable; light regional adaptation or hierarchical pooling may help. Text also carries post‑treatment risk: it is too easy for agent traces to smuggle in facts that occur because of poverty changes rather than predict them. And while multimodality helps most in the 1990s—precisely when imagery coverage is thinner—that is also where uncertainty is largest, so calibrated intervals matter.
We are releasing a large‑scale multimodal corpus—~60k clusters with Landsat images, LLM‑generated descriptions, and agent traces—keyed to IWI labels. To link against full DHS microdata, you’ll need to register with DHS and accept their terms. The repository includes code to reproduce the splits, embeddings, and evaluation harness so you can sanity‑check, swap components, or plug in your own encoders.
A few reasonable questions
Isn’t the LLM just memorizing? It certainly leverages broad priors about countries, cities, and eras; that is the point of pretraining. But memorization alone can’t explain the observed cross‑modal alignment with satellite features, the durability under out‑of‑time shifts, or the additive gains from fusing image and text.
Why didn’t the search agent do better? Retrieval is noisy, and long, loosely filtered contexts dilute rather than concentrate signal. The weights already encode the stable regularities of the domain; the agent helps when it adds focused, local information that the LLM is unlikely to have internalized, but that is rarer than one might hope.
Should we chase bigger models? Larger models helped, but the cost curve matters. In our tests, a very large LLM led the pack, yet smaller models were competitive. For deployment, moderate‑sized LLMs paired with imagery often hit the right balance of accuracy and cost.
Limitations and what’s next
Our outcome is predictive, not causal, and text is especially prone to post‑treatment leakage; we recommend explicit timing controls and provenance filters for any causal use. Country‑specific heterogeneity suggests room for hierarchical models and light regional finetuning. Representation diagnostics could go deeper: object‑level attributions for imagery; counterfactual editing for text; and finer‑grained spatial analyses of where alignment fails. On the agent side, we suspect stricter summarization and source selection—aimed at lowering entropy, not inflating context—will be more valuable than additional search steps.
If you’re here to build something
A pragmatic recipe is simple. Use NMR when you need fast continental coverage; it’s cheap and gives strong performance. Add CV when you care about temporal robustness or can afford vision processing at scale; the fusion gives you the biggest lift. Log how any text was obtained and when it refers to, and keep out‑of‑country and out‑of‑time evaluations sacred—your real‑world error will look more like those than your random split.
Acknowledgments. This post summarizes “Platonic Representations for Poverty Mapping: Unified Vision–Language Codes or Agent–Induced Novelty?” The figures in the paper (test-split scatter, residual maps, and similarity matrices) provide the full visual story; the accompanying tables include R², RMSE, and 95% confidence intervals across all settings. If you use the dataset or code, please cite the paper and respect DHS data policies.