>>>Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty? (@ICSC)
Authors: Satiyabooshan Murugaboopathy | Connor T. Jerzak | Adel Daoud
The research addresses the limitations of traditional poverty surveys, which are often costly and infrequent. The proposed solution involves fusing satellite imagery with text generated by Large Language Models (LLMs) and AI search agent swarms to create more accurate and timely poverty maps. The presentation explores two key hypotheses:
- Platonic Representation Hypothesis: Different data modalities (like images and text) share a common, underlying latent code.
- Agent-Induced Novelty Hypothesis: AI agents add unique, valuable information through dynamic interaction and search.
Key contributions and findings include:
- Fused vision-language models significantly outperform vision-only models (R-squared=0.77 vs. 0.63).
- Evidence of partial convergence in embeddings (cosine sim. ~ 0.60) supports the Platonic hypothesis.
- The release of a new multimodal dataset of 60k DHS clusters.
Tags:
AI, Artificial Intelligence, Poverty Mapping, Machine Learning, Vision-Language Models, Platonic Representations, LLM, Large Language Models, AI Agent, Development Economics, Satellite Imagery, Remote Sensing, Fraunhofer, Computer Science, Research, Global South, Sustainable Development Goals, SDG
Video Transcript
(00:00) [Connor Jerzak] Thank you, everyone, for joining us. We’re here today to discuss one of the new papers coming out of our lab on Platonic Representations for Poverty Mapping. I’m here with Satiya, and we’re going to present this together. So, looking forward to the discussion. I’ll pass the ball over to you, Satiya.
(00:22) [Satiyabooshan Murugaboopathy] Thank you, Connor. The task we are focusing on in our research is measuring poverty and health data in low- and middle-income countries. The task is challenging in itself, but the best way to do it is through traditional surveys, which have the drawbacks of being costly and not being conducted in all places, resulting in infrequent data collection. What researchers have been doing, and very successfully, is using satellite images to predict those scores. But satellite images do not capture, for example, historical or cultural context.
Hence, we thought what could be done is fusing satellite images with textual data that could be sourced from the internet. Two ways of doing that are using an AI agent to perform a web search or using LLM-generated text, where the assumption is that LLMs, having been trained on internet data, have the poverty information imprinted in their weights. We fuse satellite images with these textual data as described and aim to explore whether there is a convergence, such as a platonic representation that can be shown across these modalities, or whether the agent induces novel information for poverty mapping. Fused models outperform vision-only scores, which are, yeah, kind of the state of the art for poverty prediction at the moment.
We outperform it significantly with our fused model, which combines all modalities. Showing that there is information in those texts that hasn’t been captured in the satellite images. We also demonstrate partial convergence in the embeddings between image and text data. The embeddings display a cosine similarity of approximately 0.6. As our final contribution, we also release all the data we generated and crawled as a dataset for over 60,000 DHS clusters.
(03:26) [Connor Jerzak] So just to add a little bit of context here, the ultimate task is predicting this Yᵢ, this poverty variable, and there are all sorts of background factors that influence poverty. Those background factors that influence poverty also affect other things, such as the perception of a particular neighborhood.
So, a rich neighborhood or a poor neighborhood will look very different from the sky, based on roofing quality, home size, and all these other factors.
The relationship between this Yᵢ and Mᵢ is based on all the built environment factors embedded in imagery that have been well studied in the literature. The difference here is this Wᵢ. We have the M, which is the text, the B, which are the background factors, and then this W, which are all the other texts that contain information about these background factors relevant to poverty.
All, you know, descriptions about a place on the internet, all the reviews about the restaurants in the neighborhood, all this large-scale, planetary-scale data source, text data source, is going to be relevant to these places and gets imprinted on the weights of the neural network during training.
It is also imprinted on the weights of the AI search agent that we will discuss in a minute. However, this search agent has an additional ability that the LLM lacks. The LLM has information about each place imprinted on its weights, but the AI agent, as we’ll discuss in a minute, can actually query the internet to learn more about blind spots, learn more about things that the LLM doesn’t understand, to get a better, more holistic understanding of the place.
A moment ago when Satiya was talking about this agent-induced novelty hypothesis, the key question was, is the ability of the agent to gather additional texts to address blind spots that it has, is that injecting an additional layer or level of novelty compared to just looking at imagery alone or compared to just looking at the imprinting of these texts, of these place-specific texts on the weights of the neural network. And then these W’s on the bottom, those are just a few different ways that we’re going to try to squeeze juice out of the weights of the LLM. We’re going to ask the LLM to describe the place. We’re going to ask the AI search agent to provide a summary of everything it read, and then we’ll also examine the traces — the whole context that the search agent saw when scouring the internet for relevant information about this place.
(06:05) [Sathiyabooshan Murugaboopathy] Yeah, great. Thanks for the context, Connor. So our two hypotheses that I described briefly at the beginning are, of course, either the platonic representation hypothesis, which has been proposed in the literature, where the assumption is that all the modalities converge to each other, meaning that they have a, yeah, shared representation kind of just encoded into different modalities, maybe. And the alternative hypothesis we’re investigating is whether the agent induces novel information. So, our search agent, which can reason independently and think, whether the agent then adds unique structures, which then induces novelty to help and aid the prediction of the scores.
(07:11) [Connor Jerzak] One of the co-authors on this paper used the analogy of the five senses when discussing the Platonic representation hypothesis. That co-author is Adel, who’s unable to join us today. But Adel was using the analogy to the senses. So we humans have five senses. We have sight, touch, smell, and these other senses. And these senses characterize different aspects of our environment. The Platonic representation hypothesis suggests that, in some profound way, all these senses correspond to the same thing.
And with enough data, these senses will, in some way, overlap, or the information generated from them can be aligned with a better and better approximation. That’s the Platonic representation hypothesis. And as Satiya was saying, the agent, if the platonic representation hypothesis doesn’t hold at least in the limit, that would suggest that we really need five unique senses. We’re asking, is agentic ability almost like a sixth sense? A sense that can’t be reduced to the others, or is it reducible? That’s what we have there. Now, let’s just recap what we’ve already talked about. The dotted arrow allows us to test this agent-induced novelty hypothesis. Now, we’ll move on to some of the prompts that we used during the experiments we’ll discuss in a minute.
(08:41) [Satiyabooshan Murugaboopathy] We tried to use as similar prompts as possible for both of the approaches, just to aid comparability, like the LLM-only pipeline and the AI agent pipeline. The given information remains the same. We then input a geocoded address, along with its coordinates, latitude, longitude, and the year. We use geocoded addresses because the literature has shown that LLMs, in general, can process addresses more effectively than just using coordinates, which makes sense. So, both of the LLMs or the LLM and the AI agent are both told to then either search the web for enough content on these given informations or just use their own knowledge that they have been building up during their training phase and then give a description in the case of the LLM doing the neural memory reconstruction as we call like the using the own knowledge and and then we we just get out of the LLM pipeline a justification why it thinks this location has this and this score and this is the description that we then use for for the location.
We also obtain a predicted value for both pipelines, along with a confidence level, which we have been investigating; however, there hasn’t been much information that can be gained from the confidence values. And so, if we go to the AI search agent, the only thing that differs is that it searches the web instead of relying solely on its own knowledge. You can see that it’s very similar in these two prompts that we show here.
We then attempt to generate one justification, which would be the summary that we mentioned, namely the summary that the AI agent builds based on the search results it has been gathering. Additionally, it predicts a value for the location and provides a confidence score. And for the other approach, where we use the traces purely, we simply ignore this output, including the justification, predicted values, and confidence, and instead use the search traces the agent has been choosing for this task.
(11:37) [Connor Jerzak] So, a trace, for those who aren’t familiar with the term, is basically the whole set of information that the AI search agent was able to see, was able to examine during the course of the agent’s investigation. So that’s what a trace is here. A question for you, Satiya, going into the project: which of these two approaches did you think would give better results? Which of these two approaches did you think was going to yield more accurate predictions for continental-scale wealth measures?
(12:04) [Satiyabooshan Murugaboopathy] I mean, I think we were all thinking that the AI agent would actually perform way better because it’s just like, if you think of it, the AI agent is the LLM NMR boosted with search. So, my intuition was that the boost or additional data, along with the ability to search like the web and utilize tools, should give it an edge over the LLM NMR, which relies solely on its own knowledge.
(12:38) [Connor Jerzak] The same, I thought the same. You give, and we’re using the same LLM backbone here. The only difference is that one is empowered with agentic abilities, and the other isn’t. So, you might think that more is more, but we might find something a bit different. So stay with us as we get closer to the results. This here is a visualization of the agentic workflow. For this workflow, we utilized LangGraph, a great open-source tool that I highly recommend. We should also include links to our code and data in the description.
This is the workflow where we have an LLM acting as the brain or orchestrator for this task. So the input was the prompt we just had. The prompt is basically, give, we give the the agent a place and we give the agent access to different tools, web search and Wikipedia search. We give it a year, and we basically say to search the internet and Wikipedia for as much information as you can find relevant to the International Wealth Index. We’ll also discuss information leakage later on, but this was the task, and this is just a visualization for the LLM, which can utilize various tools. In this case, the two tools are web search and Wikipedia search, which are operationalized as two distinct tools here.
The LLM can act by performing searches on those different tools, retrieving results and observations, and then taking additional actions on the same or different tools. We set the total number of recursive tool call episodes to 20 in these investigations. However, we found that in practice, the agent concluded its search after at most three different iteration cycles of searching, obtaining results, and then searching again. So, we’re actively thinking about how we can use not only one agent but also multiple agents in more sophisticated chains of agentic tool calling in future research. However, you’ll have to wait until our next paper to learn more about that. This is a brief overview of our agentic workflow, and now we’ll discuss the three resulting data assets that are hosted on Hugging Face. I’ll throw the ball back to you, Satiya, to talk a bit about this.
(14:52) [Satiyabooshan Murugaboopathy] Great. The data we used is based on the DHS surveys that we have, as they are our primary source of ground truth data. And we’ll come to the problems with that later as well. However, our data set is built around over 60,000 DHS clusters, where we have survey data on the poverty score assigned to each neighborhood. This is a trend that has been prevalent across Africa, spanning from the 1990s to the 2020s.
(15:28) [Connor Jerzak] Just to two-finger on that. That involved people going to neighborhoods and asking hundreds of questions of citizens across dozens of countries on the continent of Africa. So, you can imagine the cost and scale involved in gathering information about household wealth in these contexts. This is a massive amount of work, involving a substantial amount of capital, with literally hundreds, tens, if not over the long term, hundreds of millions of dollars being spent to obtain the ground truth data from these DHS surveys. And then that leads us to the satellite images as well. So, Satiya, can you discuss how satellite images are used to explore platonic convergence?
(16:10) [Satiyabooshan Murugaboopathy] We need those because that is our second modality, right? We have the textual modality, and we also test the convergence based on the comparison between those two textual modalities. However, the most interesting one is probably the comparison of visual data against textual data. There is also a huge difference in how we perceive the information, particularly if you consider each modality as just a representation of poverty. For example, the representation between satellite images and LLM-generated text is very different. And what we then try to test is whether these representations are actually showing the same or whether they may have their own filter, kind of like a context that each modality cannot see, where the other one provides more insight.
(17:30) [Connor Jerzak] Great.
(17:31) [Satiyabooshan Murugaboopathy] Yeah. So let’s dive into the implementation. We have those different pipelines. We have the visual-only pipeline, which has been successfully produced in the literature by many researchers. That is, essentially, our baseline. That was the one we were trying to beat, and we successfully did so as well. The second pipeline utilizes LLM knowledge only, which is our LLM neural memory reconstruction.
But back to the visual model. We take the satellite images and use a pre-trained model that has been proven effective. A visual transformer encodes those images and generates embeddings, which we then process to predict our scores. And the LLM-generated text is then also encoded, and so those embeddings are also the ones that we will compare against. So, embeddings produced either by a vision transformer or a transformer-based encoder model for the textual data.
And so we have the two text-only pipelines. One is the LLM NMR, and the other is the AI search agent, where we used the complete traces because that was the best-performing version of the AI agent, rather than using, for example, the summary that the AI agent produced based on its search traces. We also investigated the joint performance of each. So, LLM NMR with vision, AI search agent with vision, and the fully combined version, where we utilize all three modalities to predict an IWI score altogether.
(19:54) [Connor Jerzak] Great. To also investigate our approach and evaluate it properly, we decided on three splits. We took a random split, but also an out-of-country split where countries where our models were trained on weren’t in the test split and vice versa. And the out-of-time was year-wise, so we never had years in the test split that were also in the train split.
And of course, we did all these through 100 iteration bootstraps. For the more intense trainings, we conducted cross-validations. The metrics we were checking were R-squared and RMSE, as they are widely accepted in the literature for poverty prediction, especially R-squared. Additionally, to compare the embeddings I mentioned, we utilized two different embedding models for the textual data.
For the visual data, there was no significant need to use different models, as a standard vision transformer was already accepted for satellite images.
However, for the textual data, we utilize a frozen OpenAI embedding model, which is quite large, and also a fine-tuned open-source embedding model, which is relatively small, to compare whether there are significant differences between different embedding models or whether the textual data performs similarly across various embeddings.
(21:49) [Connor Jerzak] For those who aren’t familiar, frozen means we don’t modify the weights of the neural network. Here, the OpenAI embedding model, at least as of mid-2025, is a closed-source model. So we don’t have direct access to the weights on, for example, Hugging Face. However, for MPNet, being an open-source model, we can modify the weights in any way we like. We modify the weights on the training set to generate the embedding structure that yields the most predictive poverty prediction or the most predictive output from that approach. So that’s how we’re fine-tuning. We’re fine-tuning on the DHS label within a training set. Do you have any other details about the evaluation, Satiya?
(22:42) [Satiyabooshan Murugaboopathy] No. If you have any questions, please don’t hesitate to ask. We can encourage a discussion, but otherwise, I guess, yeah, that’s it.
(22:52) [Connor Jerzak] All right. Let’s get to performance. What do we find?
(22:55) [Satiyabooshan Murugaboopathy] This is the most interesting part. The most shocking result for us is that we could beat the vision-only baseline by such huge margins, and that the LLM, without combining with anything, did beat the vision-only baseline that we use. And if combined with visuals, it was even better. It’s very interesting to see that, even for each combination or modality added to the fusion, the models perform better and better. You can see that the AI search agent, very surprisingly, could not match the performance of the LLM NMR. The AI search agent performed best on its own on the search traces and complete traces.
And hence we only used that version of the AI agent for our fusion experiments. The same applies to the LLM NMR, where we only used our best-performing approach, which was LLaMA for Maverick. And we we then inspected the performance of that fused with the vision-only and with the with the traces. So I think many surprises here. AI search agent not performing as good as the LLM NMR and LLM NMR performing even better than vision-only is I think very very interesting. The embedding models show very consistent results. So, the MPNet, being significantly smaller than the OpenAI embedding model, performed less well; however, it still shows similar comparative performance.
(25:05) [Connor Jerzak] Those were some of the surprises. Another thing that wasn’t surprising, I guess, was that there were tons of surprises. The one thing that wasn’t surprising, in addition to the larger OpenAI embedding model performing better, was that the kind of neural reconstructions from larger models generally perform better than those from smaller models.
The LLaMA-4-Maverick model, which was the largest examined here, yielded the best neural reconstructions compared to the other approaches that utilize smaller LLMs. So that would be another finding from this. However, the big picture is that there is a boost. We find that there’s an additional signal that we gain when we use these different modalities and combine them together. Another significant outcome is that the agent appears to become confused by the lengthy context.
There is recent work emerging about how very long reasoning traces can impair model performance. And I think, that may be what’s happening here where the neural network that we’re using does know a lot about the places that it’s being asked to give information about, but it’s also getting a bunch of other information from the search traces, some of which is relevant, but some of which isn’t relevant, and that additional context seems to be in a sense overwhelming the neural reconstruction abilities and actually reducing performance on a relative basis. Although there is an overall gain over the image-only baseline.
(26:48) [Satiyabooshan Murugaboopathy] So we also investigated how our models performed for different regions, kind of. We compare the residual values, specifically the absolute mean residual for each neighborhood, with the vision-only model and then the best-performing combined model. So just to see, like CV being our, so computer vision being our baseline and we comparing, “Okay, where does textual data seem to add the most of information and hence performance gains?”
And I think what can be seen is that very, like conflict areas and also dense areas seem to be where textual data helps the most because, maybe it’s maybe the the textual data could be more more fine-grained than the satellite images in that case. But we can also see that in many other areas, they are actually performing very, very similarly. There is no significant difference, no huge difference, between the vision-only and the combined performance. For example, if we look at the Sahara region, we can assume that satellite images struggle a lot because the images are very uniform. Hence, the textual data provides additional context. For both extremes, if the data is too homogeneous or too heterogeneous, then the textual data seems to add an edge. Additionally, there are dynamic places, such as conflict areas and similar locations.
(29:08) [Connor Jerzak] For those trying to make sense of this plot, red means the combined approach wins. Blue means the CV-only approach wins. Overall, we see that there’s more red than blue. However, as Satiya points out, there are some places where the difference is not significant. We see a gray color, indicating that the two are the same. but those regions that are more red, meaning the combined agent-text approach with the imagery is doing better, that’s in Somalia, Yemen, in the Central Africa region, both of which, both of these areas are affected by conflict, as well as this South Africa core down here in the south. so definitely some really interesting spatial insights here. However, we will now discuss the temporal insights. So what have we learned about the temporal dynamics here, Satiya?
(29:57) [Satiyabooshan Murugaboopathy] I think the temporal dynamics are very interesting because we can probably also gain some insight for the leakage issues that we will talk about later. However, what we can see here is again the residual difference between the CV-only and the vision-only models, and compared to the combined version, the best-performing model. And so, if the score is high, then the combined approach is one; if it’s low, then it’s very similar. And we can see that it consistently outperforms the vision-only model.
Therefore, there is no year in which the vision-only model outperformed the combined approach. However, surprisingly, especially in the 90s, they performed very well. On the one hand, there is the issue that in the 1990s, satellite images were scarce. Perhaps that is one reason why the textual data could have made such a difference in that time period. But on the other hand, it’s very interesting because one could argue that the information density that the LLMs were trained on actually was way higher in the 2000s, 2010s, up to the 20s because the internet was booming in those years, right? In the 90s, it was on the rise, but not yet as prominent as it became in the 2000s and 2010s.
(31:43) [Connor Jerzak] It is interesting. We might find significant gains because we’re performing better, or we might find significant gains because the combined computer-only and computer vision-only approach is performing significantly worse. And I think the hypothesis here is that the gains from the 90s are due to the computer vision approach performing significantly worse, largely because of image scarcity issues and Landsat privatization, among other factors, that occurred in the 1990s. So we have this data scarcity issue there. So interesting stuff, Satiya. However, let’s now move on to the Platonic representation hypothesis. The title of the paper, after all, you know, talks about that. So what do we find about the convergence of modalities?
(32:29) [Sathiyabooshan Murugaboopathy] Yeah, I mean, one of our most surprising findings was, of course, that the LLM NMR was beating the AI agent by quite a margin. This supports the Platonic hypothesis more than the AI agent-induced novelty, because the AI agent, presumably, couldn’t introduce so much novel information that significantly aided the prediction performance.
And also, like as I mentioned in the beginning, the convergence of the different modalities was quite high, so it it had a cosine similarity of 0.6 for for the if we compare those embeddings which are, yeah, in the same dimension and we also align them with canonical correlation analysis, just to, to show more realistic results, not just using the embeddings as they come out of the different embedding models, but aligning them through CCA. The search agent was also similar, much like LLM. So, so I guess, yeah, Connor, what do you say? Do, is the AI agent, did the AI agent induce some novelty, or what are your thoughts on that?
(34:04) [Connor Jerzak] So we saw that the best-performing approach did combine the computer vision, the neural memory reconstruction, plus the agent-agent traces. So, from that sense, perspective, there may be a degree of novel information contained within the agent traces. Although there are marginal gains, and these cosine similarities within aligned representation space across modalities are pretty high, you know, 0.6 cosine similarity. My sense is that, overall, there’s strong evidence for platonic representations here and weak evidence at best for agent-induced novelty.
One thing I will say is that the agent traces are long; those contexts are also quite lengthy. Therefore, more sophisticated summarization strategies might extract better information from those agent traces to, in a sense, emphasize or highlight the unique or novel portions of the agent traces. So I think there’s definitely a lot of work to be done on reinforcement learning approaches towards agentic data extraction that we can pursue in future work. However, broadly speaking, I would say we have more support for platonic relationships over novelty. Future work will likely answer that question more conclusively when we incorporate more sophisticated data extraction and information retrieval mechanisms into the agent pipeline.
(35:34) [Satiyabooshan Murugaboopathy] I mean, we already dove into the limitations kind of, right, during our talk. We have issues with the DHS clusters being our main ground truth, and that’s how our entire data structure was built around those clusters. These surveys, of course, are taken or attempted to be taken in a way that captures the continent, but they cannot capture all of it. Therefore, there may be some bias in using only the clusters and neighborhoods that the DHS used, and we may overlook other underrepresented neighborhoods by relying solely on those data points.
(36:40) [Connor Jerzak] So that would be really rural places. In reality, rural places aren’t included within the sampling frame of the DHS. So we’re not capturing those within our approach here.
(36:50) [Satiyabooshan Murugaboopathy] Exactly. Additionally, our entire pipeline has the potential to be extended significantly. So we, of course, can scale the agent a lot and also we can dive into the text processing more so that we maybe have a text processing method that performs equally good on the shorter descriptions generated for example by LLM NMR as it does on long texts, for example, concatenating all the search traces that the AI agent yielded because that inequality is existing in our embedding models because they are, as Connor mentioned, said to be sensitive to to very very long contexts and andare said to be underperforming um in those long-context data samples. So what do we want to do? What can be taken out of our research?
I believe we demonstrate that multimodal fusion significantly enhances poverty monitoring. I think that’s the most surprising, yet also the most important, thing we learned from our research. Then we could also answer the question or the hypothesis that we posed at the beginning and show some partial convergence that would support the Platonic hypothesis. The most surprising learning is that our LLM NMR outperforms the AI agent. Although we all already mentioned the issues that could be underlying.
However, for now, with this pipeline and this relatively cost-efficient method for processing the texts, we demonstrate that the LLM NMR outperforms the agent and yields excellent results on the prediction data, particularly in the prediction task. This can be taken into account for future investigations. As we mentioned, the agent could be improved, but the LLM NMR could also be improved. Additionally, we provide our dataset. So that you can take a look at it and come up with new ideas, which you can use for your own research and hopefully yield exciting results.
(39:27) [Connor Jerzak] All right. Thank you, Satiya. Thank you all for listening. If you have any questions, post a comment; reach out via email; take a look at the paper; check out aidevlab.org.