Speaker: Isaac Corley

Transcript: I’m Isaac. I just joined as Director of AI at Taylor Geospatial. We’re a nonprofit that pushes geospatial projects from research into production. We identified a huge gap in the geospatial industry: academics publish papers and prototypes, but they often disappear into publications and conferences, while industry teams are trying to make machine learning work with satellites, geospatial analytics, stakeholders, and countries to turn remote sensing data into real insights.

There’s a big gap between those worlds. A lot of people make claims in papers and produce bad spaghetti code, and nobody wants to spend time turning that into something usable. It’s a little easier with agents now, but our main goal is to work with professors and researchers and actually put their work into production, whether that’s in a web app or another deployable form. We recently did a global-scale inference of all agricultural field boundaries across the globe for the past two years, which, surprisingly, had never been done before. Doing more projects like that is our goal.

I’ve also been working on a library called TorchGeo as a core maintainer for about five or six years. I’m deep into PyTorch, deep learning, geospatial coding, and open-source software development. If you have any questions about that, I’m happy to answer them as well.

What is cloud-native geospatial? The first thing we have to talk about is the old way versus the new way. The old way was that you had some cloud storage, a NAS, or a storage server with satellite imagery from NASA or someone else, and you would download 50 to 100 gigabytes—or even a terabyte—of data before you could start your analysis. As you can imagine, some of you may not even have a terabyte of storage, and five years ago that was even more of a problem. If you only care about a small subset of the globe, you don’t want to download the entire globe. You just want the subset you care about.

There’s been a huge effort in the geospatial industry, mostly driven by open-source development, to create new formats and schemas that let us convert the old way into the new way: data stored in cloud storage that lets you slice out only the subset you need. This helps everybody because you can start streaming data quickly and you don’t need huge amounts of compute just to get started.

The cloud-native stack is built on these new formats. Satellite imagery or rasters used to be stored as TIFF files. Now we have Cloud Optimized GeoTIFFs, or COGs. A COG is basically a TIFF, but the large image is chunked internally so you can request only the part you need instead of downloading the whole thing.

GeoParquet is a similar idea for geometry. Think of building footprints in OpenStreetMap or Google Maps: those are polygons, not images. GeoParquet stores them in a way that lets you efficiently slice out only the subset you need. The GeoParquet 1.0 spec came out recently.

Zarr is the new hotness right now. It lets you structure time series and other N-dimensional arrays. If I have a bunch of satellite images over time, I can store them in Zarr. You can do 3D time series, or really as many dimensions as you want.

PMTiles is what powers some of the web apps I’m going to show you. When you look at Google Maps and zoom in and out, it renders the visualization on the fly, and getting that to be responsive is actually a big engineering challenge.

Cloud Optimized Point Cloud, or COPC, works like a COG for point clouds. If I have a huge point cloud of the Earth derived from satellite data, I can slice out just the subset I care about.

Then there’s the question of discovery. If I have terabytes of satellite imagery, how do I know which files intersect the area I care about? If I only care about the bounding box around Austin, Texas, and the past 24 hours, how do I search that? That’s where STAC comes in. STAC sits on top of all these formats as a catalog and index that lets you search the data very quickly with a few lines of code.

What you’re going to see in the next few slides wasn’t really doable four or five years ago. You had to inspect filenames, try to infer datetimes from them, and figure out intersections manually. You had to do that separately for every satellite because each company and organization had its own standard. The big win here is standardization. People finally sat down and said, “This sucks. We need to do something about it.”

The timeline of cloud-native geospatial is pretty recent. In 2015, Landsat imagery finally made its way onto AWS. Before that, you were often requesting bulk data to be sent to you. Then came the first STAC sprint run by Radiant Earth, STAC 1.0, PMTiles, COPC, GeoParquet, and Zarr v3 with chunking and sharding, which is really cool. It even supports geometries now, so there’s GeoZarr.

You can see how recent all of this is. We’re at the cusp of a lot of exciting things happening, where you can do more with less compute.

The cool thing is that the data already exists. We’ve just converted it into better formats. You can get petabytes of climate data going back decades. We have point clouds of the entire Earth over time. The Overture Maps Foundation and OpenStreetMap have a ton of geometry data with rich tags, so you can get a building and its height, for example. We’re also about to release a global field boundary dataset for 2024 and 2025 with more than 119 million field boundaries across the globe, all generated by a machine learning model that we deployed.

To explain how COGs work, you take a large image and chunk it into many subchunks. Then you use HTTP range requests to say, “I only want this little subset.” Before, you had to download the whole file. For visualization, there’s also the concept of overviews. You take the original image, downsample it by half, then downsample that in half, and so on. When you’re rendering a visualization or a web app, those precomputed overviews let you zoom smoothly and respond on the fly.

GeoParquet works similarly. Imagine I have 2.6 billion building geometries and I only care about Austin. I can slice out that subset. This enables predicate pushdown, which means I don’t have to scan every row to figure out which geometry I care about. The metadata already tells me which chunks cover which bounds, so I can go directly to the chunks I need. That’s much faster than scanning row by row.

As I mentioned, STAC lets you specify a bounding box, a datetime, and other metadata. For example, satellite imagery often includes lots of cloudy scenes, so maybe I want the least cloudy image in a month or a year. You can do that in a few lines of code using PySTAC Client. Microsoft’s Planetary Computer is a big STAC catalog. Earth Search from Element 84 is another. USGS has LandsatLook. These are catalogs run by organizations that expose APIs, and many of them are free. Once you search, they return the assets you care about, which you can then feed into other libraries to load the data.

If you want to query only the buildings in Austin, this is basically the code. Install DuckDB, select the building type, height, area, and geometry from an S3 URL that the Overture Maps Foundation maintains, and restrict it to the bounds of Austin. That’s it. It runs directly from your local computer, on the fly, with no cluster required. There’s a visualization of that area that you can zoom around in entirely in your browser, without needing a ton of money or cloud compute.

I’m doing the same thing with Zarr. I can take public ERA5 data, request the 2-meter temperature, and visualize it live. You can also see 10-meter wind data, streaming on the fly every day or every hour. I’m not downloading data ahead of time; it’s live in the browser.

If you want to search satellite data for UT Austin’s campus, this is the query. I open the Planetary Computer STAC API with PySTAC Client, request the Sentinel-2 collection, provide the bounding box for Austin, set a date range, and ask for the best image sorted by least cloud cover, under 15%. The search returns a Python dictionary of assets. You still need to clip those results because they return the full satellite image intersecting your AOI, but this gets you started.

From there, you can plot the RGB bands and get a 10-meter resolution image of UT Austin’s campus. I also made a false-color visualization by swapping the blue band with near-infrared. Near-infrared is very useful for vegetation analysis, so the red highlights trees and other vegetation. NDVI is the normalized difference between near-infrared and red, and it gives you a quick sense of vegetation health: bright green is healthy vegetation, while browner areas are more built up.

Another dataset is NAIP, the National Agriculture Imagery Program. Every two years, fixed-wing aircraft fly the entire continental U.S. and collect imagery. That gives you one-meter resolution, which is ten times finer than Sentinel-2, so you can see much more detail and do more analysis. NAIP also includes RGB and near-infrared, so you can compute NDVI there as well.

I’ve talked a lot about data, but let’s talk about AI and what’s out there, because we do a lot of machine learning on this data. Research has generally trended from task-specific models toward foundation models. We still have task-specific models, but everyone is trying to follow the LLM path: pretrain on a huge amount of data and get a general model that knows a lot. In geospatial, the goal is to build foundation models that can handle RGB imagery, climate data, radar data, and more, all at once. These models can produce embeddings—compressed intermediate representations of the data—or they can be fine-tuned on downstream tasks. Because they’ve already seen a lot of data, you usually need less labeled data than if you train from scratch.

There are still a lot of issues. No single model dominates. There are many benchmarks, and everyone proposes a new foundation model every week. A lot of this is driven by organizations like IBM. There are also privacy and coverage issues, because some pretraining data only covers certain areas. In geospatial machine learning, you want models that generalize. I want a model that works in the U.S., Europe, and Brazil, not just one place. Spatial autocorrelation makes that hard, and it’s easy to overfit to a specific region unless that’s the only place you plan to deploy.

Resolution is another issue. Different data sources come at different resolutions, and each has trade-offs. Climate data has also been underserved. One of the biggest issues for me is that many later benchmarks show that if you skip pretraining and just train on your labeled dataset, that still remains a strong competitor. That raises the question: do we really need to spend so much money on pretraining? A lot of current work is about getting more from less data rather than just throwing compute at the problem.

Most pretraining also uses data that is freely available online. Meanwhile, many commercial providers—BlackSky, Planet, Maxar—have some open data, but most of their data is not included in these foundation models. That means the resulting models often don’t work well on commercial imagery.

A slightly nerdier point is how pretraining works. Traditionally, one common approach was contrastive learning. You start with a single image and create two augmented versions—change the brightness, rotate it, or do something similar. You pass both through the model and get vector representations. Then you train the model so the vectors from the same image stay close together, while vectors from different images stay far apart. That turns out to be a strong pretraining objective even without labels.

Another popular approach is masked autoencoding. Transformer models, including the ones used in LLMs, operate on tokens. For images, they break the image into patches and tokenize those. You can exploit that by dropping some of the tokens and asking the model to reconstruct the missing parts. That’s a very effective pretraining task. It does come with trade-offs, though. It tends to overfit to RGB color information more than to some of the other modalities.

There’s another axis here: patch embeddings versus pixel embeddings. If I have a 256 by 256 satellite image, a pixel embedding gives me an embedding aligned to each pixel, while a patch embedding gives me a single embedding for the whole image patch. Patch embeddings are more efficient because they compress that spatial extent into one vector. Pixel embeddings preserve more detail but are much larger. Each has pros and cons.

Patch embeddings are especially nice for retrieval. If I care about a forest patch, I can search for other patches that are most similar. That works a lot like retrieval in LLMs: you embed the query, compare it against a large document or knowledge base, and pull back the closest matches. You can think of the Earth as a very large document.

I’ve talked a bit about modeling, but the key thing is that modeling is the sexy part, not the hard part. Eighty percent of the work is always the pipeline. When you try to deploy something, you get stuck on post-processing. You train a model, run inference, and then realize you need to put the output back into a cloud-native format. You have to store it in cloud storage, chunk it correctly, partition it correctly, georeference it, tile it for visualization, generate overviews, log metadata, and build a STAC catalog around it. A lot happens outside “train model, run inference, deliver result.” That’s the hard part, although cloud-native tooling is making it better.

A lot of tooling is being rewritten in Rust. Traditionally, this ecosystem was dominated by C++, especially GDAL, which is a core geospatial library. Many people in the cloud-native space are rewriting components in Rust because it works well with Arrow and Parquet. It’s actually a really fun time. I wrote one library that optimized a GDAL function in Rust with Claude, and it improved performance by up to 10x. There’s a lot of performance still left on the table because GDAL is such a core library and won’t accept very ambitious pull requests. So people are realizing they can rewrite pieces themselves, open-source them, and get adoption from anyone who cares about performance.

That brings me to the GeoAI pipeline and TorchGeo. I don’t talk about TorchGeo enough in this slide deck, but the meat and potatoes of TorchGeo is that PyTorch’s original vision library, TorchVision, assumes you have small images like photos from your phone and labels like “cat” or “dog.” It doesn’t handle large satellite scenes well. TorchGeo adopts a cloud-native structure. In geospatial training, you typically have large satellite scenes, labels, and a need to sample patches from those scenes. You also need reprojection, resampling, and other preprocessing to align everything. TorchGeo does that on the fly instead of forcing you to preprocess everything offline and duplicate your data.

In a relatively small number of lines of code, you can train a model. I point TorchGeo at Sentinel-2 imagery in one folder and EnviroAtlas labels in another, choose a patch size of 256 pixels, tell it to sample 1,000 patches, create a data loader, and iterate. Behind the scenes, it loads, reprojects, and resamples everything on the fly. That’s really the core of TorchGeo, along with some other features.

The inference story is similar. At scale, instead of sampling random patches, I want to sample a grid across a large satellite scene. That’s what TorchGeo inference does. You give it a large scene, use a GridGeoSampler instead of a RandomGeoSampler, run the model over those patches, and reconstruct the prediction into a COG or Zarr. To do this at global scale, you need distributed infrastructure, because the Earth has a lot of scenes and they overlap. If you’re doing the whole globe, you want a Kubernetes cluster or some AWS service. The nice thing is that this is embarrassingly parallel. If I know how many patches there are, I can distribute them to as many workers as I’m willing to pay for.

That’s exactly what we did for Fields of the World. We extracted patches, ran distributed inference, and handled overlap merging. We did inference on overlapping regions and then used weighted averaging so the outputs looked good. If you use non-overlapping patches, you get ugly tile-edge effects, because models usually perform worse at the borders of a patch. Don’t neglect that, because people will absolutely roast you for visible tiling artifacts.

From there, you still need more post-processing. AI is not perfect. We used morphological operations to clean up predictions. Because we were running globally, we also had to filter out obviously wrong regions. Otherwise, the model would happily predict fields in Alaska or in the ocean if it hadn’t seen those regions during training. We used an external land-cover dataset to filter to cropland instead of trying to perfectly prefilter all scenes up front.

Then we still had to polygonize the results: converting field masks into geometries. That involves a bunch of algorithms and knobs and becomes tedious very quickly. After all of that, you still have to store the result and serve it to customers or open-source users. The main thing I’m trying to get across is that there’s a lot of work here beyond modeling. It’s not conceptually impossible; it’s just extra work. Claude is actually really good at helping with it.

Here are the predictions. This is running live in the browser. If you zoom in, the visualization changes because it uses different overviews. The green indicates field boundaries, and at different zoom levels we also aggregate how many fields are in an area so you can see where density is high and decide where to zoom in. This is a development server we’re planning to launch on Earth Day, so this is a sneak peek.

We don’t have everything built yet, but eventually you’ll be able to draw a box and download those geometries on the fly from cloud storage. The idea is that it’s freely available to anyone, from individual farmers to organizations doing field-boundary analysis. We eat the cost.

The really interesting next step is what you join this with. There’s crop yield data, crop stress data, and crop health data. Once you have the geometries, you can join them with those other datasets. We’re not prescribing how people should use it. Many countries have these kinds of field datasets for their own territory, but not all do. The spirit of this project is just: make it available, and good things will happen.

You can also run inference on the fly for older areas. In the demo, you click a region of interest and it kicks off an inference job that returns the field boundaries. I won’t wait for that here, but you can play with it.

The other cool idea in geospatial foundation models is to use embeddings directly instead of only fine-tuning. After pretraining, if you chop off the final task head and keep the intermediate outputs, you get learned features—embeddings—that are useful representations and much smaller than the raw image. You’ve taken an image, passed it through a model, and compressed it into a vector. In that latent space, similar things end up close together. So if you compare vectors using distance or dot-product similarity, you can find the top-k most similar results.

The challenge is that these embeddings are often huge because they use 32-bit floating point values. That makes large-scale retrieval hard to do interactively. One thing my buddy Caleb from the Microsoft AI for Good Lab and I have been exploring is whether we actually need all that precision. It turns out we often don’t. A lot of design choices around embedding size are not strongly data-driven; they’re just preset values. In practice, many of those dimensions are redundant. You can often remove 75 to 90 percent of them and still get good retrieval. You can also quantize them from float32 down to int8, int4, or even binary. In the binary case, the vector is just ones and zeros.

Once you have binary vectors, you can use a very fast algorithm called popcount to compute Hamming distance, which lets you measure similarity by comparing bit patterns. That means you can do retrieval directly in the browser, without a cluster, with everything sitting in cloud storage. That’s what I’m showing in the app I built. You draw a region, it downloads the embeddings that intersect that region, and then you click positive and negative example points. A positive point says, “Find more things like this.” A negative point says, “Find things like this, but unlike that.” The app computes Hamming distance on the fly and highlights the most similar results.

I also added some fun examples, such as the most unique examples in the dataset. The embeddings themselves were produced by a company called Legend. They do embedding-based retrieval as a paid product on a cluster, but they generously produced global Sentinel-2 embeddings on S3 for free. All I did was download them, repartition them, and upload them back to S3 in a form that worked for this app. It’s a lot of fun to explore. It may take a little while if your network is slow, but we’ve gotten great feedback on it.

I also want to talk about how I ship geospatial products today versus a year ago. Everything you saw was created with Claude—the entire presentation and all the apps. That has kind of blown me away. My goal with this presentation was to avoid making a boring PowerPoint and instead build videos and web apps. For the last few months, I’ve been pushing Claude as far as I can, seeing how close it can get me to 90 percent before I take over the final 10 percent.

The retrieval app I showed was completely built with Claude. I’m not a frontend developer. I know my way around the stack, but I’m not going to design something from scratch. I know what looks cool, though. One of my prompts was, “Make this look like Anthropic wrote it.” This slide deck was built with JavaScript driven by a Markdown file. All the animations and videos were built with ThreeBlue1Brown’s Manim library. I would talk into my microphone, describe what I wanted on a slide, and Claude would generate the video. Then I’d embed it in the presentation.

We’ve also been drowning in open source with AI-generated contributors. To deal with that, I now use AI to review all the open pull requests every day and summarize which ones are easy to review and which ones are not. It can even give an initial review. Because Claude is effectively talking to me, and I then manually review things and feed that back into Claude, it’s often easier to cut myself out of that loop until the last mile.

The broader point is that domain expertise plus agents is a force multiplier. This slide deck would probably have taken me a couple of weeks of on-and-off work, especially because I have two girls now and not a lot of spare time. Being able to prompt Claude from my phone and review later has been huge. I highly recommend using these tools creatively. Don’t depend on them blindly, but do use them for design, iteration, and review. You still need to know what success looks like.

I’ve also been using Claude heavily for research papers. When I review a paper now, I often feed it into Claude and ask for the major issues before I read it myself. You’d be surprised how many people seem to submit papers without even having Codex review them. The bar is higher now. You can’t just write everything from scratch and assume it will get accepted, because everybody else is using AI too.

This is also the blog Caleb and I have been running for a few weeks. The entire site and blog are built with Claude and Markdown. The ideas are ours, because we’ve been in this space for a long time, but a lot of the experiments and web apps are built through prompting. We use Claude and Codex together to revise the writing. AI-written text isn’t automatically bad if it comes from a good idea, but you absolutely need to review it before you publish. Hallucinations are better than they used to be, but they still happen. Trust me—models will just make up numbers.

We run the blog weekly or biweekly. It has a lot of fun ideas in GeoAI that we never had time to implement before. Now we can just tell Claude to build a prototype on the side, review it, and see whether it’s actually useful.

Another thing I built a few months ago is a STAC MCP server and a VS Code extension for the Planetary Computer. Instead of looking up which satellite or climate dataset you want and how to query it, you can just tell your agent and the MCP server figures it out, downloads the data, and even plots it. That last part matters because a lot of geospatial data comes in raw sensor values, not pretty RGB 8-bit imagery, so you usually have to convert it into a useful visualization. I already built tools for most of the Planetary Computer datasets.

I also noticed that every time I started a new research idea, I would repeat the same setup steps: create a GitHub repo, add uv and a bunch of Python tools, add LaTeX templates for papers and slides, and so on. Instead of repeating that every time, I made a research repository you can use as a template, with all the goodies I use already built in. For example, the slides.js folder uses the same format as the slide deck I’m showing here. Lots of good stuff is in there.

I basically write all my papers directly alongside the experiments now. As experiments run, I keep an update report in LaTeX. I also have a GitHub integration to Overleaf, which is basically Google Docs for research papers. Definitely check that out. It’s free and public.

I made one more example around USDA crop data. The USDA publishes a crop dataset called the Crop Sequence Boundaries Dataset, but it comes in a really ugly format. I got annoyed with it, so I had Claude download it, convert it to GeoParquet, partition it, put it into S3 storage, and build a frontend visualization. Now you can see the fields and their crop types—soybeans, whatever—with a legend. You can toggle layers on and off, including satellite imagery, and if you care about a particular field, you can hold Shift, draw a box, click Download AOI, and it automatically downloads the data for you.

That’s essentially the same idea as the Fields of the World inference app, which is still a little in development because we’re a couple of weeks away from announcing it.

Wrapping up, I’ve talked about a lot of different things, so you may be wondering what the goal was. The big picture is that there’s a flywheel here. People work with models and datasets, realize the tooling is painful, and then come together to design better schemas and standards. Those standards make it easier to build better models and global datasets, which exposes new problems, which leads to better tooling. We’re iteratively doing gradient descent toward a better solution for everyone. If you’re working anywhere in that loop, you’re creating value.

I’d encourage you to use AI coding agents to build web apps, show your data, and convert it into formats that don’t suck. I’d also encourage you to work on machine learning for these datasets. It doesn’t even have to be deep learning. A lot of people still use random forests, linear regression, and logistic regression. As long as it produces something useful, it doesn’t matter how trendy the model is. In GeoAI, the simplest things often end up being the most useful.

Strip out the for-profit marketing nonsense, give people something genuinely useful, and that’s often what goes viral.

What can you do today? Use PySTAC Client to search petabytes of data. Load that data with ODC-STAC. Those two libraries get you surprisingly far. With GeoAI, you can build retrieval systems with embeddings, and you can fine-tune foundation models with TorchGeo. We have a bunch of foundation models, training tools, and trainers. If you bring your data and align the code, you can train a state-of-the-art model yourself.

If you have comments, we also have a TorchGeo Slack channel. It started as a TorchGeo community, but at this point a lot of GeoAI researchers hang out there. People share papers, tools, and news, and you can talk directly with some of the best people in GeoAI just by joining and lurking—or participating.

That’s all I had today. There’s a QR code for my website with all my socials on it—GitHub, Twitter, LinkedIn, and email. Feel free to reach out with questions, or even just ask for advice. I’d be happy to review anything you’re building.