Lesson 4 · 14:00–14:45

Can a model read the pixels for us?

This morning we learned to find and fetch data. This afternoon we let a model look at it. A geospatial foundation model has studied years of satellite pictures of the whole planet, without anyone labelling them, and written down a short list of numbers for every 10 m square of land, every year. Similar places get similar numbers.
If you know word2vec, you know the idea: words used in similar sentences get nearby vectors. Here, squares of ground that look and behave alike through the year (their colour in every season, their radar echo, their texture, their neighbours) get nearby vectors. These vectors are called embeddings. Two open sets cover Bengaluru:
  • AlphaEarth Foundations from Google DeepMind: 64 numbers per 10 m square per year, 2017–2025, for all land. The embeddings are open (CC BY 4.0); the model is not.
  • TESSERA from the University of Cambridge: 128 numbers per square per year, learned from Sentinel-1 radar and Sentinel-2 time series. Both the embeddings and the model are open.

Find and read the embeddings

Everything from the morning applies. AlphaEarth's embeddings are Cloud-Optimized GeoTIFFs, one per 82 km square per year with 64 bands, and a GeoParquet index lists every file. Source Cooperative mirrors them. First the index:
our_box = (77.6494, 12.9085, 77.7143, 12.9530)  # west, south, east, north: 7 × 5 km around Bellandur lake
tiles = em.aef_tiles(2025, bbox=our_box)        # DuckDB on the GeoParquet index: which file has it?
tiles[["year", "utm_zone", "path"]]
year utm_zone path
2025 43N s3://us-west-2.opendata.source.coop/tge-labs/aef/v1/annual/2025/43N/xehci7l41lq9kp9i5-0000008192-0000008192.tiff
That file is 3 GB. We read only our 7 × 5 km box from it: 64 bands, each in its own tiles, so the helper fetches them in parallel. (The files happen to be stored upside down, south at the top; the helper flips them back.)
aef = em.alphaearth(2025, bounds=STUDY)   # the same box in UTM metres (config.py); offline? datapack.alphaearth(2025)
f"{aef.shape[0]} numbers for each of {aef[0].size:,} pixels"

64 numbers for each of 350,000 pixels

A pixel is 64 numbers

Hover over the picture. The bars are the 64 numbers of the 100 m square under your mouse (averaged from the 10 m pixels). Water, roofs, trees and bare ground each have their own pattern, but no single number means "water": the meaning is in the whole pattern.
Every pixel's vector has length 1, so all of them sit on the surface of a sphere in 64 dimensions. Two pixels are alike when their vectors point the same way, and the dot product measures exactly that: 1 for the same direction, 0 for unrelated. Here is every pixel compared with one pixel of open water in the lake.
row, col = em.pixel(LAKE)                          # the lake's pixel in our grid
lake = aef[:, row, col]                            # its 64 numbers
similarity = np.einsum("dhw,d->hw", aef, lake)     # dot product with every pixel

charts.image(similarity, cmap=charts.SIMILAR, vmin=0.3, vmax=1,
             label="similarity to one lake pixel", marker=(row, col))
The lake lights up, and so do the small ponds to the south, although nobody told the model what water is. Try it yourself: click anywhere on the picture below and the map on the right shows how similar every 70 m square is to the one you clicked. Switch between the two models.
Click a block of offices, a road, a patch of trees. In the browser each square is compared using 16 numbers squeezed from the full vector so the page stays fast; the code above uses all 64.

Two models, one place

TESSERA is read through its own package, geotessera, from a Zarr store on Source Cooperative. Its vectors are not length 1, so we scale them before comparing.
tessera = em.unit(em.tessera(2025))   # offline? em.unit(datapack.tessera(2025))
f"{tessera.shape[0]} numbers for each of {tessera[0].size:,} pixels"

128 numbers for each of 350,000 pixels

Each picture paints a model's three most important directions as red, green and blue, so colours mean nothing in themselves; what matters is which places share a colour. Both separate the lake, the airport, the office parks and the fields. AlphaEarth is smoother; TESSERA keeps sharper 10 m detail of roads and buildings. They are different spaces: never compare a vector from one model with a vector from the other.

Try it

  1. Change LAKE in the similarity cell to the coordinates of a park (try 77.682, 12.925) and see what else the model thinks is park.
  2. Compute the similarity map with TESSERA (tessera instead of aef). Where do the two models disagree?
Made with open data and open-source Python · marimo · Altair