Lesson 4 · 14:00–14:45
Can a model read the pixels for us?
This morning we learned to find and fetch data. This afternoon we let a model
look at it. A geospatial foundation model has studied years of satellite pictures of the whole
planet, without anyone labelling them, and written down a short list of numbers for every 10 m
square of land, every year. Similar places get similar numbers.
If you know word2vec, you know the idea: words used in similar sentences get nearby vectors.
Here, squares of ground that look and behave alike through the year (their colour in every
season, their radar echo, their texture, their neighbours) get nearby vectors. These vectors are
called embeddings.
Two open sets cover Bengaluru:
- AlphaEarth Foundations from Google DeepMind: 64 numbers per 10 m square per year, 2017–2025, for all land. The embeddings are open (CC BY 4.0); the model is not.
- TESSERA from the University of Cambridge: 128 numbers per square per year, learned from Sentinel-1 radar and Sentinel-2 time series. Both the embeddings and the model are open.
Find and read the embeddings
Everything from the morning applies. AlphaEarth's embeddings are Cloud-Optimized GeoTIFFs, one per 82 km square per year with 64 bands, and a GeoParquet index lists every file. Source Cooperative mirrors them. First the index:our_box = (77.6494, 12.9085, 77.7143, 12.9530) # west, south, east, north: 7 × 5 km around Bellandur lake
tiles = em.aef_tiles(2025, bbox=our_box) # DuckDB on the GeoParquet index: which file has it?
tiles[["year", "utm_zone", "path"]]
| year | utm_zone | path |
|---|---|---|
| 2025 | 43N | s3://us-west-2.opendata.source.coop/tge-labs/aef/v1/annual/2025/43N/xehci7l41lq9kp9i5-0000008192-0000008192.tiff |
That file is 3 GB. We read only our 7 × 5 km box from it: 64 bands, each in its own tiles, so
the helper fetches them in parallel. (The files happen to be stored upside down, south at the
top; the helper flips them back.)
aef = em.alphaearth(2025, bounds=STUDY) # the same box in UTM metres (config.py); offline? datapack.alphaearth(2025)
f"{aef.shape[0]} numbers for each of {aef[0].size:,} pixels"
64 numbers for each of 350,000 pixels
A pixel is 64 numbers
Hover over the picture. The bars are the 64 numbers of the 100 m square under your mouse (averaged from the 10 m pixels). Water, roofs, trees and bare ground each have their own pattern, but no single number means "water": the meaning is in the whole pattern.Every pixel's vector has length 1, so all of them sit on the surface of a sphere in 64
dimensions. Two pixels are alike when their vectors point the same way, and the dot product
measures exactly that: 1 for the same direction, 0 for unrelated. Here is every pixel compared
with one pixel of open water in the lake.
row, col = em.pixel(LAKE) # the lake's pixel in our grid
lake = aef[:, row, col] # its 64 numbers
similarity = np.einsum("dhw,d->hw", aef, lake) # dot product with every pixel
charts.image(similarity, cmap=charts.SIMILAR, vmin=0.3, vmax=1,
label="similarity to one lake pixel", marker=(row, col))
The lake lights up, and so do the small ponds to the south, although nobody told the model what
water is. Try it yourself: click anywhere on the picture below and the map on the right shows how
similar every 70 m square is to the one you clicked. Switch between the two models.
Click a block of offices, a road, a patch of trees. In the browser each square is compared using
16 numbers squeezed from the full vector so the page stays fast; the code above uses all 64.
Two models, one place
TESSERA is read through its own package,geotessera, from a Zarr store on Source Cooperative.
Its vectors are not length 1, so we scale them before comparing.tessera = em.unit(em.tessera(2025)) # offline? em.unit(datapack.tessera(2025))
f"{tessera.shape[0]} numbers for each of {tessera[0].size:,} pixels"
128 numbers for each of 350,000 pixels
Each picture paints a model's three most important directions as red, green and blue, so
colours mean nothing in themselves; what matters is which places share a colour. Both separate
the lake, the airport, the office parks and the fields. AlphaEarth is smoother; TESSERA keeps
sharper 10 m detail of roads and buildings. They are different spaces: never compare a vector
from one model with a vector from the other.
Try it
- Change
LAKEin the similarity cell to the coordinates of a park (try 77.682, 12.925) and see what else the model thinks is park. - Compute the similarity map with TESSERA (
tesserainstead ofaef). Where do the two models disagree?