Lesson 5 · 14:45–15:25
What changed around the lake since 2017?
Embeddings earn their keep when we ask questions with them. Three questions,
each a few lines of scikit-learn or numpy: which places belong together, what each place is,
and where things changed.
Groups without labels
k-means sorts the pixels into k groups whose vectors are close. It learns from every 20th pixel and then labels them all.from sklearn.cluster import KMeans
aef = datapack.alphaearth(2025)
pixels = em.table(aef) # one row per pixel: 350,000 x 64
kmeans = KMeans(n_clusters=6, n_init="auto", random_state=0)
kmeans.fit(pixels[::20]) # learn from every 20th pixel
groups = kmeans.predict(pixels).reshape(aef.shape[1:])
f"{len(pixels):,} pixels in {kmeans.n_clusters} groups"
350,000 pixels in 6 groups
With two groups you get roughly "city" and "not city". As you add groups they start to follow
the lake, the airport's grass, different kinds of built-up land and the remaining fields.
Nobody gave the model a single label. But a group has no name: you have to look and name it yourself.
A few labels go a long way
Now suppose we can afford to label only a handful of pixels. As labels we borrow ESA's WorldCover map of 2021 (trees, grass and crops, built-up, bare, water), take 20 pixels of each class, train a logistic regression on their 2021 embeddings, and let it map everything else.from sklearn.linear_model import LogisticRegression
labels = classes(datapack.raster("worldcover")[0]).ravel() # 0-4; -1 = other
x = em.table(datapack.alphaearth(2021))
rng = np.random.default_rng(0)
train = np.concatenate([rng.choice(np.flatnonzero(labels == k), 20, replace=False)
for k in range(len(CLASS_NAMES))]) # 20 pixels a class
model = LogisticRegression(max_iter=2000).fit(x[train], labels[train])
predicted = model.predict(x)
agree = (predicted == labels)[labels >= 0].mean()
f"{len(train)} labelled pixels: a map that agrees with WorldCover on {agree:.0%}"
100 labelled pixels: a map that agrees with WorldCover on 71%
How this file was made · making it reads about 280 MB (AlphaEarth, TESSERA and Sentinel-2 for 2021), then 240 small models
The chart reads a small prepared file (3 kB): your copy in data/ if you have one, otherwise it downloads once. To make it yourself from the original data, run uv run cngeo datapack build --step aef --step tessera --step label_curve --force. This is the code that runs:
def label_curve(force: bool = False, seeds: int = 10): """Agreement with WorldCover for a classifier trained on n labels per class, for each kind of input.""" from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from .embeddings import dequantise, table, unit out = datapack.path("label_curve") if out.exists() and not force: return y = classes(_read(datapack.path("worldcover"))[0]).ravel() inputs = { "AlphaEarth": table(dequantise(_read(_cache("alphaearth_2021.tif")))), "TESSERA": table(unit(_read(_cache("tessera_2021.tif")).astype("float32") * _read(_cache("tessera_2021_scales.tif")))), "Sentinel-2 bands": table(sentinel2_bands(2021)), } keep = (y >= 0) & np.all([np.isfinite(x).all(1) for x in inputs.values()], axis=0) rows = [] for n in (1, 2, 5, 10, 20, 50, 100, 200): for seed in range(seeds): rng = np.random.default_rng(seed) train = np.concatenate([rng.choice(np.flatnonzero(keep & (y == k)), n, replace=False) for k in np.unique(y[keep])]) test = np.setdiff1d(np.flatnonzero(keep), train) test = rng.choice(test, min(50_000, len(test)), replace=False) for name, x in inputs.items(): model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(x[train], y[train]) rows.append({"input": name, "labels": n, "seed": seed, "accuracy": float((model.predict(x[test]) == y[test]).mean())}) print(f"labels {n}: done") df = pd.DataFrame(rows).groupby(["input", "labels"], as_index=False)["accuracy"].mean() df.to_parquet(out, index=False)
In both maps brown is built-up, yellow grass and crops, green trees, blue water and beige bare
ground. Ours is blotchier but gets the big picture right from 100 clicks' worth of labels.
The curve repeats this for different numbers of labels. The grey line is the same classifier
given the raw Sentinel-2 colours instead of embeddings.
With embeddings, ten labels per class do about as well as 200 labels on raw colours. That is the
point of a foundation model: it did the expensive learning once, so your map needs only a few
labels. Take the exact numbers with salt: WorldCover has its own errors, and pixels picked at
random from one small area flatter any method.
What changed since 2017?
If a square of ground stays the same, its vector stays put; if it changes, the vector swings away. One minus the dot product between two years is a change map, in one line.before, after = datapack.alphaearth(2017), datapack.alphaearth(2025)
change = 1 - (before * after).sum(axis=0) # 0: same as in 2017; higher: changed
charts.image(change, cmap=charts.CHANGE, vmin=0, vmax=0.6,
label="change from 2017 to 2025")
How this file was made · making it reads about 460 MB of AlphaEarth (nine years of the study area), about 3 minutes
The chart reads a small prepared file (90 kB): your copy in data/ if you have one, otherwise it downloads once. To make it yourself from the original data, run uv run cngeo datapack build --step aef --step aef_change --force. This is the code that runs:
def aef(force: bool = False): """AlphaEarth for every year 2017-2025 (cached); the pack keeps three of them.""" from .embeddings import aef_tiles, alphaearth_int8 for year in range(2017, 2026): out = _cache(f"alphaearth_{year}.tif") if not out.exists() or force: t0 = time.time() q = alphaearth_int8(year, STUDY, tiles=aef_tiles(year)) write_cog(out, q, nodata=-128) print(f"alphaearth {year}: {q.shape} in {time.time() - t0:.0f}s") if year in datapack.YEARS["aef"]: datapack.path("aef", year).parent.mkdir(parents=True, exist_ok=True) datapack.path("aef", year).write_bytes(out.read_bytes()) def aef_change(force: bool = False): """How far each 50 m square's AlphaEarth vector moved from 2017, for every later year.""" from .embeddings import coarsen, dequantise out = datapack.path("aef_change") if out.exists() and not force: return first = coarsen(dequantise(_read(_cache("alphaearth_2017.tif"))), 5) d, h, w = first.shape r, c = np.divmod(np.arange(h * w), w) df = pd.DataFrame({"c": c, "r": r}) for year in range(2018, 2026): e = coarsen(dequantise(_read(_cache(f"alphaearth_{year}.tif"))), 5) df[f"s{year}"] = np.clip(np.round((e * first).sum(0).ravel() * 100), -100, 100).astype("int16") out.parent.mkdir(parents=True, exist_ok=True) df.to_parquet(out, index=False)
The lake dominates: its surface went from a mat of weeds to a dug-out bed and open water. The
small bright patches around it are buildings that went up and plots that were cleared. Hover
one and follow it through the years to see when it happened.
Two cautions. First, slide to 2022 and 2023: almost every square moves a little, then comes back.
The ground didn't change; most likely the model's inputs did (one of the two Sentinel-1 radar
satellites failed at the end of 2021). Always compare a change with places you know stayed
the same.
Second, each vector summarises a whole year, so a flood that lasted a week barely registers.
Put it in the catalog
Last, our map of change and the embeddings themselves go into the catalog. For the embeddings we add the fields of the draft embeddings convention for STAC, so a reader knows what the 64 bands are and how to turn the integers back into numbers.change_path = mycatalog.DATA / "change_2017_2025.tif"
mycatalog.write_cog(change_path, change.astype("float32"))
mycatalog.add(change_path, datetime="2025-01-01",
title="Change of each 10 m square, 2017 to 2025 (AlphaEarth)")
embeddings_path = mycatalog.DATA / "alphaearth_2025.tif"
embeddings_path.write_bytes(datapack.fetch("aef", 2025).read_bytes())
embeddings_item = mycatalog.add(
embeddings_path, datetime="2025-01-01",
title="AlphaEarth Foundations embeddings, 2025 (Google DeepMind)",
**{"emb:type": "pixel", "emb:dimensions": 64, "emb:temporal_resolution": "annual",
"emb:quantization": {"method": "signed_square", "nodata": -128,
"quantized_dtype": "int8"}})
mycatalog.items()
| item | title | date | file | type |
|---|---|---|---|---|
| lake_2025 | Bellandur lake, February 2025 (Sentinel-2) | 2025-02-06 | lake_2025.tif | image/tiff |
| rain_flood_night | Rain over Bengaluru, night of 4-5 Sep 2022 (IMERG) | 2022-09-04 | rain_flood_night.zarr | application/vnd.zarr |
| waterbodies | Water bodies and their water area per year (CoRE Stack) | 2025-01-01 | waterbodies.parquet | application/vnd.apache.parquet |
| watersheds | Micro-watersheds around the lake (CoRE Stack) | 2025-01-01 | watersheds.parquet | application/vnd.apache.parquet |
| buildings | Building footprints east of the lake (Overture Maps) | 2025-01-01 | buildings.parquet | application/vnd.apache.parquet |
| change_2017_2025 | Change of each 10 m square, 2017 to 2025 (AlphaEarth) | 2025-01-01 | change_2017_2025.tif | image/tiff |
| alphaearth_2025 | AlphaEarth Foundations embeddings, 2025 (Google DeepMind) | 2025-01-01 | alphaearth_2025.tif | image/tiff |
# the whole folder again; the embeddings item is open, with its emb: fields
folder = mo.ui.file_browser(mycatalog.FOLDER, value=embeddings_item.get_self_href(),
multiple=False)
folder
my_catalog/ catalog.json alphaearth_2025/ alphaearth_2025.json buildings/ buildings.json change_2017_2025/ change_2017_2025.json data/ alphaearth_2025.tif 19.6 MB buildings.parquet 1.5 MB change_2017_2025.tif 1.6 MB lake_2025.tif 1.2 MB rain_flood_night.zarr/ 19 kB waterbodies.parquet 78 kB watersheds.parquet 0.4 MB lake_2025/ lake_2025.json rain_flood_night/ rain_flood_night.json waterbodies/ waterbodies.json watersheds/ watersheds.json
alphaearth_2025/alphaearth_2025.json
{
"type": "Feature",
"stac_version": "1.1.0",
"stac_extensions": ["https://stac-extensions.github.io/projection/v1.1.0/schema.json"],
"id": "alphaearth_2025",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[77.64937857188345, 12.908474585304221],
[77.7138348439426, 12.907813011974559],
[77.71432326139788, 12.95297581113992],
[77.64985542244153, 12.95363977682847],
[77.64937857188345, 12.908474585304221]
]
]
},
"bbox": [77.64937857188345, 12.907813011974559, 77.71432326139788, 12.95363977682847],
"properties": {
"proj:epsg": 32643,
"proj:geometry": {
"type": "Polygon",
"coordinates": [
[
[787500.0, 1428500.0],
[794500.0, 1428500.0],
[794500.0, 1433500.0],
[787500.0, 1433500.0],
[787500.0, 1428500.0]
]
]
},
"proj:bbox": [787500.0, 1428500.0, 794500.0, 1433500.0],
"proj:shape": [500, 700],
"proj:transform": [10.0, 0.0, 787500.0, 0.0, -10.0, 1433500.0, 0.0, 0.0, 1.0],
"title": "AlphaEarth Foundations embeddings, 2025 (Google DeepMind)",
"emb:type": "pixel",
"emb:dimensions": 64,
"emb:temporal_resolution": "annual",
"emb:quantization": {
"method": "signed_square",
"nodata": -128,
"quantized_dtype": "int8"
},
"datetime": "2025-01-01T00:00:00Z"
},
"links": [
{
"rel": "root",
"href": "../catalog.json",
"type": "application/json"
},
{
"rel": "parent",
"href": "../catalog.json",
"type": "application/json"
}
],
"assets": {
"data": {
"href": "../data/alphaearth_2025.tif",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"roles": []
}
}
}
That folder is a complete, small, cloud-native dataset about one lake. In the last lesson we
open it all again, starting from
catalog.json.
What embeddings can't do
- One summary per year. No dates within the year, and short events wash out.
- 10 m squares. A single rooftop or a narrow drain is below what they can see.
- Inputs change. Satellites fail and are replaced, so the "no change" level drifts from year to year.
- Learned mostly elsewhere. AlphaEarth's training labels were mostly from the United States. Check results against what you know on the ground.
- Read the licences. AlphaEarth's embeddings are CC BY 4.0 but the model is closed; TESSERA's embeddings and model are open.
Try it
- Change
n_clustersandpixels[::20]in the k-means cell. Does learning from every 5th pixel change the groups? - Give the classifier 5, then 100 labels per class. Does it match the curve above?
- Make the change map with TESSERA for 2018 and 2025 (
em.tessera(2018), normalised withem.unit). Does it agree with AlphaEarth about where things changed?