Lesson 5 · 14:45–15:25

What changed around the lake since 2017?

Embeddings earn their keep when we ask questions with them. Three questions, each a few lines of scikit-learn or numpy: which places belong together, what each place is, and where things changed.

Groups without labels

k-means sorts the pixels into k groups whose vectors are close. It learns from every 20th pixel and then labels them all.
from sklearn.cluster import KMeans

aef = datapack.alphaearth(2025)
pixels = em.table(aef)                     # one row per pixel: 350,000 x 64
kmeans = KMeans(n_clusters=6, n_init="auto", random_state=0)
kmeans.fit(pixels[::20])                   # learn from every 20th pixel
groups = kmeans.predict(pixels).reshape(aef.shape[1:])
f"{len(pixels):,} pixels in {kmeans.n_clusters} groups"

350,000 pixels in 6 groups

With two groups you get roughly "city" and "not city". As you add groups they start to follow the lake, the airport's grass, different kinds of built-up land and the remaining fields. Nobody gave the model a single label. But a group has no name: you have to look and name it yourself.

A few labels go a long way

Now suppose we can afford to label only a handful of pixels. As labels we borrow ESA's WorldCover map of 2021 (trees, grass and crops, built-up, bare, water), take 20 pixels of each class, train a logistic regression on their 2021 embeddings, and let it map everything else.
from sklearn.linear_model import LogisticRegression

labels = classes(datapack.raster("worldcover")[0]).ravel()   # 0-4; -1 = other
x = em.table(datapack.alphaearth(2021))

rng = np.random.default_rng(0)
train = np.concatenate([rng.choice(np.flatnonzero(labels == k), 20, replace=False)
                        for k in range(len(CLASS_NAMES))])     # 20 pixels a class
model = LogisticRegression(max_iter=2000).fit(x[train], labels[train])
predicted = model.predict(x)

agree = (predicted == labels)[labels >= 0].mean()
f"{len(train)} labelled pixels: a map that agrees with WorldCover on {agree:.0%}"

100 labelled pixels: a map that agrees with WorldCover on 71%

How this file was made · making it reads about 280 MB (AlphaEarth, TESSERA and Sentinel-2 for 2021), then 240 small models

The chart reads a small prepared file (3 kB): your copy in data/ if you have one, otherwise it downloads once. To make it yourself from the original data, run uv run cngeo datapack build --step aef --step tessera --step label_curve --force. This is the code that runs:

def label_curve(force: bool = False, seeds: int = 10):
    """Agreement with WorldCover for a classifier trained on n labels per class, for each kind of input."""
    from sklearn.linear_model import LogisticRegression
    from sklearn.pipeline import make_pipeline
    from sklearn.preprocessing import StandardScaler

    from .embeddings import dequantise, table, unit

    out = datapack.path("label_curve")
    if out.exists() and not force:
        return
    y = classes(_read(datapack.path("worldcover"))[0]).ravel()
    inputs = {
        "AlphaEarth": table(dequantise(_read(_cache("alphaearth_2021.tif")))),
        "TESSERA": table(unit(_read(_cache("tessera_2021.tif")).astype("float32") * _read(_cache("tessera_2021_scales.tif")))),
        "Sentinel-2 bands": table(sentinel2_bands(2021)),
    }
    keep = (y >= 0) & np.all([np.isfinite(x).all(1) for x in inputs.values()], axis=0)
    rows = []
    for n in (1, 2, 5, 10, 20, 50, 100, 200):
        for seed in range(seeds):
            rng = np.random.default_rng(seed)
            train = np.concatenate([rng.choice(np.flatnonzero(keep & (y == k)), n, replace=False) for k in np.unique(y[keep])])
            test = np.setdiff1d(np.flatnonzero(keep), train)
            test = rng.choice(test, min(50_000, len(test)), replace=False)
            for name, x in inputs.items():
                model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(x[train], y[train])
                rows.append({"input": name, "labels": n, "seed": seed, "accuracy": float((model.predict(x[test]) == y[test]).mean())})
        print(f"labels {n}: done")
    df = pd.DataFrame(rows).groupby(["input", "labels"], as_index=False)["accuracy"].mean()
    df.to_parquet(out, index=False)
In both maps brown is built-up, yellow grass and crops, green trees, blue water and beige bare ground. Ours is blotchier but gets the big picture right from 100 clicks' worth of labels. The curve repeats this for different numbers of labels. The grey line is the same classifier given the raw Sentinel-2 colours instead of embeddings. With embeddings, ten labels per class do about as well as 200 labels on raw colours. That is the point of a foundation model: it did the expensive learning once, so your map needs only a few labels. Take the exact numbers with salt: WorldCover has its own errors, and pixels picked at random from one small area flatter any method.

What changed since 2017?

If a square of ground stays the same, its vector stays put; if it changes, the vector swings away. One minus the dot product between two years is a change map, in one line.
before, after = datapack.alphaearth(2017), datapack.alphaearth(2025)
change = 1 - (before * after).sum(axis=0)   # 0: same as in 2017; higher: changed
charts.image(change, cmap=charts.CHANGE, vmin=0, vmax=0.6,
             label="change from 2017 to 2025")
How this file was made · making it reads about 460 MB of AlphaEarth (nine years of the study area), about 3 minutes

The chart reads a small prepared file (90 kB): your copy in data/ if you have one, otherwise it downloads once. To make it yourself from the original data, run uv run cngeo datapack build --step aef --step aef_change --force. This is the code that runs:

def aef(force: bool = False):
    """AlphaEarth for every year 2017-2025 (cached); the pack keeps three of them."""
    from .embeddings import aef_tiles, alphaearth_int8

    for year in range(2017, 2026):
        out = _cache(f"alphaearth_{year}.tif")
        if not out.exists() or force:
            t0 = time.time()
            q = alphaearth_int8(year, STUDY, tiles=aef_tiles(year))
            write_cog(out, q, nodata=-128)
            print(f"alphaearth {year}: {q.shape} in {time.time() - t0:.0f}s")
        if year in datapack.YEARS["aef"]:
            datapack.path("aef", year).parent.mkdir(parents=True, exist_ok=True)
            datapack.path("aef", year).write_bytes(out.read_bytes())


def aef_change(force: bool = False):
    """How far each 50 m square's AlphaEarth vector moved from 2017, for every later year."""
    from .embeddings import coarsen, dequantise

    out = datapack.path("aef_change")
    if out.exists() and not force:
        return
    first = coarsen(dequantise(_read(_cache("alphaearth_2017.tif"))), 5)
    d, h, w = first.shape
    r, c = np.divmod(np.arange(h * w), w)
    df = pd.DataFrame({"c": c, "r": r})
    for year in range(2018, 2026):
        e = coarsen(dequantise(_read(_cache(f"alphaearth_{year}.tif"))), 5)
        df[f"s{year}"] = np.clip(np.round((e * first).sum(0).ravel() * 100), -100, 100).astype("int16")
    out.parent.mkdir(parents=True, exist_ok=True)
    df.to_parquet(out, index=False)
The lake dominates: its surface went from a mat of weeds to a dug-out bed and open water. The small bright patches around it are buildings that went up and plots that were cleared. Hover one and follow it through the years to see when it happened. Two cautions. First, slide to 2022 and 2023: almost every square moves a little, then comes back. The ground didn't change; most likely the model's inputs did (one of the two Sentinel-1 radar satellites failed at the end of 2021). Always compare a change with places you know stayed the same. Second, each vector summarises a whole year, so a flood that lasted a week barely registers.

Put it in the catalog

Last, our map of change and the embeddings themselves go into the catalog. For the embeddings we add the fields of the draft embeddings convention for STAC, so a reader knows what the 64 bands are and how to turn the integers back into numbers.
change_path = mycatalog.DATA / "change_2017_2025.tif"
mycatalog.write_cog(change_path, change.astype("float32"))
mycatalog.add(change_path, datetime="2025-01-01",
              title="Change of each 10 m square, 2017 to 2025 (AlphaEarth)")

embeddings_path = mycatalog.DATA / "alphaearth_2025.tif"
embeddings_path.write_bytes(datapack.fetch("aef", 2025).read_bytes())
embeddings_item = mycatalog.add(
    embeddings_path, datetime="2025-01-01",
    title="AlphaEarth Foundations embeddings, 2025 (Google DeepMind)",
    **{"emb:type": "pixel", "emb:dimensions": 64, "emb:temporal_resolution": "annual",
       "emb:quantization": {"method": "signed_square", "nodata": -128,
                            "quantized_dtype": "int8"}})
mycatalog.items()
item title date file type
lake_2025 Bellandur lake, February 2025 (Sentinel-2) 2025-02-06 lake_2025.tif image/tiff
rain_flood_night Rain over Bengaluru, night of 4-5 Sep 2022 (IMERG) 2022-09-04 rain_flood_night.zarr application/vnd.zarr
waterbodies Water bodies and their water area per year (CoRE Stack) 2025-01-01 waterbodies.parquet application/vnd.apache.parquet
watersheds Micro-watersheds around the lake (CoRE Stack) 2025-01-01 watersheds.parquet application/vnd.apache.parquet
buildings Building footprints east of the lake (Overture Maps) 2025-01-01 buildings.parquet application/vnd.apache.parquet
change_2017_2025 Change of each 10 m square, 2017 to 2025 (AlphaEarth) 2025-01-01 change_2017_2025.tif image/tiff
alphaearth_2025 AlphaEarth Foundations embeddings, 2025 (Google DeepMind) 2025-01-01 alphaearth_2025.tif image/tiff
# the whole folder again; the embeddings item is open, with its emb: fields
folder = mo.ui.file_browser(mycatalog.FOLDER, value=embeddings_item.get_self_href(),
                            multiple=False)
folder
my_catalog/
  catalog.json
  alphaearth_2025/
    alphaearth_2025.json
  buildings/
    buildings.json
  change_2017_2025/
    change_2017_2025.json
  data/
    alphaearth_2025.tif  19.6 MB
    buildings.parquet  1.5 MB
    change_2017_2025.tif  1.6 MB
    lake_2025.tif  1.2 MB
    rain_flood_night.zarr/  19 kB
    waterbodies.parquet  78 kB
    watersheds.parquet  0.4 MB
  lake_2025/
    lake_2025.json
  rain_flood_night/
    rain_flood_night.json
  waterbodies/
    waterbodies.json
  watersheds/
    watersheds.json
alphaearth_2025/alphaearth_2025.json
{
  "type": "Feature",
  "stac_version": "1.1.0",
  "stac_extensions": ["https://stac-extensions.github.io/projection/v1.1.0/schema.json"],
  "id": "alphaearth_2025",
  "geometry": {
    "type": "Polygon",
    "coordinates": [
      [
        [77.64937857188345, 12.908474585304221],
        [77.7138348439426, 12.907813011974559],
        [77.71432326139788, 12.95297581113992],
        [77.64985542244153, 12.95363977682847],
        [77.64937857188345, 12.908474585304221]
      ]
    ]
  },
  "bbox": [77.64937857188345, 12.907813011974559, 77.71432326139788, 12.95363977682847],
  "properties": {
    "proj:epsg": 32643,
    "proj:geometry": {
      "type": "Polygon",
      "coordinates": [
        [
          [787500.0, 1428500.0],
          [794500.0, 1428500.0],
          [794500.0, 1433500.0],
          [787500.0, 1433500.0],
          [787500.0, 1428500.0]
        ]
      ]
    },
    "proj:bbox": [787500.0, 1428500.0, 794500.0, 1433500.0],
    "proj:shape": [500, 700],
    "proj:transform": [10.0, 0.0, 787500.0, 0.0, -10.0, 1433500.0, 0.0, 0.0, 1.0],
    "title": "AlphaEarth Foundations embeddings, 2025 (Google DeepMind)",
    "emb:type": "pixel",
    "emb:dimensions": 64,
    "emb:temporal_resolution": "annual",
    "emb:quantization": {
      "method": "signed_square",
      "nodata": -128,
      "quantized_dtype": "int8"
    },
    "datetime": "2025-01-01T00:00:00Z"
  },
  "links": [
    {
      "rel": "root",
      "href": "../catalog.json",
      "type": "application/json"
    },
    {
      "rel": "parent",
      "href": "../catalog.json",
      "type": "application/json"
    }
  ],
  "assets": {
    "data": {
      "href": "../data/alphaearth_2025.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "roles": []
    }
  }
}
That folder is a complete, small, cloud-native dataset about one lake. In the last lesson we open it all again, starting from catalog.json.

What embeddings can't do

  • One summary per year. No dates within the year, and short events wash out.
  • 10 m squares. A single rooftop or a narrow drain is below what they can see.
  • Inputs change. Satellites fail and are replaced, so the "no change" level drifts from year to year.
  • Learned mostly elsewhere. AlphaEarth's training labels were mostly from the United States. Check results against what you know on the ground.
  • Read the licences. AlphaEarth's embeddings are CC BY 4.0 but the model is closed; TESSERA's embeddings and model are open.

Try it

  1. Change n_clusters and pixels[::20] in the k-means cell. Does learning from every 5th pixel change the groups?
  2. Give the classifier 5, then 100 labels per class. Does it match the curve above?
  3. Make the change map with TESSERA for 2018 and 2025 (em.tessera(2018), normalised with em.unit). Does it agree with AlphaEarth about where things changed?
Made with open data and open-source Python · marimo · Altair