Lesson 1 · 11:35–12:10
What did the lake look like in 2017, and now?
The Sentinel-2 satellites photograph every patch of land on Earth about every
five days, in 10 m pixels, and have done so since 2015. The archive is free and enormous. To
compare two pictures of one lake we need to find them in it, and then read only the lake.
Two inventions make that easy, and they are the whole of this lesson: a catalog you can
search, and files you can read a piece of.
Find the pictures: a STAC catalog
STAC (SpatioTemporal Asset Catalog) describes every picture as a small record called an item: where it was taken, when, how cloudy it was, and links to the actual files, called assets. Items are grouped into collections (here, "Sentinel-2, with the atmosphere removed"), and collections into a catalog. Element 84 runs a free one, Earth Search, over the Sentinel-2 archive on Amazon's cloud. Let's ask it for every picture of our study area, the 7 × 5 km around the lake, since 2017.from pystac_client import Client
earth_search = Client.open("https://earth-search.aws.element84.com/v1")
search = earth_search.search(
collections=["sentinel-2-l2a"], # Sentinel-2, atmosphere removed
bbox=STUDY_LONLAT, # west, south, east, north, in degrees
datetime="2017-01-01/2025-12-31",
)
on_tile = [i for i in search.items() if "43PGQ" in i.id] # the tile with the lake
by_version = sorted(on_tile, key=lambda i: i.properties["s2:sequence"])
per_day = {i.datetime.date(): i for i in by_version} # one picture per day, the newest
items = sorted(per_day.values(), key=lambda i: i.datetime)
scenes = pd.DataFrame({
"date": [i.datetime.replace(tzinfo=None) for i in items],
"cloud": [i.properties["eo:cloud_cover"] for i in items],
})
f"{len(items)} pictures found"
606 pictures found
Here is one of them, picked at random. A Sentinel-2 picture covers 110 × 110 km; the white box
is our study area. To be quick we read only a small zoomed-out copy that is stored inside the
file (more on that below). Run the cell again for another picture.
import random
import rasterio
one = random.choice(items)
with rasterio.open(one.assets["visual"].href, overview_level=3) as tile:
whole = tile.read() # 3 colours x 686 x 686 pixels
edges = tile.bounds
cloud = one.properties["eo:cloud_cover"]
charts.tile_preview(whole.transpose(1, 2, 0), edges, STUDY,
title=f"{one.datetime:%d %B %Y}, {cloud:.0f}% cloud")
From June to September the monsoon hides Bengaluru: a picture almost every week, and almost none
of them clear. February is reliably clear, so we compare one February with another.
Inside one item
Here is the clearest picture from February 2017 and the files it links to. Each band of light the satellite measures is its own file, plus a ready-made true-colour picture calledvisual.def clearest(year, month=2):
"""The least cloudy picture of one month."""
same_month = [i for i in items
if i.datetime.year == year and i.datetime.month == month]
return min(same_month, key=lambda i: i.properties["eo:cloud_cover"])
feb_2017 = clearest(2017)
pd.DataFrame([{"asset": key, "what it is": a.title, "file": a.href.split("/")[-1]}
for key, a in feb_2017.assets.items() if a.href.endswith(".tif")])
| asset | what it is | file |
|---|---|---|
| aot | Aerosol optical thickness (AOT) | AOT.tif |
| blue | Blue (band 2) - 10m | B02.tif |
| coastal | Coastal aerosol (band 1) - 60m | B01.tif |
| green | Green (band 3) - 10m | B03.tif |
| nir | NIR 1 (band 8) - 10m | B08.tif |
| nir08 | NIR 2 (band 8A) - 20m | B8A.tif |
| nir09 | NIR 3 (band 9) - 60m | B09.tif |
| red | Red (band 4) - 10m | B04.tif |
| rededge1 | Red edge 1 (band 5) - 20m | B05.tif |
| rededge2 | Red edge 2 (band 6) - 20m | B06.tif |
| rededge3 | Red edge 3 (band 7) - 20m | B07.tif |
| scl | Scene classification map (SCL) | SCL.tif |
| swir16 | SWIR 1 (band 11) - 20m | B11.tif |
| swir22 | SWIR 2 (band 12) - 20m | B12.tif |
| visual | True color image | TCI.tif |
| wvp | Water vapour (WVP) | WVP.tif |
Read only the lake: a Cloud-Optimized GeoTIFF
Thevisual file covers a 110 × 110 km square. We want 7 × 5 km of it. A Cloud-Optimized
GeoTIFF (COG) is an ordinary image file laid out so that a program can ask the server for one
byte range at a time: first the header, which says where everything is, then only the tiles it
needs. Your laptop never downloads the rest.from rasterio.windows import from_bounds
href = feb_2017.assets["visual"].href
with net.count_bytes() as fetched, rasterio.open(href) as image:
window = from_bounds(*STUDY, image.transform) # our box, in the file's metres
lake_2017 = image.read(window=window) # 3 colours x 500 x 700 pixels
file_mb = net.size_mb(href)
f"Read {fetched.mb:.1f} MB in {fetched.requests} requests from a {file_mb:.0f} MB file"
Read 5.3 MB in 2 requests from a 363 MB file
At full detail our box touches two of the file's 121 tiles. The file also keeps zoomed-out copies
of itself, called overviews, each a quarter the size of the last; a web map showing the
whole tile reads one small overview tile instead of hundreds of megabytes. Hover a square to see
how many bytes it holds.
2017 and now
The same two steps for February 2025, then the two pictures side by side.feb_2025 = clearest(2025)
with rasterio.open(feb_2025.assets["visual"].href) as _image:
lake_2025 = _image.read(window=from_bounds(*STUDY, _image.transform))
The lake is the large shape left of centre. In 2017 a mat of water hyacinth and weeds covers
almost all of it, fed by the sewage that flows in. By 2025 the western half has been drained
and dug out, and open water fills the eastern basin. Around it, the grey blocks of offices and
apartments have spread east and south. The straight grey line at the top is the HAL airport
runway; the diagonal is the Outer Ring Road.
Where catalogs live
Earth Search is one of many. STAC Index lists about 150 public catalogs. A few you will meet again:- Microsoft Planetary Computer: 130+
collections; reading files needs a free, automatic signature (
planetary_computer.sign). - NASA CMR-STAC: all of NASA's archive; searching is open, downloading needs a free Earthdata login.
- Source Cooperative: a shared space where anyone can publish open data in cloud-native formats. The embeddings of lesson 4 live there.
- From India: CoRE Stack, which we open in lesson 3, and ISRO's Bhoonidhi, which speaks STAC but needs a login.
Start your own catalog
Now we turn the 2025 picture into our own cloud-native data: write it as a COG, describe it with a STAC item, and put the item in a catalog.rio-stac reads the file and fills in its footprint,
size and projection for us.import pystac
from rasterio.transform import from_origin
from rio_stac import create_stac_item
# 1. the picture, as a Cloud-Optimized GeoTIFF on our 10 m grid
path = mycatalog.DATA / "lake_2025.tif"
grid = dict(crs=CRS, transform=from_origin(STUDY[0], STUDY[3], 10, 10),
width=700, height=500, count=3, dtype="uint8")
with rasterio.open(path, "w", driver="COG", compress="deflate", **grid) as out:
out.write(lake_2025)
# 2. a STAC item describing it; rio-stac reads the footprint, size and projection
item = create_stac_item(str(path), id="lake_2025", input_datetime=feb_2025.datetime,
asset_name="data", asset_media_type=pystac.MediaType.COG,
with_proj=True)
item.properties["title"] = "Bellandur lake, February 2025 (Sentinel-2)"
# 3. a catalog holding the item, saved as a folder of JSON files. The first time we
# make a new one; after that we open it, so items from later lessons are kept
catalog_file = mycatalog.FOLDER / "catalog.json"
if catalog_file.exists():
catalog = pystac.Catalog.from_file(str(catalog_file))
catalog.remove_item("lake_2025")
else:
catalog = pystac.Catalog(id="bellandur",
description="What we found out about Bellandur lake")
catalog.add_item(item)
catalog.normalize_hrefs(str(mycatalog.FOLDER))
catalog.make_all_asset_hrefs_relative() # links that still work when copied
catalog.save(pystac.CatalogType.SELF_CONTAINED)
mycatalog.items()
| item | title | date | file | type |
|---|---|---|---|---|
| rain_flood_night | Rain over Bengaluru, night of 4-5 Sep 2022 (IMERG) | 2022-09-04 | rain_flood_night.zarr | application/vnd.zarr |
| waterbodies | Water bodies and their water area per year (CoRE Stack) | 2025-01-01 | waterbodies.parquet | application/vnd.apache.parquet |
| watersheds | Micro-watersheds around the lake (CoRE Stack) | 2025-01-01 | watersheds.parquet | application/vnd.apache.parquet |
| buildings | Building footprints east of the lake (Overture Maps) | 2025-01-01 | buildings.parquet | application/vnd.apache.parquet |
| change_2017_2025 | Change of each 10 m square, 2017 to 2025 (AlphaEarth) | 2025-01-01 | change_2017_2025.tif | image/tiff |
| alphaearth_2025 | AlphaEarth Foundations embeddings, 2025 (Google DeepMind) | 2025-01-01 | alphaearth_2025.tif | image/tiff |
| lake_2025 | Bellandur lake, February 2025 (Sentinel-2) | 2025-02-06 | lake_2025.tif | image/tiff |
Your catalog is the folder
my_catalog/. Click through it below: catalog.json links to the
item, lake_2025/lake_2025.json, and the item links to the picture in data/. That is all a
static catalog is.# click a folder to go inside it, click a file to see it below
folder = mo.ui.file_browser(mycatalog.FOLDER, value=catalog.get_self_href(),
multiple=False)
folder
my_catalog/ catalog.json alphaearth_2025/ alphaearth_2025.json buildings/ buildings.json change_2017_2025/ change_2017_2025.json data/ alphaearth_2025.tif 19.6 MB buildings.parquet 1.5 MB change_2017_2025.tif 1.6 MB lake_2025.tif 1.2 MB rain_flood_night.zarr/ 19 kB waterbodies.parquet 78 kB watersheds.parquet 0.4 MB lake_2025/ lake_2025.json rain_flood_night/ rain_flood_night.json waterbodies/ waterbodies.json watersheds/ watersheds.json
catalog.json
{
"type": "Catalog",
"id": "bellandur",
"stac_version": "1.1.0",
"description": "What we found out about Bellandur lake",
"links": [
{
"rel": "root",
"href": "./catalog.json",
"type": "application/json"
},
{
"rel": "item",
"href": "./rain_flood_night/rain_flood_night.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./waterbodies/waterbodies.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./watersheds/watersheds.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./buildings/buildings.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./change_2017_2025/change_2017_2025.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./alphaearth_2025/alphaearth_2025.json",
"type": "application/geo+json"
},
{
"rel": "item",
"href": "./lake_2025/lake_2025.json",
"type": "application/geo+json"
}
]
}
Try it
Each of the three cells below is ready to run. Change the values at the top and run it again.- A place you know. Which months are clear there?
- Near-infrared instead of true colour. Plants reflect near-infrared light strongly: where is the picture brightest?
- The whole tile from a zoomed-out copy. How many megabytes does the whole 110 km tile take?
# Try 1: change the box and the year
my_box = (77.58, 12.96, 77.61, 12.99) # west, south, east, north: around Cubbon Park
my_year = "2024"
found = earth_search.search(collections=["sentinel-2-l2a"], bbox=my_box,
datetime=my_year)
days = {i.datetime.date(): i for i in found.items()} # one picture per day
by_month = pd.DataFrame({"month": [d.month for d in days],
"cloud": [i.properties["eo:cloud_cover"] for i in days.values()]})
(by_month.groupby("month").cloud
.agg(pictures="size", nearly_clear=lambda c: int((c < 10).sum()))
.rename(index=dict(enumerate(charts.MONTHS, start=1))))
| pictures | nearly_clear | |
|---|---|---|
| month | ||
| Jan | 6 | 2 |
| Feb | 6 | 4 |
| Mar | 6 | 6 |
| Apr | 6 | 3 |
| May | 6 | 1 |
| Jun | 6 | 0 |
| Jul | 7 | 0 |
| Aug | 6 | 0 |
| Sep | 5 | 0 |
| Oct | 6 | 0 |
| Nov | 6 | 0 |
| Dec | 6 | 0 |
# Try 2: change the band ("nir", "red", "swir16", ...) or the picture (feb_2017)
band = "nir"
picture = feb_2025
with rasterio.open(picture.assets[band].href) as _image:
values = _image.read(1, window=from_bounds(*STUDY, _image.transform))
charts.image(values, cmap="Greens", label=f"{band}, {picture.datetime:%d %b %Y}")
# Try 3: change the zoom level (0 = half size ... 3 = one sixteenth)
zoom_level = 3
with net.count_bytes() as _fetched, rasterio.open(href, overview_level=zoom_level) as _small:
whole_tile = _small.read()
charts.picture({"tile": whole_tile.transpose(1, 2, 0)}, width=420,
title=f"The whole tile at {whole_tile.shape[2]:,} x {whole_tile.shape[1]:,} pixels",
subtitle=f"read {_fetched.mb:.1f} MB in {_fetched.requests} requests")