error
Lesson 2 · 12:10–12:35
How much rain fell the night the lake overflowed?
On the night of 4 September 2022 it poured on east Bengaluru. By morning the
Outer Ring Road beside Bellandur was a river, lakes had spilled into apartment blocks, and
people were rowing boats past the IT parks. Rain gauges recorded about 131 mm overnight. What
does the satellite record say?
Weather and climate data come as grids: one number for every cell of a map, for every time
step, often for decades. NASA's IMERG rain estimate covers the whole planet in 0.1° cells
(about 11 km), every half hour since 1998. That is half a million maps.
The usual way: one file per year
The India Meteorological Department publishes its gridded rainfall as one file per year, which you download from a web form (theimdlib package automates the form). To get one place you
download the whole country; for 25 years you download 25 files. That is fine for one dataset.
For an archive of petabytes it is not.
Zarr: a grid cut into chunks
Zarr stores a big array as many small compressed pieces, called chunks, each its own file or cloud object, plus a little JSON that describes the whole. A reader opens the JSON, works out which chunks hold what you asked for, and fetches only those. dynamical.org keeps IMERG as a Zarr store on Amazon's cloud and lists it in its own STAC catalog.import icechunk
import pystac
dynamical = pystac.Catalog.from_file("https://stac.dynamical.org/catalog.json")
imerg = dynamical.get_child("nasa-imerg-analysis-late")
storage = icechunk.http_storage(imerg.assets["icechunk-https"].href)
session = icechunk.Repository.open(storage).readonly_session("main")
rain = xr.open_zarr(session.store, chunks=None)["precipitation_surface"]
pd.DataFrame({
"": ["time steps", "latitudes", "longitudes", "size, if loaded", "one chunk"],
"IMERG rain": [f"{rain.shape[0]:,}", rain.shape[1], rain.shape[2],
f"{rain.nbytes / 1e12:.0f} TB", str(rain.encoding["chunks"])],
})
| IMERG rain | |
|---|---|
| time steps | 504,480 |
| latitudes | 1800 |
| longitudes | 3600 |
| size, if loaded | 13 TB |
| one chunk | (1440, 100, 100) |
Thirteen terabytes, and nothing has been downloaded yet: xarray has only read the description.
Each chunk holds 1,440 half-hours (30 days) of a 100 × 100 cell square, so the store is built for
exactly our question: a stretch of time at one place. Let's take the cell over the lake for the
three weeks around the flood.
at_lake = rain.sel(longitude=LAKE[0], latitude=LAKE[1], method="nearest")
weeks = at_lake.sel(time=slice("2022-08-20", "2022-09-15")).load() # bytes move now
flood = (weeks.to_series() * 1800).rename("mm") # mm per second over half an hour
flood.index = flood.index + pd.Timedelta(hours=5, minutes=30) # UTC -> India time
f"{len(flood)} half-hours, {flood.sum():.0f} mm in total"
1296 half-hours, 295 mm in total
Drag across the night of the 4th: about 50 mm fell between seven in the evening and five in the
morning, and the 5th ended with about 70 mm. The gauges caught 131 mm overnight. A satellite estimate averages over an 11 km square and
smooths out the heaviest bursts, so it is useful for patterns and comparisons, not for the
exact number at one street.
Was it unusual? The data pack has the same cell for every day since 1998, and the night's total
for the cells around the city.
How this file was made · making it reads about 0.3 GB of NASA IMERG (28 years at one point), about 15 minutes
The chart reads a small prepared file (0.1 MB): your copy in data/ if you have one, otherwise it downloads once. To make it yourself from the original data, run uv run cngeo datapack build --step rain --force. This is the code that runs:
def imerg(): """NASA IMERG half-hourly rain (0.1 degree), as an xarray Dataset over a cloud Icechunk store.""" import os import icechunk import xarray as xr os.environ.setdefault("ICECHUNK_LOG", "error") repo = icechunk.Repository.open(icechunk.http_storage(IMERG)) return xr.open_zarr(repo.readonly_session("main").store, chunks=None) def rain(force: bool = False): """Rain over the lake: every day since 1998, the half-hours around the 2022 flood, and a map of that night.""" from .config import LAKE out = datapack.path("rain") if out.exists() and not force: return p = imerg()["precipitation_surface"] # mm per second, averaged over each half hour lon, lat = LAKE series = p.sel(latitude=lat, longitude=lon, method="nearest").sel(time=slice("1998-01-01", "2025-12-31")).load() s = series.to_series() * 1800 # mm in each half hour s.index = s.index.tz_localize("UTC").tz_convert("Asia/Kolkata").tz_localize(None) daily = s.resample("1D").sum().rename("mm").to_frame() out.parent.mkdir(parents=True, exist_ok=True) daily.round(1).to_parquet(out) s.loc["2022-08-20":"2022-09-15"].rename("mm").round(2).to_frame().to_parquet(datapack.path("rain_flood")) night = p.sel(time=slice("2022-09-04T13:30", "2022-09-05T03:30"), latitude=slice(13.45, 12.45), longitude=slice(77.15, 78.15)) m = (night.sum("time") * 1800).to_dataframe(name="mm").reset_index()[["latitude", "longitude", "mm"]] m.round(2).to_parquet(datapack.path("rain_map"), index=False)
The heaviest rain fell in a band from the north-west down through the centre of the city,
close to 100 mm. The lake's own cell got about 50. But the centre of Bengaluru drains south-east
through the Koramangala valley into Bellandur, so much of that band's water ran towards the
lake. Which land drains where is the question of the next lesson.
Chunks decide what is fast
The shape of the chunks is a choice the data's publisher makes, and it decides which questions are quick. NASA's POWER project stores the same daily weather twice: once chunked as maps (one day of the whole globe per chunk) and once chunked as time series (40 years of a small square per chunk). Let's ask both the same two questions and time them.import time
power = ("https://nasa-power.s3.us-west-2.amazonaws.com/merra2/"
"{0}/power_merra2_daily_{0}_utc.zarr")
place = dict(lon=LAKE[0], lat=LAKE[1], method="nearest")
questions = {
"one place, every day": lambda t: t.sel(**place).sel(time="2024"),
"one day, every place": lambda t: t.sel(time="2024-05-01",
lat=slice(6, 37), lon=slice(68, 98)),
}
rows = []
for layout, name in [("spatial", "chunked as maps"),
("temporal", "chunked as time series")]:
t2m = xr.open_zarr(net.store(power.format(layout)))["T2M"] # air temperature
for question, ask in questions.items():
start = time.perf_counter()
ask(t2m).load()
rows.append({"layout": name, "question": question,
"chunk shape": str(t2m.encoding["chunks"]),
"seconds": round(time.perf_counter() - start, 1)})
timings = pd.DataFrame(rows)
timings
| layout | question | chunk shape | seconds |
|---|---|---|---|
| chunked as maps | one place, every day | (1, 361, 576) | 26.2 |
| chunked as maps | one day, every place | (1, 361, 576) | 0.5 |
| chunked as time series | one place, every day | (5844, 15, 15) | 2.7 |
| chunked as time series | one day, every place | (5844, 15, 15) | 9.4 |
A year at one place, from the map-shaped store, means fetching 366 global maps to keep one
number from each; from the time-series store it is one chunk. A map of India for one day is the
reverse. Neither layout is wrong; they serve different people. When you publish a grid, ask who
will read it and how.
Two more tools you will meet:
- Virtual Zarr (VirtualiZarr, Kerchunk) writes a small index of where each chunk sits inside existing NetCDF or GRIB files, so an old archive of files can be read like Zarr without copying it.
- Icechunk adds versions to a Zarr store: every update is a commit
you can go back to. The IMERG store above is an Icechunk repository; we read its
mainbranch.
Store your own
We save the flood night over Bengaluru as our own Zarr store, chunked for maps (the whole area in one chunk per half hour), and add it to the catalog.night = rain.sel(time=slice("2022-09-04T13:30", "2022-09-05T03:30"), # 7 pm-9 am
latitude=slice(13.45, 12.45), longitude=slice(77.15, 78.15))
night_mm = (night.load() * 1800).rename("rain_mm").to_dataset().drop_encoding()
zarr_path = mycatalog.DATA / "rain_flood_night.zarr"
maps = night_mm.chunk({"time": 1, "latitude": -1, "longitude": -1}) # a map a chunk
maps.to_zarr(zarr_path, mode="w", zarr_format=3, consolidated=False)
mycatalog.add(zarr_path, title="Rain over Bengaluru, night of 4-5 Sep 2022 (IMERG)",
datetime="2022-09-04T13:30:00", bbox=[77.15, 12.45, 78.15, 13.45])
files = sorted(p.relative_to(zarr_path) for p in zarr_path.rglob("*") if p.is_file())
[f.as_posix() for f in files[:8]]
['latitude/c/0', 'latitude/zarr.json', 'longitude/c/0', 'longitude/zarr.json', 'rain_mm/c/0/0/0', 'rain_mm/c/1/0/0', 'rain_mm/c/10/0/0', 'rain_mm/c/11/0/0']
Each file under
rain_mm/c/ is one chunk: one half-hour map. zarr.json is the description.
Try it
- Change the place in
at_laketo your own neighbourhood and the dates to this year's monsoon. - In the timing cell, ask for ten years at one place instead of one. How long does each layout take now? (Start with the time-series store.)
- Rechunk
night_mmwith{"time": -1, "latitude": 1, "longitude": 1}and save it again. How many files do you get, and which question is that layout good for?