Atera kidney: review candidate cells with Yearbook¶
Use a tissue-wide Landscape for context, then send candidate cell IDs to a paginated Yearbook. Select the portraits that deserve a closer look and get their IDs back in Python.
Why glomerular endothelial cells and podocytes? In the glomerulus, podocytes wrap their foot processes around capillaries lined by endothelial cells, so the two cell types sit right next to each other. They are otherwise very different cells with distinct expression profiles: EMCN (endomucin) marks endothelial cells and NPHS2 (podocin) marks podocytes, and podocin is not expected in an endothelial cell. An endothelial cell that contains podocin transcripts is therefore a potential doublet: an endothelial cell and a podocyte segmented as one cell (a segmentation merge), or a few transcripts from a neighboring podocyte assigned to the wrong cell. These are review candidates, not doublet calls: marker counts alone cannot distinguish technical artifacts from biology.
The data are the public 10x Genomics Atera WTA FFPE human non-diseased kidney dataset. We use the same prepared DegaFiles as the Atera kidney tutorial. The analysis downloads cell IDs and positions, cluster labels, two sparse gene vectors, and cell areas (about 30 MB in total); the viewers stream images and spatial tiles as needed.
This example requires Celldega 0.28.0a6 or later:
pip install --pre "celldega>=0.28.0a6".
import os
from pathlib import Path
from urllib.request import urlretrieve
import json
# Load the front end that matches this release.
os.environ["CELLDEGA_ESM_VERSION"] = "0.28.0-alpha.6"
import anndata as ad
import fsspec
import h5py
from IPython.display import Markdown, display
import numpy as np
import pandas as pd
from scipy import sparse
import celldega as dega
SAMPLE = "Atera_v1_WTA_Kidney_non_diseased_FFPE"
DATA_REVISION = "3a46ab4db67821eef3b8ff8d1c3704fb7b3bb48f"
BASE_URL = f"https://raw.githubusercontent.com/broadinstitute/Celldega_{SAMPLE}/{DATA_REVISION}"
ANNDATA_URL = f"https://huggingface.co/datasets/broadinstitute/Celldega_Supporting_Data/resolve/main/{SAMPLE}.h5ad"
cwd = Path.cwd().resolve()
repo = next((p for p in [cwd, *cwd.parents] if (p / "pyproject.toml").is_file()), None)
cache = repo / "data/atera_yearbook_review" if repo else cwd / "data/atera_yearbook_review"
cache.mkdir(parents=True, exist_ok=True)
for relative in ["landscape_parameters.json", "micron_to_image_transform.csv"]:
path = cache / relative
if not path.exists():
urlretrieve(f"{BASE_URL}/{relative}", path)
parameters = json.loads((cache / "landscape_parameters.json").read_text())
transform = np.loadtxt(cache / "micron_to_image_transform.csv")
1. Inspect the tissue context¶
This Landscape shows the whole section with its original default clusters. In the
glomeruli, cluster 16 contains podocytes and cluster 20 glomerular endothelial cells.
Type NPHS2, EMCN (comma-separated) in the gene search box to show both markers'
transcripts at once; the Yearbook below will focus on a small set of cells.
landscape = dega.viz.Landscape(
technology="Xenium", base_url=BASE_URL, transform=transform,
dataset_name="Atera kidney — tissue context", width=0, height=600,
max_tiles_to_view=25,
)
landscape.layout.width = "100%"
landscape
2. Select large endothelial cells that contain podocin transcripts with dega.select¶
Build a small AnnData with EMCN and NPHS2 counts, the default cluster labels, and the
cell area for all 399,120 cells. This pinned DegaFiles export uses numeric row
positions in its sparse gene files; the observation index below preserves the
original cell IDs from cell_metadata.parquet. Cell areas come from the obs table
of the annotated counts H5AD;
only the cell IDs and areas are read over HTTP, not the whole 1.31 GiB file.
Selector.attr(...) and Selector.gene(...) build predicates, & combines them, and
selector.samplers.rank(...) orders the matching cells. These thresholds are
exploratory settings that you can change. See the
Select API for more
query and sampling options.
MARKERS = ["NPHS2", "EMCN"] # podocyte, endothelial
for relative in [
"cell_metadata.parquet",
"cell_clusters/cluster.parquet",
*[f"cbg/{gene}.parquet" for gene in MARKERS],
]:
path = cache / relative
path.parent.mkdir(parents=True, exist_ok=True)
if not path.exists():
urlretrieve(f"{BASE_URL}/{relative}", path)
cell_ids = pd.Index(
pd.read_parquet(cache / "cell_metadata.parquet", columns=["name"])["name"].astype(str),
name="cell_id",
)
assert cell_ids.is_unique
assert parameters["use_int_index"], "This example expects the pinned indexed DegaFiles."
columns = []
for gene in MARKERS:
frame = pd.read_parquet(cache / "cbg" / f"{gene}.parquet")
rows = frame.index.to_numpy(dtype=np.int64)
assert ((rows >= 0) & (rows < len(cell_ids))).all()
columns.append(sparse.csc_matrix(
(frame[gene].to_numpy(), (rows, np.zeros(len(rows), dtype=np.int64))),
shape=(len(cell_ids), 1),
))
cells = ad.AnnData(
X=sparse.hstack(columns, format="csr"),
obs=pd.DataFrame(index=cell_ids),
var=pd.DataFrame(index=MARKERS),
)
clusters = pd.read_parquet(cache / "cell_clusters/cluster.parquet")["cluster"]
cells.obs["cluster"] = clusters.astype(str).reindex(cells.obs_names)
area_path = cache / "cell_area.parquet"
if not area_path.exists():
# Read only the cell IDs and areas from the remote H5AD (about 20 MB of 1.31 GiB).
with (
fsspec.open(ANNDATA_URL, block_size=2**20, cache_type="blockcache") as remote,
h5py.File(remote, "r") as h5,
):
pd.DataFrame(
{"cell_area": h5["obs/cell_area"][:]},
index=pd.Index(h5["obs/_index"].asstr()[:], name="cell_id"),
).to_parquet(area_path)
cells.obs["cell_area"] = pd.read_parquet(area_path)["cell_area"].reindex(cells.obs_names)
assert cells.obs[["cluster", "cell_area"]].notna().all().all()
print(f"{cells.n_obs:,} cells; {cells.n_vars} marker genes, clusters, and cell areas; original cell IDs retained")
399,120 cells; 2 marker genes, clusters, and cell areas; original cell IDs retained
ENDOTHELIAL = "20" # default cluster of glomerular endothelial cells (podocytes are "16")
selector = dega.select.Selector(cells)
endothelial = selector.attr("cluster") == ENDOTHELIAL
podocin = selector.gene("NPHS2") >= 2
# Larger than 90% of endothelial cells.
area_cutoff = cells.obs.loc[cells.obs["cluster"] == ENDOTHELIAL, "cell_area"].quantile(0.9)
large_cells = selector.attr("cell_area") >= area_cutoff
candidate_review = selector.select(
query=endothelial & podocin & large_cells,
sampler=selector.samplers.rank(attr=selector.attr("cell_area"), n=100, by="high"),
)
# To sample the matching candidates instead, use:
# sampler=selector.samplers.random(n=100, seed=0)
print(f"cell area >= {area_cutoff:.0f} µm²: {candidate_review.candidate_count:,} candidates; "
f"{len(candidate_review)} cells for review")
candidate_review.to_dataframe().head()
cell area >= 70 µm²: 232 candidates; 100 cells for review
| id | rank | score | |
|---|---|---|---|
| 0 | lahifclf-1 | 0 | 149.862610 |
| 1 | jmlonpne-1 | 1 | 144.399109 |
| 2 | enbjaomo-1 | 2 | 143.721802 |
| 3 | lgojfnkd-1 | 3 | 138.122849 |
| 4 | mcaabbga-1 | 4 | 133.472092 |
The counts support this reasoning. The two cell types are close: 87% of endothelial cells have a podocyte within 20 µm (centroid to centroid). NPHS2 is podocyte-specific: outside podocytes and endothelial cells, only 0.2% of cells have at least 2 NPHS2 counts, but 26% of endothelial cells (1,091 of 4,118) do. Endothelial cells with podocin transcripts are also larger than other endothelial cells (median 52 vs. 40 µm², about the size of a podocyte), consistent with some being merged with a podocyte. The query keeps those larger than 90% of endothelial cells (at least 70 µm²): 232 candidates, of which the 100 largest are shown.
EMCN is the endothelial marker here because it is both strong and specific: at least 2 counts in 91% of glomerular endothelial cells (and 98% of the 100 candidates), compared with 48% for FLT1 (VEGF receptor 1). VWF, a common endothelial marker, is low in glomerular capillaries (0.1 counts per cell on average in cluster 20). The query itself identifies endothelial cells by their cluster.
3. Review portraits across pages¶
Pass the candidate cell IDs to Yearbook. Click the checkbox at the lower right of a
portrait to select that cell. To view both markers, type NPHS2, EMCN in the
Yearbook's gene search box; only those genes' transcripts are shown, each in its own
color. A single boundary enclosing two nuclei suggests a segmentation merge; NPHS2
transcripts concentrated along the edge an endothelial cell shares with a podocyte
suggest spillover.
review_selection = candidate_review
candidate_ids = review_selection.names()
yearbook = dega.viz.Yearbook(
name="kidney_candidate_review", base_url=BASE_URL, cells=candidate_ids,
rows=3, cols=5, portrait_size_um=50,
dataset_name="Atera kidney — candidate review", width=0, height=850,
)
yearbook.layout.width = "100%"
display(Markdown(f"**Genes to highlight: {', '.join(MARKERS)}** "
"(type them into the Yearbook's gene search box)"))
yearbook
Genes to highlight: NPHS2, EMCN (type them into the Yearbook's gene search box)
4. Get the selected cell IDs¶
yearbook.selected_cells returns the IDs of the cells you selected. Run this cell in
a live notebook after selecting portraits.
yearbook.selected_cells
[]
Optional: sample high-count cells using observation metadata¶
A second dega.select example combines an obs threshold with seeded random
sampling. The annotated counts H5AD
contains total_counts in obs. If you already downloaded it for the full kidney
tutorial, the code below reads only metadata in backed mode, not the count matrix.
It deliberately does not download that 1.31 GiB file just for this example.
Cells in the highest 1% of total counts are another review queue, not a doublet
classification. To inspect this alternative, replace review_selection = candidate_review
with review_selection = high_count_review in the Yearbook cell and rerun it.
source_path = (
repo / "notebooks/data/atera_data" / SAMPLE / f"{SAMPLE}.h5ad"
if repo else cwd / "data" / f"{SAMPLE}.h5ad"
)
high_count_review = None
if source_path.exists():
source = ad.read_h5ad(source_path, backed="r")
qc_cells = ad.AnnData(obs=source.obs[["total_counts"]].copy())
source.file.close()
qc_selector = dega.select.Selector(qc_cells)
count_cutoff = qc_cells.obs["total_counts"].quantile(0.99)
high_count_review = qc_selector.select(
query=qc_selector.attr("total_counts") >= count_cutoff,
sampler=qc_selector.samplers.random(n=100, seed=0),
)
print(f"total_counts >= {count_cutoff:g}: {high_count_review.candidate_count:,} candidates; "
f"{len(high_count_review)} sampled for review")
else:
print("Optional high-count example skipped: no cached H5AD. The marker-based review above is ready.")
total_counts >= 9225: 3,993 candidates; 100 sampled for review