Chromium PBMC 3k: Landscape, Clustergram, and Enrich¶
Cluster the 10x Genomics Chromium PBMC 3k dataset with Scanpy, save the AnnData to Zarr, and explore it with Celldega:
- Landscape reads the cells, clusters, UMAP, and per-gene expression straight from the AnnData Zarr store in the browser, with no DegaFiles. Color cells by a gene, or plot two genes against each other in the XY view.
- Clustergram shows each cluster's
rank_genes_groupsmarkers, clustered hierarchically withMatrixand saved to Zarr too. - A single-cell Clustergram shows the same markers in every cell, so you can see how they vary within each cluster.
- Enrich finds gene sets enriched in a Clustergram selection.
Like a baking show, the recipe comes first and the finished result comes out of the oven: the steps below were already run, and the notebook loads their output from Hugging Face.
This notebook needs Celldega 0.28 or later (pip install --pre celldega until
0.28.0 is released), zarr>=3, and anndata>=0.12,<0.13. The stores use Zarr
format 3: an older Zarr 2 environment can report PathNotFoundError even when
the URL is valid.
import anndata as ad
import scanpy as sc
import celldega as dega
/Users/feni/Documents/celldega/dega/lib/python3.12/site-packages/scanpy/_utils/__init__.py:27: FutureWarning: `__version__` is deprecated, use `importlib.metadata.version('anndata')` instead.
from anndata import __version__ as anndata_version
/Users/feni/Documents/celldega/dega/lib/python3.12/site-packages/scanpy/__init__.py:36: FutureWarning: `__version__` is deprecated, use `importlib.metadata.version('anndata')` instead.
if Version(anndata.__version__) >= Version("0.11.0rc2"):
/Users/feni/Documents/celldega/dega/lib/python3.12/site-packages/scanpy/readwrite.py:15: FutureWarning: `__version__` is deprecated, use `importlib.metadata.version('anndata')` instead.
if Version(anndata.__version__) >= Version("0.11.0rc2"):
# Hugging Face serves each file at its `resolve` URL, like GitHub's raw URLs.
HF_URL = "https://huggingface.co/datasets/broadinstitute/Celldega_Supporting_Data/resolve/main/pbmc3k"
zarr_url = f"{HF_URL}/pbmc3k.zarr"
matrix_url = f"{HF_URL}/pbmc3k_leiden_markers.zarr"
The recipe: cluster with Scanpy¶
The standard Scanpy workflow on the 10x PBMC 3k dataset. The raw counts are kept in a layer: Landscape shows them, the same values DegaFiles store.
from pathlib import Path
import numpy as np
import scanpy as sc
import zarr
data_dir = Path("data/pbmc3k")
sc.settings.datasetdir = data_dir
adata = sc.datasets.pbmc3k()
adata.var_names_make_unique()
sc.pp.filter_cells(adata, min_genes=200)
sc.pp.filter_genes(adata, min_cells=3)
adata.var["mt"] = adata.var_names.str.startswith("MT-")
sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], percent_top=None, log1p=False, inplace=True)
adata = adata[(adata.obs["n_genes_by_counts"] < 2500) & (adata.obs["pct_counts_mt"] < 5)].copy()
adata.layers["counts"] = adata.X.copy()
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000)
sc.tl.pca(adata, mask_var="highly_variable", random_state=0)
sc.pp.neighbors(adata, n_neighbors=10, n_pcs=40, random_state=0)
sc.tl.umap(adata, random_state=0)
sc.tl.leiden(adata, flavor="igraph", directed=False, n_iterations=2, resolution=0.9, random_state=0)
sc.tl.rank_genes_groups(adata, groupby="leiden", method="wilcoxon", use_raw=False)
# Stores the cluster colors in adata.uns["leiden_colors"].
sc.pl.umap(adata, color="leiden", legend_loc="on data")
The recipe: save the AnnData to Zarr for Landscape¶
Landscape reads one gene at a time from a column-compressed (CSC) copy of the
counts: each gene's values are contiguous, so the browser fetches only that gene.
The per-gene mean and max, used to rank the gene bars and scale cell colors, go in
var. Consolidated metadata lets the browser open the store in one request.
adata.layers["counts_csc"] = adata.layers["counts"].tocsc()
adata.var["mean"] = np.asarray(adata.layers["counts_csc"].mean(axis=0)).ravel()
adata.var["max"] = adata.layers["counts_csc"].max(axis=0).toarray().ravel()
# Write every chunk, even all-zero ones: Hugging Face answers 403 rather than
# 404 for a missing file.
with zarr.config.set({"array.write_empty_chunks": True}):
adata.write_zarr(data_dir / "pbmc3k.zarr")
zarr.consolidate_metadata(data_dir / "pbmc3k.zarr")
The recipe: cluster the markers with Matrix¶
Matrix(adata, groupby="leiden") aggregates the cells of every gene into a dot
plot (mean expression as color, the fraction of expressing cells as size) and
reuses the rank_genes_groups markers. cluster(view="rank_genes_groups") adds a
slider for the top markers per cluster, with the full transcriptome as its "all"
stop. write_zarr saves the clustered result.
import celldega as dega
mat = dega.clust.Matrix(adata, groupby="leiden", col_attr=["leiden"], name="pbmc3k_leiden")
mat.norm("row", by="zscore")
mat.cluster(view="rank_genes_groups", levels=[1, 3, 5, 10, 25, 50, 75, 100, 200])
mat.write_zarr(data_dir / "pbmc3k_leiden_markers.zarr")
Both stores were then uploaded to a Hugging Face dataset:
hf upload broadinstitute/Celldega_Supporting_Data data/pbmc3k/pbmc3k.zarr pbmc3k/pbmc3k.zarr --repo-type dataset
hf upload broadinstitute/Celldega_Supporting_Data data/pbmc3k/pbmc3k_leiden_markers.zarr pbmc3k/pbmc3k_leiden_markers.zarr --repo-type dataset
Out of the oven: explore from Hugging Face¶
zarr_url points the Landscape at the AnnData Zarr store. Python only passes the
URL along: the browser reads the store's metadata, clusters, colors, and UMAP, and
then fetches each gene's values when you select it, the same way it reads
DegaFiles. So the Landscape also works in a saved, kernel-free notebook.
Matrix.read_zarr loads the clustered markers without re-clustering.
Click a cluster or a gene to color the UMAP. To plot two genes against each other, select them (shift-click the gene bars) and click XY, or click XY and type them in the autocompleting X and Y inputs. Select Clustergram rows to run Enrich, and click a gene in Enrich to color the cells by it.
landscape = dega.viz.Landscape(
zarr_url=zarr_url,
cell_attr=["leiden"],
cluster_attr="leiden",
dataset_name="PBMC 3k (Scanpy)",
height=600,
nbhd_edit=True,
ini_zoom=-4.5,
ini_x=5000,
ini_y=5000,
)
mat = dega.clust.Matrix.read_zarr(matrix_url)
cgm = dega.viz.Clustergram(matrix=mat, width=470, height=500, viz_mode="dotplot", rank_dim=10)
linked = dega.viz.spatial_clustergram(
landscape,
cgm,
width="500px",
height="750px",
enrich=True,
enrich_kwargs={"inst_lib": "CellMarker_2024"},
)
linked
Single-cell Clustergram¶
The Clustergram above summarizes each cluster as one column. To see how the
markers vary from cell to cell, load the AnnData itself (about 25 MB from Hugging
Face) and build a Matrix with genes as rows and individual cells as columns,
colored by their Leiden cluster. The values are the log-normalized expression in
adata.X, z-scored across cells for each gene. col_entity="cell" marks the
columns as individual cells, so Clustergram selections highlight those cells in
a linked Landscape.
The AnnData carries Scanpy's rank_genes_groups results, so
cluster(view="rank_genes_groups") adds the same RANK slider for the top
markers per cluster. The rows are each cluster's top 50 markers, the slider's
"all" stop, which keeps the matrix small enough to cluster and display.
adata = ad.read_zarr(zarr_url)
# Each cluster's top 50 markers, the RANK slider's "all" stop.
markers = sc.get.rank_genes_groups_df(adata, group=None)
marker_genes = markers.groupby("group", observed=True).head(50)["names"].unique()
mat_cells = dega.clust.Matrix(
adata[:, marker_genes].copy(),
col_attr=["leiden"],
col_entity="cell",
name="pbmc3k_cells",
)
mat_cells.norm("row", by="zscore")
mat_cells.cluster(view="rank_genes_groups", levels=[3, 5, 10, 25])
cgm_cells = dega.viz.Clustergram(matrix=mat_cells, width=470, height=500, rank_dim=10)
# A second Landscape of the same store, so each Clustergram has its own UMAP.
landscape_cells = dega.viz.Landscape(
zarr_url=zarr_url,
cell_attr=["leiden"],
cluster_attr="leiden",
dataset_name="PBMC 3k (Scanpy)",
height=600,
ini_zoom=-4.5,
ini_x=5000,
ini_y=5000,
)
linked_cells = dega.viz.spatial_clustergram(
landscape_cells,
cgm_cells,
width="500px",
height="750px",
enrich=True,
enrich_kwargs={"inst_lib": "CellMarker_2024"},
)
Select a column dendrogram branch to highlight its cells in the UMAP, or click a Leiden bar above the columns to highlight that cluster's cells. Select a gene dendrogram group or crop rows to color the cells by those genes and send them to Enrich: gene groups that separate subsets of cells within a cluster are a starting point for finding cell subtypes or states. Move the RANK slider to show more or fewer markers per cluster.
linked_cells
Selected cells¶
With a running kernel, get_selected_cells() returns the IDs of the selected
cells: a cluster, a Clustergram dendrogram selection, or the cells inside a drawn
gate. Pass the AnnData loaded above to subset it to those cells:
selected = landscape.get_selected_cells(adata)