Atera Pre-process with H&E¶
Turn a 10x Genomics Atera WTA output bundle into Celldega DegaFiles, and add the post-run H&E image as an aligned background. The DegaFiles used in the Brain Hippocampus, Breast Cancer FF and Colorectal Cancer H&E tutorials were made this way.
Steps:
- Download the bundle, H&E image and H&E alignment matrix
- Extract the files Celldega needs
- Pre-process into DegaFiles (optionally with parquet row groups)
- Convert the H&E image so libvips can read it
- Add the aligned H&E image
- View the result in a Landscape
Requirements: pip install "celldega[pre]>=0.29" and a working libvips installation. tifffile, used to convert the H&E image, is installed with Celldega.
Disk space: Atera bundles are large. The hippocampus _outs.zip is 49 GB (colorectal 75 GB, breast 89 GB), and its extracted files take about 28 GB more, before the DegaFiles are written. On an Apple M3 Max with 64 GB of RAM, the hippocampus run below took about 30 minutes, its largest process used about 31 GB of memory, and the DegaFiles, H&E layer included, take 10.8 GB in about 150,000 files. See Atera Pre-process for the run time and memory of a larger region.
1. Download¶
Datasets used in the Celldega documentation (10x Genomics Atera, Atera Onboard Analysis 1.1.1):
| Tissue | Sample name | 10x dataset page | DegaFiles in the docs |
|---|---|---|---|
| Human brain hippocampus, FF | Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF |
Fresh Frozen Human Brain | celldega_atera-wta-ff-human-nd-brain |
| Human breast cancer, FF | Atera_v1_WTA_S4C_Breast_Cancer_FF |
Fresh Frozen Human Breast Cancer | celldega_atera-wta-ff-human-breast-cancer |
| Human colorectal cancer, FF | Atera_v1_WTA_S4B_Colorectal_Cancer_FF |
Fresh Frozen Human Colorectal Cancer | celldega_atera-wta-ff-human-crc |
The original data are licensed under CC BY 4.0.
Each sample has four downloads on its 10x Genomics dataset page. Which ones Celldega needs:
| File | Needed? | Purpose |
|---|---|---|
<sample>_outs.zip |
yes | the Atera output bundle (cells, transcripts, cell-by-gene matrix, 2D morphology images) |
<sample>_he_image.ome.tif |
yes | the post-run H&E image (listed on the dataset page as Supplemental: Post-Run H&E Image (OME-TIFF)). This is the image shown as the H&E background. |
<sample>_he_imagealignment.csv |
yes | the H&E alignment: a headerless 3x3 affine matrix mapping H&E pixels to the reference (morphology) image pixels |
<sample>_keypoints.csv |
no | the matched keypoints used by 10x to compute the alignment; informational only, Celldega does not read it |
Use the _he_image.ome.tif that belongs to the same sample as the bundle, together with that sample's own _he_imagealignment.csv; each alignment matrix is specific to its image.
sample = "Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF" # change to process another dataset
data_root = "data/atera" # bundle is extracted to data_root/sample
download_dir = f"data/atera_dl/{sample}"
url_bundle = f"https://s3-us-west-2.amazonaws.com/10x.files/samples/atera/1.1.1/{sample}/{sample}_outs.zip"
url_cf = f"https://cf.10xgenomics.com/samples/atera/1.1.1/{sample}/{sample}"
he_image = f"{download_dir}/{sample}_he_image.ome.tif"
he_alignment = f"{download_dir}/{sample}_he_imagealignment.csv"
%%bash -s "$download_dir" "$url_bundle" "$url_cf"
# -C - resumes interrupted downloads (the bundles are large)
mkdir -p "$1" && cd "$1"
curl -sSf -C - -O "$2"
curl -sSf -C - -O "$3_he_image.ome.tif"
curl -sSf -C - -O "$3_he_imagealignment.csv"
2. Extract¶
Celldega reads cells.zarr.zip, transcripts.zarr.zip, cell_feature_matrix.zarr.zip directly (leave them zipped) and the 2D morphology images in morphology_2d/. The 3D morphology stack (morphology_3d/, 21 GB for the hippocampus) is not used and is skipped below. The bundle must end up in data_root/sample.
%%bash -s "$download_dir" "$data_root" "$sample"
mkdir -p "$2/$3"
unzip -q -o "$1/$3_outs.zip" -x 'morphology_3d/*' -d "$2/$3"
# optional: free up space once extracted
# rm "$1/$3_outs.zip"
3. Pre-process into DegaFiles¶
dega.pre.main detects the Atera bundle and writes the DegaFiles (image pyramids, transcript tiles, cell boundary tiles, gene matrices, ...).
Choose one of two layouts:
use_row_groups=False(default): one parquet file per tile. Works well on static hosts such as GitHub Pages.use_row_groups=True: tiles are stored as row groups in a small number of chunked parquet files (max_row_groups_per_fileper file). Fewer files, suited to hosting such as Hugging Face.
import celldega as dega
use_row_groups = False # True -> row-group DegaFiles
suffix = "row_groups" if use_row_groups else "tiles"
path_dega_files = f"data/dega_files/{suffix}/{sample}"
dega.pre.main(
sample=sample,
data_root_dir=data_root,
tile_size=250,
image_tile_layer="all",
path_dega_files=path_dega_files,
max_workers=8, # parallel tile generation; adjust to your machine
use_row_groups=use_row_groups,
max_row_groups_per_file=400, # only used when use_row_groups=True
)
Starting preprocessing for sample: Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF
Created directory: data/dega_files/tiles/Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF
========Write xenium transform file from the Zarr folder========
Transformation matrix saved to 'data/dega_files/tiles/Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF/micron_to_image_transform.csv'.
========Check if all required files or directories exist========
All required files or directories for technology 'Xenium' are present in 'data/atera/Atera_v1_WTA_S4A_Brain_Hippocampus_ND_FF'.
========Make meta cells in pixel space========
meta_cell after _load_meta_cell_by_technology
center_x center_y name
cell_id
aaaaokel-1 2328.647461 926.136536 aaaaokel-1
aaacmjlk-1 2321.163330 1014.239014 aaacmjlk-1
aaacobbg-1 2334.642822 939.636719 aaacobbg-1
aaadbnfg-1 2336.228516 925.882812 aaadbnfg-1
aaaedpnn-1 2333.466309 917.556702 aaaedpnn-1
Done.
========Create cluster gene expression (df_sig)========
Cluster-specific gene expression signatures saved successfully.
========Write meta gene files========
cbg is a sparse DataFrame. Proceeding with sparse operations.
Calculating mean expression
Calculating variance
All meta gene files are succesfully saved.
Processing gene 0: A1BG
Processing gene 1000: ARMC9
Processing gene 2000: CALCB
Processing gene 3000: CLDN17
Processing gene 4000: DGKE
Processing gene 5000: FADS3
Processing gene 6000: GLUD1
Processing gene 7000: IFIT5
Processing gene 8000: KRTAP4-1
Processing gene 9000: MFNG
Processing gene 10000: NOD1
Processing gene 11000: PATL2
Processing gene 12000: PRAMEF9
Processing gene 13000: RNASE3
Processing gene 14000: SLC25A26
Processing gene 15000: SUGCT
Processing gene 16000: TNFRSF14
Processing gene 17000: VSTM2A
Processing gene 18000: ZPBP2
Processing gene 19000: NegControlCodeword_20208
Processing gene 20000: NegControlCodeword_21208
Processing gene 21000: NegControlCodeword_31607
Processing gene 22000: NegControlCodeword_42006
Processing gene 23000: NegControlCodeword_52405
Processing gene 24000: NegControlCodeword_9419
All gene-specific parquet files are successfully saved.
========Create clusters and meta clusters files========
Cell clusters and meta cluster files created successfully.
======== Image Tiles========
========Generating image tiles========
------ xenium
LUT: found 'Suggested LUT max' for 1 annotation(s); matched to channel indices: {0: 3186.0}
dapi: ch0000_dapi.ome.tif
generating dapi image tiles ...
dapi: scaling to display max 15930
bound: ch0001_atp1a1_cd45_e-cadherin.ome.tif
generating bound image tiles ...
bound: scaling to display max 51780
rna: ch0002_18s.ome.tif
generating rna image tiles ...
rna: scaling to display max 69445
prot: ch0003_alphasma_vimentin.ome.tif
generating prot image tiles ...
prot: scaling to display max 25315
Image tiles created successfully.
======== Transcript Tiles========
Using disk-backed traditional transcript tiles (streaming source).
Processing chunks: 22it [00:41, 1.88s/it]
Grid: 246 x 171 = 42066 tiles (streaming → per-tile parquets) Spilling per-tile transcript batches (streaming)...
Tile spill: 100%|██████████| 22/22 [06:22<00:00, 17.37s/it] Writing transcript tile parquets: 100%|██████████| 246/246 [06:23<00:00, 1.56s/it]
Wrote 33212 non-empty transcript tile parquet files
tile bounds: {'x_min': 0.0, 'x_max': 61370.41, 'y_min': 0.0, 'y_max': 42549.28}
======== Cell Boundary Tiles ========
========Create cell boundary spatial tiles========
technology Xenium
Processing coarse tiles: 100%|██████████| 25/25 [00:21<00:00, 1.14it/s]
Done. ========Save landscape parameters======== Done. Preprocessing completed successfully.
4. Convert the H&E image for libvips¶
The Atera H&E OME-TIFF stores its red, green and blue values as three separate planes (planar configuration 2, shape 3 x height x width) with a built-in image pyramid. libvips, which Celldega uses to build the H&E tile pyramid, cannot read this layout and fails with
tiff2vips: tiled separate planes not supported
So convert the full-resolution image once to a standard interleaved (height x width x 3) tiled RGB TIFF. This does not change the pixels or their coordinates, so the alignment CSV still applies. Reading the image needs enough RAM for the uncompressed image: the hippocampus image is 3 x 45,694 x 34,152 pixels, about 4.7 GB, and the conversion needs about twice that.
import numpy as np
import tifffile
he_interleaved = f"{download_dir}/{sample}_he_interleaved.tif"
with tifffile.TiffFile(he_image) as tif:
page = tif.pages[0] # full-resolution image
assert page.planarconfig == 2 and page.shape[0] == 3 and page.dtype == np.uint8, (
page.shape, page.dtype, page.planarconfig
)
rgb = page.asarray() # (3, H, W)
rgb = np.ascontiguousarray(np.moveaxis(rgb, 0, -1)) # (H, W, 3)
tifffile.imwrite(
he_interleaved, rgb, photometric="rgb", tile=(512, 512),
compression="zlib", predictor=True, bigtiff=True,
)
del rgb
5. Add the aligned H&E image¶
dega.pre.add_aligned_he resamples the H&E image onto the reference (fluorescence) image canvas and adds a WebP DeepZoom pyramid to an existing DegaFiles folder. A Fluorescence / H&E selector then appears in the viewer.
- Pass the converted H&E image from step 4 (the interleaved TIFF), not the original planar OME-TIFF.
alignmentis the sample's_he_imagealignment.csv: a headerless 3x3 affine matrix mapping H&E pixels to reference-image pixels.- The H&E image must be 8-bit RGB or RGBA.
- An existing H&E layer is not overwritten.
- Run this for each DegaFiles folder you create (for example both the tile and row-group versions).
dega.pre.add_aligned_he(
path_dega_files,
he_interleaved, # converted image from step 4
alignment=he_alignment, # <sample>_he_imagealignment.csv
)
{'name': 'h_and_e',
'button_name': 'H&E',
'image_format': '.webp',
'alignment': [[0.0027415923126132453, 1.289035480320539, 2898.1812485916216],
[-1.289035480320539, 0.0027415923126132453, 40318.17766101257],
[0.0, 0.0, 1.0]]}
6. View¶
Serve the DegaFiles folder locally and open it in a Landscape. Reload the Landscape after adding the H&E image to see the Fluorescence / H&E selector.
dega.viz.Landscape(
technology="Xenium",
base_url=f"http://localhost:{dega.viz.get_local_server()}/{path_dega_files}",
)
Hosting¶
- GitHub: push the one-file-per-tile DegaFiles folder to a repository. Large folders need several commits;
scripts/upload_degafiles_batched.pyin the Celldega repository pushes an existing DegaFiles Git repository in bounded, resumable batches. - Hugging Face: upload the row-group DegaFiles folder to a dataset repository.
Point base_url at the hosted folder to share the Landscape, for example base_url="https://raw.githubusercontent.com/<org>/<repo>/main".