gnomad_qc.v5.annotations.compute_coverage
Compute coverage, allele number, and quality histograms for gnomAD v5 genomes.
v5 genomes = AoU v8 (new) + gnomAD v4 genomes minus the consent-drop set. This script computes per-reference-site AN per project, coverage (gnomAD only), and qual histograms (AoU only) and joins them into the v5 release HT/TSVs.
Execution roles (mutually exclusive; dispatched by early return in main):
ORCHESTRATOR (
--use-batch-fanout/--merge-cov-chunks): never initializes Hail; submits a Hail Batch of relay jobs and returns. Used for the prod-scale AoU compute.WORKER (
--run-chunk/--run-merge): one relay container; initializes Hail and runs exactly one chunk/merge unit. The chunk worker is what callscompute_all_release_stats_per_ref_site.IN-PROCESS PIPELINE (neither): the strict single-job compute (gnomAD consent-drop, dev/test AoU), the setup writers, and the release assembly.
A full prod AoU release is three separate invocations: fan-out compute,
merge-cov-chunks, then the in-process assembly. The setup HTs (downsampling,
group membership, sites, chunk intervals) are written by their own
in-process invocations first; the fan-out reads them from disk. The AoU sample
artifact JSONs (release samples, ID collisions) come from
create_sample_qc_metadata_ht.py --write-vds-sample-artifact-jsons and are
read inside get_aou_vds.
Workflow:
1. (AoU) --write-aou-downsampling-ht
2. --write-group-membership-ht. AoU: one aggregation cell per distinct
membership pattern; every AN group is reconstructed by summing cells.
gnomAD: the consent-drop samples, one boolean per group.
3. --write-vep-context-sites: deduped, telomere/centromere/chrM-stripped
sites HT -- the definition of "every site the compute must cover".
4. Compute: --compute-all-cov-release-stats-ht (strict single job), or
--write-chunk-intervals, then --use-batch-fanout, then
--merge-cov-chunks. Chunk outputs are namespaced by a content hash of
the intervals JSON: rerunning the fan-out retries only missing chunks,
and a regenerated JSON (new layout, new hash) never mixes with old
chunks.
5. --validate-cov-and-an: the merged HT must cover every site exactly.
6. (gnomad) --merge-gnomad-coverage / --merge-gnomad-an: subtract the
consent-drop samples from the release HTs. AN: the samples in the v4
release, from the v4.1 AN release. Coverage: the samples in the v3.0
release, from the 3.0.1 coverage release (the only genomes coverage
ever computed; v4 reused it).
7. (aou) Release: --export-coverage-release-files (gnomAD-only; AoU
computes no coverage stats), --export-an-release-files,
--merge-qual-hists (gnomAD v4 hists reused as-is).
Usage Examples:
# Per-chunk fan-out compute + merge (on batch). Run setup steps 1-3 +
# --write-chunk-intervals first; the fan-out reads those outputs:
python compute_coverage.py --project-name aou --environment batch \\
--use-batch-fanout --partitions-per-chunk 3
python compute_coverage.py --project-name aou --environment batch \\
--merge-cov-chunks --n-partitions 10000
# gnomAD v4 consent-drop subtraction (on dataproc):
hailctl dataproc submit cluster-name compute_coverage.py \\
--project-name gnomad --environment dataproc \\
--merge-gnomad-coverage --merge-gnomad-an
# AoU/gnomAD release assembly (on batch):
python compute_coverage.py --project-name aou --environment batch \\
--export-coverage-release-files --export-an-release-files \\
--merge-qual-hists
usage: gnomad_qc.v5.annotations.compute_coverage.py [-h]
[--project-name {aou,gnomad}]
[--overwrite]
[--environment {rwb,batch,dataproc}]
[--results-environment {rwb,batch,dataproc}]
[--tmp-dir-days TMP_DIR_DAYS]
[--gcp-billing-project GCP_BILLING_PROJECT]
[--experimental]
[--app-name APP_NAME]
[--driver-cores DRIVER_CORES]
[--driver-memory {standard,highmem}]
[--jvm-heap-size JVM_HEAP_SIZE]
[--worker-cores WORKER_CORES]
[--worker-memory {standard,highmem}]
[--n-partitions N_PARTITIONS]
[--cov-and-an-output-suffix COV_AND_AN_OUTPUT_SUFFIX]
[--qual-hists-output-suffix QUAL_HISTS_OUTPUT_SUFFIX]
[--test]
[--test-sample-subset]
[--test-n-partitions TEST_N_PARTITIONS]
[--chrom CHROM]
[--test-region TEST_REGION [TEST_REGION ...]]
[--write-group-membership-ht]
[--write-aou-downsampling-ht]
[--write-vep-context-sites]
[--write-chunk-intervals]
[--validate-cov-and-an]
[--compute-all-cov-release-stats-ht]
[--partitions-for-rep-on-read [PARTITIONS_FOR_REP_ON_READ]]
[--merge-gnomad-coverage]
[--export-coverage-release-files]
[--merge-gnomad-an]
[--export-an-release-files]
[--merge-qual-hists]
[--use-batch-fanout]
[--submit-orchestrator]
[--fanout-retry-passes FANOUT_RETRY_PASSES]
[--merge-cov-chunks]
[--assemble-chrom-coverage]
[--partitions-per-chunk PARTITIONS_PER_CHUNK]
[--read-subintervals-per-chunk READ_SUBINTERVALS_PER_CHUNK]
[--total-partitions TOTAL_PARTITIONS]
[--wave-size WAVE_SIZE]
[--merge-group-size MERGE_GROUP_SIZE]
[--methods-branch METHODS_BRANCH]
[--batch-image BATCH_IMAGE]
[--batch-billing-project BATCH_BILLING_PROJECT]
[--batch-remote-tmpdir BATCH_REMOTE_TMPDIR]
[--chunk-cpu CHUNK_CPU]
[--chunk-memory {lowmem,standard,highmem}]
[--chunk-storage CHUNK_STORAGE]
[--chunk-driver-cores CHUNK_DRIVER_CORES]
[--chunk-driver-memory {standard,highmem}]
[--chunk-worker-cores CHUNK_WORKER_CORES]
[--chunk-worker-memory {standard,highmem}]
[--chunk-attempts CHUNK_ATTEMPTS]
[--merge-cpu MERGE_CPU]
[--merge-memory MERGE_MEMORY]
[--merge-storage MERGE_STORAGE]
[--final-merge-storage FINAL_MERGE_STORAGE]
[--batch-dry-run]
[--run-chunk]
[--chunk-start CHUNK_START]
[--chunk-stop CHUNK_STOP]
[--chunk-output CHUNK_OUTPUT]
[--run-merge]
[--merge-inputs MERGE_INPUTS [MERGE_INPUTS ...]]
[--merge-output MERGE_OUTPUT]
[--merge-coalesce-to MERGE_COALESCE_TO]
Named Arguments
- --project-name
Possible choices: aou, gnomad
Project name.
Default: “aou”
- --overwrite
Overwrite existing hail Tables.
Default: False
- --n-partitions
Partitions for the output Table. Default None: no final naive_coalesce.
- --cov-and-an-output-suffix
Optional suffix before the .ht extension (A/B comparison). Applies to writes and downstream reads; pass consistently.
- --qual-hists-output-suffix
Like –cov-and-an-output-suffix, for the –merge-qual-hists output.
- --write-aou-downsampling-ht
Write v5 downsampling HT.
Default: False
- --write-vep-context-sites
Preprocess vep_context once into the deduped, stripped, locus-keyed sites HT every chunk reads. Prerequisite for the compute. With –test, writes a scoped _test table.
Default: False
- --write-chunk-intervals
Precompute every chunk’s balanced read sub-intervals in one VDS open. REQUIRED before the fan-out. Uses –total-partitions / –partitions-per-chunk / –read-subintervals-per-chunk; –test writes to the test path.
Default: False
- --validate-cov-and-an
Anti-join the sites HT against the merged cov_and_an HT; fail if any site is missing. Run after –merge-cov-chunks.
Default: False
- --compute-all-cov-release-stats-ht
Compute the all sites coverage, allele number, and quality histogram HT.
Default: False
- --partitions-for-rep-on-read
Strict single-job compute only: co-partition the VDS and vep_context reads (shuffle-free join). Flag alone = the 1.1x-native default (recommended for full runs); a value = an explicit count. Honors –chrom.
- --merge-qual-hists
Merge variant quality histograms from AoU v8 and gnomAD v4 genomes.
Default: False
environment configuration
- --environment
Possible choices: rwb, batch, dataproc
Compute environment.
Default: “rwb”
- --results-environment
Possible choices: rwb, batch, dataproc
Override the bucket the release artifacts write to, independent of the compute –environment. Default: prod writes to ‘dataproc’, a –test run stays in the compute environment.
- --tmp-dir-days
Number of days for temp directory retention. Default is 4.
Default: 4
- --gcp-billing-project
Google Cloud billing project for reading requester pays buckets.
Default: “broad-mpg-gnomad”
batch configuration
Optional parameters for batch/QoB backend (only used when –environment=batch).
- --experimental
Route QoB init through hl.experimental.init and attach the driver to an existing Hail Batch (HAIL_BATCH_ID; raises if unset). Without it, each invocation creates its own Batch.
Default: False
- --app-name
Job name for batch/QoB backend.
- --driver-cores
Number of cores for the driver node. Pass a power of two between 0.25 and 16 (as a string, e.g. ‘2’ or ‘0.5’).
- --driver-memory
Possible choices: standard, highmem
Memory type for driver node. ‘lowmem’ is not offered: Hail Batch rejects it for JVM jobs (‘400: jvm jobs cannot be on lowmem machines’).
- --jvm-heap-size
Max JVM heap (-Xmx) for the in-process QoB driver under –experimental, e.g. ‘5g’. Set to ~50-70% of container memory (the rest is off-heap/Python/OS). Rejected without –experimental.
- --worker-cores
Number of cores per worker node. Pass a power of two between 0.25 and 16 (as a string, e.g. ‘1’ or ‘0.5’).
- --worker-memory
Possible choices: standard, highmem
Memory type for worker nodes. Default None = QoB’s ‘standard’, already generous (272 MiB peak measured). ‘lowmem’ is rejected for JVM jobs.
Test mode
Route inputs/outputs to test paths and (optionally) filter data.
- --test
Route reads/writes to test paths (group_membership, cov_and_an, release HTs, etc.). Independent of
--chrom: combine for a chrom-filtered test run.Default: False
- --test-sample-subset
With –test on AoU, subsample to ~0.1% of samples. Default off: a –test run keeps all samples so AN values stay comparable.
Default: False
- --test-n-partitions
Cheap test over the first N VDS partitions. With –write-chunk-intervals, scopes the probe (-> ceil(N/ppc) chunks; combine with –chrom). With the strict compute, reads the first N native partitions (genome start; does not compose with –chrom). Default None: full run.
- --chrom
Filter to a single contig (e.g. chr22). Appended to the output suffix so per-contig runs never collide; assemble with –assemble-chrom-coverage.
- --test-region
Scope a test to explicit locus intervals (e.g. chr1:55058666-55108666) instead of the first N VDS partitions; auto-enables test mode. Sites and VDS reads are region-scoped; straddling ref blocks are split (strict path / n_sub=1) or handled by leading-edge widening (fan-out). Mutually exclusive with –test-n-partitions.
Get gnomAD genomes group membership HT.
- --write-group-membership-ht
Write group membership HT.
Default: False
Compute coverage release stats HT.
- --merge-gnomad-coverage
Subtract consent drop samples from v4 release HT to create gnomAD v5 genomes coverage HT.
Default: False
- --export-coverage-release-files
Export the gnomAD v5 coverage HT and TSV. Coverage is gnomAD-only: AoU computes no coverage stats (its reference blocks carry no DP).
Default: False
Compute AN release stats HT.
- --merge-gnomad-an
Subtract consent drop samples from v4 release HT to create gnomAD v5 genomes AN HT.
Default: False
- --export-an-release-files
Exports the joint AoU + gnomAD v5 AN release HT and TSV file.
Default: False
Hail Batch fan-out for ``–compute-all-cov-release-stats-ht``.
Submit one Hail Batch job per partition chunk; each chunk is a tiny relay container that spawns its own QoB driver to do the densify. Idempotent: chunks whose _SUCCESS exists are skipped on rerun.
- --use-batch-fanout
Fan the coverage/AN compute out as one Batch job per chunk. Mutually exclusive with the other entry points; run –merge-cov-chunks afterward.
Default: False
- --submit-orchestrator
Run the orchestrator inside Hail Batch: submit one small non-spot job that re-invokes this command (minus this flag) and return. Requires –use-batch-fanout or –merge-cov-chunks; resubmit the same command to resume if the job dies.
Default: False
- --fanout-retry-passes
Extra automatic re-dispatch passes for still-missing chunks after the last wave. Unlike –chunk-attempts, a pass starts only after every prior relay has exited, so it cannot race an orphaned inner batch. 0 disables. Default 1.
Default: 1
- --merge-cov-chunks
Submit the recursive tree-reduce merge of the fan-out’s chunk HTs (group size –merge-group-size) down to the canonical HT. Missing chunks fail loudly.
Default: False
- --assemble-chrom-coverage
Union the per-contig HTs from –merge-cov-chunks –chrom runs into the canonical HT (contigs from the chunk-intervals JSON). Runs in-process.
Default: False
- --partitions-per-chunk
Number of VDS partitions per chunk job. Default 3, a shape from early testing; each chunk pays fixed overhead (relay + QoB driver + scan combine), so larger values are substantially cheaper at production scale (2026-08 calibration: ~$0.35/chunk fixed + ~$0.068/partition). IMPORTANT: keep identical across –write-chunk-intervals / –use-batch-fanout / –merge-cov-chunks (the JSON fixes the chunk set).
Default: 3
- --read-subintervals-per-chunk
Balanced sub-intervals per chunk (computed at –write-chunk-intervals time); the VDS and vep_context are both read with these intervals – co-partitioned, no shuffle. Default 50.
Default: 50
- --total-partitions
Total VDS partitions to scatter (full prod extent). Default 145192 (prod AoU VDS). Consumed by –write-chunk-intervals; –test-n-partitions overrides for tests.
Default: 145192
- --wave-size
Max chunks per Hail Batch under –use-batch-fanout; waves run sequentially, bounding concurrent relays and their QoB drivers. Capacity is the shared Hail Batch service pool, not gnomad-production GCP quota. <= 0 for a single batch. Default 1000.
Default: 1000
- --merge-group-size
Chunk HTs per group-merge job. Default 500.
Default: 500
- --methods-branch
Branch/commit of gnomad_methods to clone in chunk containers.
Default: “main”
- --batch-image
Docker image for the relay containers. The default tag is HAIL_VERSION, which the relay reinstalls regardless.
Default: “us-central1-docker.pkg.dev/broad-mpg-gnomad/images/v5_freq_batch:0.2.137”
- --batch-billing-project
Hail Batch billing project for the orchestrator pipeline.
Default: “gnomad-production”
- --batch-remote-tmpdir
GCS scratch for the orchestrator’s ServiceBackend. Default None: host Hail config.
- --chunk-cpu
CPU per chunk relay container. Default 0.5: the relay only builds IR and waits on its inner QoB batch.
Default: 0.5
- --chunk-memory
Possible choices: lowmem, standard, highmem
Hail Batch memory preset per chunk relay container.
Default: “standard”
- --chunk-storage
/io storage per chunk relay container. Default 5Gi.
Default: “5Gi”
- --chunk-driver-cores
Cores for the QoB driver pod each chunk/merge relay spawns. Power of two between 0.25 and 16 (as a string, e.g. ‘2’ or ‘0.5’). Forwarded to the relay’s –driver-cores; decoupled from the orchestrator’s own –driver-cores. Default ‘2’: with the highmem memory default this is a 13 GB driver, the smallest configuration measured to survive a coverage chunk (2-core standard, 8 GB, died; 1-core highmem, 6.5 GB, is smaller than what died and untested on coverage). The driver is billed for the whole worker barrier while idle, so cores beyond the memory they bring are near-pure waste.
Default: “2”
- --chunk-driver-memory
Possible choices: standard, highmem
Memory profile for the QoB driver pod. Default ‘highmem’: on ‘standard’ the driver died lowering the big fused execute (‘unexpected end of stream in jvm’ – a JVM-total limit, not a RegionPool OOM). ‘lowmem’ is rejected for JVM jobs.
Default: “highmem”
- --chunk-worker-cores
Cores per QoB worker pod the chunk/merge driver dispatches. Power of two between 0.25 and 16 (as a string, e.g. ‘1’ or ‘0.5’). Forwarded to the relay’s –worker-cores; decoupled from the orchestrator’s own –worker-cores.
- --chunk-worker-memory
Possible choices: standard, highmem
Memory per QoB worker pod. Default None = QoB’s ‘standard’, already generous (272 MiB peak measured). ‘lowmem’ is rejected for JVM jobs.
- --chunk-attempts
Max attempts per relay job. Default 1 = no auto-retry: Batch cannot cancel a lost attempt’s inner QoB batch, so a retry would race it. Failures land in _failed_chunks.json; rerun the fan-out.
Default: 1
- --merge-cpu
CPU per merge container.
Default: 4
- --merge-memory
Hail Batch memory preset per merge container.
Default: “standard”
- --merge-storage
Extra /io storage per group-merge container.
Default: “50Gi”
- --final-merge-storage
Extra /io storage for the final merge container.
Default: “100Gi”
- --batch-dry-run
If set, validate the Hail Batch DAG without submitting.
Default: False
Per-chunk worker entry-points for –use-batch-fanout (set by the orchestrator; not for direct use unless debugging a single chunk).
- --run-chunk
Worker entry-point: compute one chunk and write to –chunk-output. Requires –chunk-start, –chunk-stop, –chunk-output.
Default: False
- --chunk-start
Chunk index for –run-chunk: the chunk’s entry in the per-chunk precompute JSON (a –test-region run uses 0). Set by the orchestrator.
Default: 0
- --chunk-stop
Chunk index end (exclusive); the orchestrator sets it to –chunk-start + 1.
Default: 0
- --chunk-output
Output path for –run-chunk’s HT. When unset, auto-derived (the orchestrator always passes it explicitly).
- --run-merge
Worker entry-point: union the HTs at –merge-inputs and write to –merge-output.
Default: False
- --merge-inputs
GCS paths of per-chunk (or per-group) HTs to union.
- --merge-output
GCS output path for –run-merge’s merged HT.
- --merge-coalesce-to
naive_coalesce the unioned HT to this many partitions before writing (set by the merge orchestrator).
Module Functions
|
Get Table with downsampling groups for all samples. |
|
Get genomes group membership HT for all sites allele number stratification. |
|
Compute coverage, allele number, and quality histograms per reference site. |
|
Subtract consent-drop samples from the 3.0.1 coverage release to create the v5 HT. |
|
Subtract consent drop samples from gnomAD v4 genomes release HT to create gnomAD v5 genomes AN HT. |
|
Join AoU and gnomAD AN HTs for release. |
|
Join AoU and gnomAD qual hists HTs for release. |
Compute all sites coverage, AN, and quality histograms for v5 genomes. |
|
|
Get script argument parser. |
Compute coverage, allele number, and quality histograms for gnomAD v5 genomes.
v5 genomes = AoU v8 (new) + gnomAD v4 genomes minus the consent-drop set. This script computes per-reference-site AN per project, coverage (gnomAD only), and qual histograms (AoU only) and joins them into the v5 release HT/TSVs.
Execution roles (mutually exclusive; dispatched by early return in main):
ORCHESTRATOR (
--use-batch-fanout/--merge-cov-chunks): never initializes Hail; submits a Hail Batch of relay jobs and returns. Used for the prod-scale AoU compute.WORKER (
--run-chunk/--run-merge): one relay container; initializes Hail and runs exactly one chunk/merge unit. The chunk worker is what callscompute_all_release_stats_per_ref_site.IN-PROCESS PIPELINE (neither): the strict single-job compute (gnomAD consent-drop, dev/test AoU), the setup writers, and the release assembly.
A full prod AoU release is three separate invocations: fan-out compute,
merge-cov-chunks, then the in-process assembly. The setup HTs (downsampling,
group membership, sites, chunk intervals) are written by their own
in-process invocations first; the fan-out reads them from disk. The AoU sample
artifact JSONs (release samples, ID collisions) come from
create_sample_qc_metadata_ht.py --write-vds-sample-artifact-jsons and are
read inside get_aou_vds.
Workflow:
1. (AoU) --write-aou-downsampling-ht
2. --write-group-membership-ht. AoU: one aggregation cell per distinct
membership pattern; every AN group is reconstructed by summing cells.
gnomAD: the consent-drop samples, one boolean per group.
3. --write-vep-context-sites: deduped, telomere/centromere/chrM-stripped
sites HT -- the definition of "every site the compute must cover".
4. Compute: --compute-all-cov-release-stats-ht (strict single job), or
--write-chunk-intervals, then --use-batch-fanout, then
--merge-cov-chunks. Chunk outputs are namespaced by a content hash of
the intervals JSON: rerunning the fan-out retries only missing chunks,
and a regenerated JSON (new layout, new hash) never mixes with old
chunks.
5. --validate-cov-and-an: the merged HT must cover every site exactly.
6. (gnomad) --merge-gnomad-coverage / --merge-gnomad-an: subtract the
consent-drop samples from the release HTs. AN: the samples in the v4
release, from the v4.1 AN release. Coverage: the samples in the v3.0
release, from the 3.0.1 coverage release (the only genomes coverage
ever computed; v4 reused it).
7. (aou) Release: --export-coverage-release-files (gnomAD-only; AoU
computes no coverage stats), --export-an-release-files,
--merge-qual-hists (gnomAD v4 hists reused as-is).
Usage Examples:
# Per-chunk fan-out compute + merge (on batch). Run setup steps 1-3 +
# --write-chunk-intervals first; the fan-out reads those outputs:
python compute_coverage.py --project-name aou --environment batch \\
--use-batch-fanout --partitions-per-chunk 3
python compute_coverage.py --project-name aou --environment batch \\
--merge-cov-chunks --n-partitions 10000
# gnomAD v4 consent-drop subtraction (on dataproc):
hailctl dataproc submit cluster-name compute_coverage.py \\
--project-name gnomad --environment dataproc \\
--merge-gnomad-coverage --merge-gnomad-an
# AoU/gnomAD release assembly (on batch):
python compute_coverage.py --project-name aou --environment batch \\
--export-coverage-release-files --export-an-release-files \\
--merge-qual-hists
- gnomad_qc.v5.annotations.compute_coverage.get_downsampling_ht(ht)[source]
Get Table with downsampling groups for all samples.
v5 downsamplings (AoU only): global 10,000 and 100,000, plus per-group sizes for afr/amr/nfe only (
gen_ancs_to_downsample). Every group still gets a per-group index.
- gnomad_qc.v5.annotations.compute_coverage.get_group_membership_ht(meta_ht, project, ds_ht=None)[source]
Get genomes group membership HT for all sites allele number stratification.
AoU: reduced to cells (
reduce_to_cells=True): one aggregation per distinct membership pattern instead of one per group, with every group – downsamplings included – reconstructed by summing cells. Compute calls must passreducible_aggs={"AN"}and pin every non-summable annotation (qual_hists) to a single group viaentry_agg_group_membership.gnomAD: consent-drop samples only. The AN groups contain the samples that were in the v4 release; the raw coverage group (
GNOMAD_COVERAGE_GROUP, appended last) contains the samples that were in the v3.0 release.
- gnomad_qc.v5.annotations.compute_coverage.compute_all_release_stats_per_ref_site(vds, ref_ht, sex_karyotype_field, project, coverage_over_x_bins=(1, 5, 10, 15, 20, 25, 30, 50, 100), interval_ht=None, group_membership_ht=None)[source]
Compute coverage, allele number, and quality histograms per reference site.
Note
Running this prior to calculating frequencies removes the need for an additional densify there.
- Parameters:
vds (
VariantDataset) – Input VDS. Reference data must carryEND(andGQ); aLENfield is added if missing. DP is read from variant data only.ref_ht (
Table) – Locus-only sites Table defining the positions to aggregate at.sex_karyotype_field (
Optional[str]) – Dotted path on the variant_data column struct to the sample’s sex karyotype (sets per-sample ploidy on sex chromosomes), or None to skip the adjustment. It is the identity on autosomes, so pass None whenref_hthas no chrX/chrY sites (seespans_sex_chromosome()): that also skips the per-sample karyotype lookup and its column checkpoint.project (
str) – “aou” or “gnomad”. AoU addsqual_histsand computes no coverage stats; gnomAD computes coverage and AN only.coverage_over_x_bins (
Sequence[int]) – DP thresholds for theover_Xfields. Default isCOVERAGE_OVER_X_BINS.interval_ht (
Optional[Table]) – Optional interval Table for partition pruning. Unused on the v5 path.group_membership_ht (
Optional[Table]) – Group-membership Table defining the per-stratum sample sets AN is fanned out across, fromget_group_membership_ht(). AN is passed as the onlyreducible_aggsentry: on the cell-reduced AoU HT it is aggregated once per cell and every group is rebuilt by summing cells; on the gnomAD HT it is aggregated per group.
- Return type:
- Returns:
HT keyed by locus with per-stratum
AN; flat coverage fields (raw, over theGNOMAD_COVERAGE_GROUPsamples) for gnomAD;qual_hists(adj GQ histogram only) for AoU. AoU reference blocks carry no DP, so its coverage would measure ~0 genome-wide (chr1:55.06-55.16Mb: mean 0.048x,over_1432 of 365,318) and is never computed, and for the same reason its DP histogram at reference sites is empty and is not computed either.
- gnomad_qc.v5.annotations.compute_coverage.merge_gnomad_coverage_hts(gnomad_ht, gnomad_release_ht, coverage_over_x_bins=(1, 5, 10, 15, 20, 25, 30, 50, 100), gnomad_sample_count=71702, consent_drop_count=849)[source]
Subtract consent-drop samples from the 3.0.1 coverage release to create the v5 HT.
- Parameters:
gnomad_ht (
Table) – Coverage HT of the consent-drop samples in the v3.0 release.gnomad_release_ht (
Table) – gnomAD 3.0.1 genomes coverage release HT.coverage_over_x_bins (
Sequence[int]) – Boundaries for the over-X fields. Default isCOVERAGE_OVER_X_BINS.gnomad_sample_count (
int) – 3.0.1 coverage release genome count. Default GNOMAD_COVERAGE_RELEASE_SAMPLE_COUNT.consent_drop_count (
int) – Consent-drop genomes in the 3.0.1 coverage release. Default GNOMAD_CONSENT_DROP_COVERAGE_SAMPLE_COUNT.
- Return type:
- Returns:
gnomAD v5 genomes coverage HT in the release schema (
mean,median_approx,total_DP, fractionover_X).
- gnomad_qc.v5.annotations.compute_coverage.merge_gnomad_an_hts(gnomad_ht, gnomad_release_ht)[source]
Subtract consent drop samples from gnomAD v4 genomes release HT to create gnomAD v5 genomes AN HT.
- gnomad_qc.v5.annotations.compute_coverage.join_aou_and_gnomad_an_ht(aou_ht, gnomad_ht)[source]
Join AoU and gnomAD AN HTs for release.
The shared strata’s combined AN is written first as the unlabeled “global” entries; the AoU-only AN (including the downsampling strata) is appended as an
{"subset": "aou"}-tagged subset.
- gnomad_qc.v5.annotations.compute_coverage.join_aou_and_gnomad_qual_hists_ht(aou_ht, gnomad_ht)[source]
Join AoU and gnomAD qual hists HTs for release.
Note
Qual hists were not computed for the gnomAD v4 genomes release, so v5 reuses the v4 hists as-is (no consent-drop subtraction).
- gnomad_qc.v5.annotations.compute_coverage.main(args)[source]
Compute all sites coverage, AN, and quality histograms for v5 genomes.
Dispatches the three mutually-exclusive roles (see the module docstring):
orchestrator: submit a Hail Batch and return.
worker: run one chunk/merge unit and return.
in-process pipeline: setup -> compute -> assemble.