gnomad.utils.constraint.counts

Functions to count observed and possible variants and sum them by group.

gnomad.utils.constraint.counts.variant_observed_expr([...])

Return 1 if a variant meets frequency criteria, 0 otherwise.

gnomad.utils.constraint.counts.variant_observed_and_possible_expr(...)

Return per-variant observed and possible count expressions.

gnomad.utils.constraint.counts.weighted_sum_agg_expr(...)

Return the weighted aggregate sum of expr weighted by weight_expr.

gnomad.utils.constraint.counts.counts_agg_expr([...])

Return an aggregation expression for variant and singleton counts.

gnomad.utils.constraint.counts.count_observed_and_possible_by_group(ht, ...)

Aggregate observed and possible variant counts by substitution context group.

gnomad.utils.constraint.counts.aggregate_constraint_metrics_expr(t)

Get an aggregation expression for the sum of specified fields and expressions.

Functions to count observed and possible variants and sum them by group.

gnomad.utils.constraint.counts.variant_observed_expr(freq_expr=None, ht=None, singleton=False, max_af=None, count_missing=False)[source]

Return 1 if a variant meets frequency criteria, 0 otherwise.

One of ht or freq_expr must be provided. If freq_expr is not provided, ht.freq is used. When freq_expr is an ArrayExpression, the first element is used.

The returned count is 1 when:

  • singleton is True and freq_expr.AC == 1.

  • max_af is not None and freq_expr.AC > 0 and freq_expr.AF <= max_af.

  • Neither singleton nor max_af is set but freq_expr is available: freq_expr.AC > 0.

  • Neither singleton nor max_af is set and no freq_expr is available: unconditionally 1.

Parameters:
  • ht (Optional[Table]) – Input Table. Used to look up freq when freq_expr is None.

  • freq_expr (Union[StructExpression, ArrayExpression, None]) – StructExpression (or ArrayExpression of structs) with AC and AF fields. When an array, freq_expr[0] is used.

  • singleton (bool) – Count only singletons (AC == 1). When True, max_af is ignored. Default is False.

  • max_af (Optional[float]) – Maximum allele frequency threshold. Default is None (no cutoff).

  • count_missing (bool) – Whether to count a variant whose count expression is missing (e.g. frequency is None). True counts it as 1, False as 0. Default is False.

Return type:

Int32Expression

Returns:

Int32Expression equal to 0 or 1.

gnomad.utils.constraint.counts.variant_observed_and_possible_expr(freq_expr, max_af=None, use_possible_adj=True)[source]

Return per-variant observed and possible count expressions.

For each element of the frequency array, observed_variants is 1 when the variant meets the frequency criteria (AC > 0, and AF <= max_af if set) and 0 otherwise. possible_variants uses the same criteria, except that a missing frequency counts as possible: a variant absent from the dataset could still have been observed. A variant with a defined frequency that fails the criteria (AC == 0, or AF > max_af) is therefore 0 in both, so it is excluded from the denominator as well as the numerator. With max_af set, this removes common variants from the possible counts.

When freq_expr itself is missing (e.g. the variant is not in the dataset), observed_variants is missing and possible_variants is 1, or missing when use_possible_adj is False.

When use_possible_adj is True (default), the possible count is a scalar derived from the first (adj) element of the frequency array. When False, it is an array with one entry per array element, matching the shape of observed_variants.

Parameters:
  • freq_expr (ArrayExpression) – Array of frequency structs with AC and AF fields.

  • max_af (Optional[float]) – Maximum allele frequency threshold. Variants above it are excluded from both counts. Default is None (no cutoff).

  • use_possible_adj (bool) – If True, compute possible count from only the adj (first) frequency element. If False, compute per-downsampling. Default is True.

Return type:

StructExpression

Returns:

Struct with observed_variants (array of int) and possible_variants (int if use_possible_adj, else array of int).

gnomad.utils.constraint.counts.weighted_sum_agg_expr(expr, weight_expr)[source]

Return the weighted aggregate sum of expr weighted by weight_expr.

Both parameters may be scalar or array numeric expressions:

  • Both arrays: elements are multiplied pairwise and summed per-element across rows (hl.agg.array_sum).

  • Both scalars: standard hl.agg.sum(expr * weight_expr).

  • Mixed: the scalar is broadcast across the array elements and summed per-element across rows.

Parameters:
Return type:

Union[ArrayExpression, NumericExpression]

Returns:

Weighted aggregate sum expression.

gnomad.utils.constraint.counts.counts_agg_expr(freq_expr=None, ht=None, count_singletons=False, max_af=None, count_missing=False)[source]

Return an aggregation expression for variant and singleton counts.

Aggregates per-variant counts (via variant_observed_expr) across rows. Each variant contributes 0 or 1 to the count based on its frequency metadata (AC, AF). The result is a struct with variant_count and, when count_singletons is True, singleton_count.

One of freq_expr or ht must be provided. If freq_expr is not provided, ht.freq is used as the fallback.

The shape of freq_expr controls whether the output counts are scalars or arrays:

  • StructExpression (single frequency entry with AC / AF): returns scalar variant_count and singleton_count.

  • ArrayExpression (array of frequency structs, e.g. one per downsampling): returns array-valued counts where each element corresponds to a position in the input array. Internally the array is mapped through variant_observed_expr and summed element-wise with hl.agg.array_sum.

When max_af is set, only variants with AF <= max_af and AC > 0 are counted. Singleton counting (count_singletons=True) counts only variants with AC == 1, independent of the max_af filter.

Parameters:
  • freq_expr (Union[StructExpression, ArrayExpression, None]) – Frequency expression — an ArrayExpression of structs or a single StructExpression with AC and AF fields. If None, falls back to ht.freq.

  • ht (Optional[Table]) – Input Table. Used to look up freq when freq_expr is None.

  • count_singletons (bool) – If True, include a singleton_count field in the result. Default is False.

  • max_af (Optional[float]) – Maximum allele frequency threshold. Variants with AF > max_af are excluded from variant_count. Does not affect singleton_count. Default is None (no cutoff).

  • count_missing (bool) – Whether to count a variant whose frequency is missing. True counts it as 1, False as 0. Default is False.

Return type:

StructExpression

Returns:

Aggregation StructExpression with variant_count (and optionally singleton_count). Values are scalars when freq_expr is a StructExpression, or arrays when it is an ArrayExpression.

gnomad.utils.constraint.counts.count_observed_and_possible_by_group(ht, possible_expr, observed_expr, additional_grouping=('methylation_level',), partition_hint=100, weight_exprs=None, additional_agg_sum_exprs=None)[source]

Aggregate observed and possible variant counts by substitution context group.

Groups rows by context, ref, alt, and any fields in additional_grouping, then sums observed and possible counts within each group.

Parameters:
  • ht (Table) – Input Table with context, ref, alt fields and any fields named in additional_grouping.

  • possible_expr (Int32Expression) – Per-variant possible count (scalar).

  • observed_expr (ArrayExpression) – Per-variant observed count (array, one element per array element of the frequency array).

  • additional_grouping (Union[List[str], Tuple[str, ...]]) – Field names to append to the base (context, ref, alt) grouping. Default is ("methylation_level",).

  • partition_hint (int) – Target number of partitions for the group_by. Default is 100.

  • weight_exprs (Union[List[str], Dict[str, Union[ArrayExpression, NumericExpression]], None]) – Weighted sums of possible_expr to include. Pass field names (looked up on ht) or a dict mapping output names to weight expressions. Each produces weighted_sum_agg_expr(possible_expr, weight).

  • additional_agg_sum_exprs (Union[List[str], Dict[str, Union[ArrayExpression, NumericExpression]], None]) – Extra fields to sum alongside observed/possible. Pass field names (looked up on ht) or a dict mapping output names to expressions. Arrays use hl.agg.array_sum; scalars use hl.agg.sum.

Return type:

Table

Returns:

Grouped Table with observed_variants, possible_variants, and any weighted/additional sum fields.

gnomad.utils.constraint.counts.aggregate_constraint_metrics_expr(t, fields_to_sum=('mu_snp', 'mu', 'observed_variants', 'possible_variants', 'predicted_proportion_observed', 'coverage_correction', 'expected_variants'), additional_exprs_to_sum=None)[source]

Get an aggregation expression for the sum of specified fields and expressions.

An aggregate sum or array sum is created for each field in fields_to_sum.

Parameters:
  • t (Union[Table, StructExpression]) – Input Table or StructExpression.

  • fields_to_sum (Union[List[str], Tuple[str, ...]]) – Fields in t to get an aggregate sum expression for. Default is DEFAULT_FIELDS_TO_SUM.

  • additional_exprs_to_sum (Optional[Dict[str, Expression]]) – Dictionary of additional expressions to get an aggregate sum expression for. Field names are the keys and expressions are the values. Default is None.

Return type:

StructExpression

Returns:

StructExpression with the sum of each field and expression.