gnomad.utils.constraint.models

Functions to build and apply the plateau and coverage models.

gnomad.utils.constraint.models.calibration_model_group_expr(...)

Get the calibration model grouping annotation for a variant.

gnomad.utils.constraint.models.build_models(ht, ...)

Build coverage and plateau models.

gnomad.utils.constraint.models.build_plateau_models(...)

Build plateau models to calibrate mutation rate against proportion observed.

gnomad.utils.constraint.models.build_coverage_model(...)

Build coverage model.

gnomad.utils.constraint.models.apply_plateau_models(...)

Compute the predicted probability observed.

gnomad.utils.constraint.models.coverage_correction_expr(...)

Compute the coverage correction expression.

gnomad.utils.constraint.models.apply_models(...)

Apply calibration models to compute expected variant counts.

Functions to build and apply the plateau and coverage models.

gnomad.utils.constraint.models.calibration_model_group_expr(exomes_coverage_expr, cpg_expr, low_cov_cutoff=None, high_cov_cutoff=30, upper_cov_cutoff=None, skip_coverage_model=False, additional_grouping_exprs=None, cpg_in_high_only=False)[source]

Get the calibration model grouping annotation for a variant.

The calibration model expression is a struct with the following fields:

  • high_or_low_coverage: Whether the variant belongs to the high or low coverage calibration model. The variant is assigned to the high coverage model if the exome coverage is greater than or equal to ‘high_cov_cutoff’ and less than or equal to ‘upper_cov_cutoff’ (if provided). The variant is assigned to the low coverage model if skip_coverage_model is False and the exome coverage is greater than ‘low_cov_cutoff’ (if provided) and less than ‘high_cov_cutoff’.

  • model_group: Struct of the fields the models are grouped by:

    • cpg: Whether the variant is a CpG (cpg_expr).

    • Any fields in additional_grouping_exprs (e.g. genomic_region).

The expression is missing for variants that belong to neither model.

Parameters:
  • exomes_coverage_expr (Int32Expression) – Exome coverage expression.

  • cpg_expr (BooleanExpression) – CpG expression.

  • low_cov_cutoff (Optional[int]) – Low coverage cutoff. Default is None.

  • high_cov_cutoff (int) – High coverage cutoff. Default is COVERAGE_CUTOFF.

  • upper_cov_cutoff (Optional[int]) – Upper coverage cutoff. Default is None.

  • skip_coverage_model (bool) – Whether to skip the coverage model. Default is False.

  • additional_grouping_exprs (Optional[Dict[str, StringExpression]]) – Optional Dictionary of additional expressions to group by. Default is None.

  • cpg_in_high_only (bool) – Whether to only include cpg_expr in the model grouping for the high coverage model. Default is False.

Return type:

StructExpression

Returns:

StructExpression with high_or_low_coverage and model_group fields.

gnomad.utils.constraint.models.build_models(ht, coverage_expr, weighted=False, keys=('context', 'ref', 'alt', 'methylation_level'), model_group_expr=None, high_cov_definition=30, upper_cov_cutoff=None, skip_coverage_model=False, log10_coverage=True, additional_grouping=())[source]

Build coverage and plateau models.

This function builds models (plateau_models) using linear regression to calibrate mutation rate estimates against the proportion observed of each substitution, context, and methylation level in ht.

Two plateau models are fit, one for CpG transitions, and one for the remainder of sites (transversions and non CpG transitions).

The plateau models only consider high coverage sites, or sites above a median coverage of high_cov_definition and median coverage below upper_cov_cutoff.

Plateau model: adjusts proportion of expected variation based on location in the genome and CpG status.

The x and y of the plateau models:
  • x: mu_snp - mutation rate

  • y: proportion observed (‘observed_variants’ / ‘possible_variants’)

This function also builds models (coverage models) to calibrate the proportion of expected variation at low coverage sites (sites below high_cov_definition).

The coverage models are built by creating a scaling factor across all high coverage sites, applying this ratio to the low coverage sites, and running a linear regression.

Coverage model: corrects proportion of expected variation at low coverage sites. Low coverage sites are defined as sites with median coverage < high_cov_definition.

The x and y of the coverage model:

  • x: groupings of exome coverage at low coverage sites (log10 transformed if requested)

  • y: sum(‘observed_variants’) / (high_coverage_scale_factor * sum(‘possible_variants’ * ‘mu_snp’)) at low coverage sites

high_coverage_scale_factor = sum(‘observed_variants’) / sum(‘possible_variants’ * ‘mu_snp’) at high coverage sites

Note

This function expects that the input Table (ht) contains only high quality synonymous variants below 0.1% frequency.

The following fields are expected in ht:

  • context: trinucleotide genomic context.

  • ref: the reference allele.

  • alt: the alternate allele.

  • methylation_level: methylation level.

  • mu_snp: mutation rate.

  • cpg: whether the variant is a CpG.

  • observed_variants: the number of observed variants in the dataset for each variant. Note that the term “variant” here refers to a specific substitution, context, methylation level, and coverage combination.

  • possible_variants: the number of possible variants in the dataset for each variant.

Parameters:
  • ht (Table) – Input Table.

  • coverage_expr (Int32Expression) – Expression that defines the coverage metric.

  • weighted (bool) – Whether to weight the plateau models (a linear regression model) by ‘possible_variants’. Default is False.

  • keys (Tuple[str, ...]) – Annotations used to group observed and possible variant counts. Default is (“context”, “ref”, “alt”, “methylation_level”).

  • model_group_expr (Optional[StructExpression]) – Expression with high_or_low_coverage and model_group annotations to group variants into high or low coverage models. If model_group includes a genomic_region field, the coverage model is built only on sites where it is “autosome_or_par”; otherwise all sites are used. If not provided, the calibration_model_group_expr function is used to define the grouping, which does not include genomic_region.

  • high_cov_definition (int) – Coverage threshold for high/low classification. Sites with coverage above this cutoff are considered well covered. Default is COVERAGE_CUTOFF.

  • upper_cov_cutoff (Optional[int]) – Upper coverage cutoff. Sites with coverage above this cutoff are excluded from the high coverage Table. Default is None.

  • skip_coverage_model (bool) – Whether to skip generating the coverage model. If set to True, None is returned instead of the coverage model. Default is False.

  • log10_coverage (bool) – Whether to convert coverage sites with log10 when building the coverage model. Default is True.

  • additional_grouping (Tuple[str, ...]) – Additional annotations to group by before counting the observed and possible variants. Default is ().

Return type:

Tuple[Optional[ArrayNumericExpression], DictExpression]

Returns:

Tuple of (coverage model, plateau models). The coverage model is an [intercept, slope] ArrayNumericExpression, or None when skip_coverage_model is True.

gnomad.utils.constraint.models.build_plateau_models(mu_snp_expr, observed_variants_expr, possible_variants_expr, weighted=False, cpg_expr=None, model_group_expr=None)[source]

Build plateau models to calibrate mutation rate against proportion observed.

Fits a linear regression of observed_variants_expr / possible_variants_expr on mu_snp_expr. When either observed or possible expressions are arrays (e.g., one model per downsampling), the regression is applied element-wise via hl.agg.array_agg.

When model_group_expr or cpg_expr is provided, the result is a DictExpression keyed by the grouping struct. Otherwise the result is the regression beta directly.

Parameters:
  • mu_snp_expr (Float64Expression) – Mutation rate expression.

  • observed_variants_expr (Union[ArrayExpression, Int64Expression]) – Observed variant counts (scalar or array).

  • possible_variants_expr (Union[ArrayExpression, Int64Expression]) – Possible variant counts (scalar or array).

  • weighted (bool) – If True, use weighted least squares with possible_variants_expr as weights. Default is False.

  • cpg_expr (Optional[BooleanExpression]) – Boolean expression indicating CpG sites. When provided, adds a cpg field to the grouping struct.

  • model_group_expr (Optional[StructExpression]) – Struct expression to group by in the aggregation.

Return type:

Union[DictExpression, ArrayExpression, StructExpression]

Returns:

Regression betas, optionally grouped by model_group_expr (and/or cpg_expr).

gnomad.utils.constraint.models.build_coverage_model(low_coverage_oe_expr, coverage_expr, log10_coverage=False)[source]

Build coverage model.

This function uses linear regression to build a model of coverage to correct proportion of expected variation at low coverage sites.

The x and y of the coverage model:

  • x: coverage_expr

  • y: low_coverage_oe_expr

Parameters:
  • low_coverage_oe_expr (Float64Expression) – The Float64Expression of observed:expected ratio for a given coverage level.

  • coverage_expr (Float64Expression) – The Float64Expression of the coverage expression.

  • log10_coverage (bool) – Whether to convert coverage sites by log10 when building the coverage model. Default is False.

Return type:

StructExpression

Returns:

StructExpression with intercept and slope of the model.

gnomad.utils.constraint.models.apply_plateau_models(mu_expr, plateau_models_expr)[source]

Compute the predicted probability observed.

The predicted probability observed is computed as the mutation rate adjusted by the plateau model.

Parameters:
  • mu_expr (Float64Expression) – Mutation rate expression.

  • plateau_models_expr (ArrayExpression) – This can be either a single plateau model, where the first element is the intercept and the second element is the slope, or an array of plateau models.

Return type:

Union[ArrayExpression, Float64Expression]

Returns:

Float64Expression of the predicted probability observed, or an ArrayExpression of them when plateau_models_expr is an array of models.

gnomad.utils.constraint.models.coverage_correction_expr(coverage_expr, coverage_model, low_coverage_expr=None, coverage_cutoff=None, log10_coverage=False)[source]

Compute the coverage correction expression.

Note

One and only one of low_coverage_expr or coverage_cutoff must be specified.

The coverage correction expression is computed as follows:

  • If the coverage is 0, the coverage correction is 0.

  • If the low coverage expression (low_coverage_expr) is True, or the coverage (coverage_expr) is below the coverage cutoff (coverage_cutoff), the coverage model is applied to the coverage.

  • Otherwise, the coverage correction is 1.

Parameters:
  • coverage_expr (Float64Expression) – Float64Expression of the coverage.

  • coverage_model (Union[Tuple[float, float], ArrayNumericExpression]) – Coverage model as [intercept, slope], either a tuple of floats or an ArrayNumericExpression (as returned by build_models).

  • low_coverage_expr (Optional[BooleanExpression]) – Optional BooleanExpression indicating whether the site is a low coverage site, and the coverage model should be applied. Default is None.

  • coverage_cutoff (Optional[int]) – Optional coverage cutoff. If specified, the coverage model is applied to sites with coverage below this cutoff. Default is None.

  • log10_coverage (bool) – Whether to convert coverage sites by log10 when applying the coverage model. Default is False.

Return type:

Float64Expression

Returns:

Float64Expression of the coverage correction.

gnomad.utils.constraint.models.apply_models(mu_expr, plateau_models_expr, possible_variants_expr, coverage_model=None, coverage_expr=None, cpg_expr=None, model_group_expr=None, high_cov_definition=30, log10_coverage=True)[source]

Apply calibration models to compute expected variant counts.

Applies plateau and (optionally) coverage models to produce a struct with mu, predicted_proportion_observed, expected_variants, and (when a coverage model is provided) coverage_correction.

Note

When a coverage model is applied, a row whose model_group_expr is missing gets missing mu, expected_variants, and coverage_correction (unless its coverage is 0, which always gives a correction of 0). calibration_model_group_expr is missing for sites in neither model, such as sites above upper_cov_cutoff or at or below low_cov_cutoff. hl.agg.sum skips missing values, so filter these rows out before applying the models if they should not silently drop out of aggregated expected counts.

Parameters:
  • mu_expr (Float64Expression) – Mutation rate expression.

  • plateau_models_expr (ArrayExpression) – Single plateau model (array of [intercept, slope]) or an array of plateau models.

  • possible_variants_expr (Int64Expression) – Possible variant counts to multiply the predicted proportion observed by.

  • coverage_model (Union[Tuple[float, float], ArrayNumericExpression, None]) – Coverage model as [intercept, slope], either a tuple of floats or an ArrayNumericExpression (as returned by build_models). Default is None.

  • coverage_expr (Optional[Int32Expression]) – Int32Expression of the coverage. Required when coverage_model is provided or model_group_expr is None.

  • cpg_expr (Optional[BooleanExpression]) – BooleanExpression indicating CpG sites. Required when model_group_expr is None.

  • model_group_expr (Optional[StructExpression]) – Expression with high_or_low_coverage annotation to group variants into high or low coverage models. If not provided, calibration_model_group_expr is used. Default is None.

  • high_cov_definition (int) – Coverage threshold for high/low classification. Default is COVERAGE_CUTOFF.

  • log10_coverage (bool) – Whether to log10-transform coverage when applying the coverage model. Default is True.

Return type:

StructExpression

Returns:

StructExpression with “mu”, “predicted_proportion_observed”, “expected_variants”, and optionally “coverage_correction”.