gnomad.utils.constraint.models
Functions to build and apply the plateau and coverage models.
|
Get the calibration model grouping annotation for a variant. |
Build coverage and plateau models. |
|
Build plateau models to calibrate mutation rate against proportion observed. |
|
Build coverage model. |
|
Compute the predicted probability observed. |
|
|
Compute the coverage correction expression. |
Apply calibration models to compute expected variant counts. |
Functions to build and apply the plateau and coverage models.
- gnomad.utils.constraint.models.calibration_model_group_expr(exomes_coverage_expr, cpg_expr, low_cov_cutoff=None, high_cov_cutoff=30, upper_cov_cutoff=None, skip_coverage_model=False, additional_grouping_exprs=None, cpg_in_high_only=False)[source]
Get the calibration model grouping annotation for a variant.
The calibration model expression is a struct with the following fields:
high_or_low_coverage: Whether the variant belongs to the high or low coverage calibration model. The variant is assigned to the high coverage model if the exome coverage is greater than or equal to ‘high_cov_cutoff’ and less than or equal to ‘upper_cov_cutoff’ (if provided). The variant is assigned to the low coverage model if skip_coverage_model is False and the exome coverage is greater than ‘low_cov_cutoff’ (if provided) and less than ‘high_cov_cutoff’.
model_group: Struct of the fields the models are grouped by:
cpg: Whether the variant is a CpG (cpg_expr).
Any fields in additional_grouping_exprs (e.g.
genomic_region).
The expression is missing for variants that belong to neither model.
- Parameters:
exomes_coverage_expr (
Int32Expression) – Exome coverage expression.cpg_expr (
BooleanExpression) – CpG expression.low_cov_cutoff (
Optional[int]) – Low coverage cutoff. Default is None.high_cov_cutoff (
int) – High coverage cutoff. Default is COVERAGE_CUTOFF.upper_cov_cutoff (
Optional[int]) – Upper coverage cutoff. Default is None.skip_coverage_model (
bool) – Whether to skip the coverage model. Default is False.additional_grouping_exprs (
Optional[Dict[str,StringExpression]]) – Optional Dictionary of additional expressions to group by. Default is None.cpg_in_high_only (
bool) – Whether to only include cpg_expr in the model grouping for the high coverage model. Default is False.
- Return type:
- Returns:
StructExpression with
high_or_low_coverageandmodel_groupfields.
- gnomad.utils.constraint.models.build_models(ht, coverage_expr, weighted=False, keys=('context', 'ref', 'alt', 'methylation_level'), model_group_expr=None, high_cov_definition=30, upper_cov_cutoff=None, skip_coverage_model=False, log10_coverage=True, additional_grouping=())[source]
Build coverage and plateau models.
This function builds models (plateau_models) using linear regression to calibrate mutation rate estimates against the proportion observed of each substitution, context, and methylation level in ht.
Two plateau models are fit, one for CpG transitions, and one for the remainder of sites (transversions and non CpG transitions).
The plateau models only consider high coverage sites, or sites above a median coverage of high_cov_definition and median coverage below upper_cov_cutoff.
Plateau model: adjusts proportion of expected variation based on location in the genome and CpG status.
- The x and y of the plateau models:
x: mu_snp - mutation rate
y: proportion observed (‘observed_variants’ / ‘possible_variants’)
This function also builds models (coverage models) to calibrate the proportion of expected variation at low coverage sites (sites below high_cov_definition).
The coverage models are built by creating a scaling factor across all high coverage sites, applying this ratio to the low coverage sites, and running a linear regression.
Coverage model: corrects proportion of expected variation at low coverage sites. Low coverage sites are defined as sites with median coverage < high_cov_definition.
The x and y of the coverage model:
x: groupings of exome coverage at low coverage sites (log10 transformed if requested)
y: sum(‘observed_variants’) / (
high_coverage_scale_factor* sum(‘possible_variants’ * ‘mu_snp’)) at low coverage sites
high_coverage_scale_factor= sum(‘observed_variants’) / sum(‘possible_variants’ * ‘mu_snp’) at high coverage sitesNote
This function expects that the input Table (ht) contains only high quality synonymous variants below 0.1% frequency.
The following fields are expected in ht:
context: trinucleotide genomic context.
ref: the reference allele.
alt: the alternate allele.
methylation_level: methylation level.
mu_snp: mutation rate.
cpg: whether the variant is a CpG.
observed_variants: the number of observed variants in the dataset for each variant. Note that the term “variant” here refers to a specific substitution, context, methylation level, and coverage combination.
possible_variants: the number of possible variants in the dataset for each variant.
- Parameters:
ht (
Table) – Input Table.coverage_expr (
Int32Expression) – Expression that defines the coverage metric.weighted (
bool) – Whether to weight the plateau models (a linear regression model) by ‘possible_variants’. Default is False.keys (
Tuple[str,...]) – Annotations used to group observed and possible variant counts. Default is (“context”, “ref”, “alt”, “methylation_level”).model_group_expr (
Optional[StructExpression]) – Expression withhigh_or_low_coverageandmodel_groupannotations to group variants into high or low coverage models. Ifmodel_groupincludes agenomic_regionfield, the coverage model is built only on sites where it is “autosome_or_par”; otherwise all sites are used. If not provided, thecalibration_model_group_exprfunction is used to define the grouping, which does not includegenomic_region.high_cov_definition (
int) – Coverage threshold for high/low classification. Sites with coverage above this cutoff are considered well covered. Default isCOVERAGE_CUTOFF.upper_cov_cutoff (
Optional[int]) – Upper coverage cutoff. Sites with coverage above this cutoff are excluded from the high coverage Table. Default is None.skip_coverage_model (
bool) – Whether to skip generating the coverage model. If set to True, None is returned instead of the coverage model. Default is False.log10_coverage (
bool) – Whether to convert coverage sites with log10 when building the coverage model. Default is True.additional_grouping (
Tuple[str,...]) – Additional annotations to group by before counting the observed and possible variants. Default is ().
- Return type:
Tuple[Optional[ArrayNumericExpression],DictExpression]- Returns:
Tuple of (coverage model, plateau models). The coverage model is an
[intercept, slope]ArrayNumericExpression, or None whenskip_coverage_modelis True.
- gnomad.utils.constraint.models.build_plateau_models(mu_snp_expr, observed_variants_expr, possible_variants_expr, weighted=False, cpg_expr=None, model_group_expr=None)[source]
Build plateau models to calibrate mutation rate against proportion observed.
Fits a linear regression of
observed_variants_expr / possible_variants_expronmu_snp_expr. When either observed or possible expressions are arrays (e.g., one model per downsampling), the regression is applied element-wise viahl.agg.array_agg.When
model_group_exprorcpg_expris provided, the result is aDictExpressionkeyed by the grouping struct. Otherwise the result is the regression beta directly.- Parameters:
mu_snp_expr (
Float64Expression) – Mutation rate expression.observed_variants_expr (
Union[ArrayExpression,Int64Expression]) – Observed variant counts (scalar or array).possible_variants_expr (
Union[ArrayExpression,Int64Expression]) – Possible variant counts (scalar or array).weighted (
bool) – If True, use weighted least squares withpossible_variants_expras weights. Default is False.cpg_expr (
Optional[BooleanExpression]) – Boolean expression indicating CpG sites. When provided, adds acpgfield to the grouping struct.model_group_expr (
Optional[StructExpression]) – Struct expression to group by in the aggregation.
- Return type:
- Returns:
Regression betas, optionally grouped by
model_group_expr(and/orcpg_expr).
- gnomad.utils.constraint.models.build_coverage_model(low_coverage_oe_expr, coverage_expr, log10_coverage=False)[source]
Build coverage model.
This function uses linear regression to build a model of coverage to correct proportion of expected variation at low coverage sites.
The x and y of the coverage model:
x: coverage_expr
y: low_coverage_oe_expr
- Parameters:
low_coverage_oe_expr (
Float64Expression) – The Float64Expression of observed:expected ratio for a given coverage level.coverage_expr (
Float64Expression) – The Float64Expression of the coverage expression.log10_coverage (
bool) – Whether to convert coverage sites by log10 when building the coverage model. Default is False.
- Return type:
- Returns:
StructExpression with intercept and slope of the model.
- gnomad.utils.constraint.models.apply_plateau_models(mu_expr, plateau_models_expr)[source]
Compute the predicted probability observed.
The predicted probability observed is computed as the mutation rate adjusted by the plateau model.
- Parameters:
mu_expr (
Float64Expression) – Mutation rate expression.plateau_models_expr (
ArrayExpression) – This can be either a single plateau model, where the first element is the intercept and the second element is the slope, or an array of plateau models.
- Return type:
Union[ArrayExpression,Float64Expression]- Returns:
Float64Expression of the predicted probability observed, or an ArrayExpression of them when
plateau_models_expris an array of models.
- gnomad.utils.constraint.models.coverage_correction_expr(coverage_expr, coverage_model, low_coverage_expr=None, coverage_cutoff=None, log10_coverage=False)[source]
Compute the coverage correction expression.
Note
One and only one of low_coverage_expr or coverage_cutoff must be specified.
The coverage correction expression is computed as follows:
If the coverage is 0, the coverage correction is 0.
If the low coverage expression (low_coverage_expr) is True, or the coverage (coverage_expr) is below the coverage cutoff (coverage_cutoff), the coverage model is applied to the coverage.
Otherwise, the coverage correction is 1.
- Parameters:
coverage_expr (
Float64Expression) – Float64Expression of the coverage.coverage_model (
Union[Tuple[float,float],ArrayNumericExpression]) – Coverage model as[intercept, slope], either a tuple of floats or an ArrayNumericExpression (as returned bybuild_models).low_coverage_expr (
Optional[BooleanExpression]) – Optional BooleanExpression indicating whether the site is a low coverage site, and the coverage model should be applied. Default is None.coverage_cutoff (
Optional[int]) – Optional coverage cutoff. If specified, the coverage model is applied to sites with coverage below this cutoff. Default is None.log10_coverage (
bool) – Whether to convert coverage sites by log10 when applying the coverage model. Default is False.
- Return type:
- Returns:
Float64Expression of the coverage correction.
- gnomad.utils.constraint.models.apply_models(mu_expr, plateau_models_expr, possible_variants_expr, coverage_model=None, coverage_expr=None, cpg_expr=None, model_group_expr=None, high_cov_definition=30, log10_coverage=True)[source]
Apply calibration models to compute expected variant counts.
Applies plateau and (optionally) coverage models to produce a struct with
mu,predicted_proportion_observed,expected_variants, and (when a coverage model is provided)coverage_correction.Note
When a coverage model is applied, a row whose
model_group_expris missing gets missingmu,expected_variants, andcoverage_correction(unless its coverage is 0, which always gives a correction of 0).calibration_model_group_expris missing for sites in neither model, such as sites aboveupper_cov_cutoffor at or belowlow_cov_cutoff.hl.agg.sumskips missing values, so filter these rows out before applying the models if they should not silently drop out of aggregated expected counts.- Parameters:
mu_expr (
Float64Expression) – Mutation rate expression.plateau_models_expr (
ArrayExpression) – Single plateau model (array of [intercept, slope]) or an array of plateau models.possible_variants_expr (
Int64Expression) – Possible variant counts to multiply the predicted proportion observed by.coverage_model (
Union[Tuple[float,float],ArrayNumericExpression,None]) – Coverage model as[intercept, slope], either a tuple of floats or an ArrayNumericExpression (as returned bybuild_models). Default is None.coverage_expr (
Optional[Int32Expression]) – Int32Expression of the coverage. Required whencoverage_modelis provided ormodel_group_expris None.cpg_expr (
Optional[BooleanExpression]) – BooleanExpression indicating CpG sites. Required whenmodel_group_expris None.model_group_expr (
Optional[StructExpression]) – Expression withhigh_or_low_coverageannotation to group variants into high or low coverage models. If not provided,calibration_model_group_expris used. Default is None.high_cov_definition (
int) – Coverage threshold for high/low classification. Default isCOVERAGE_CUTOFF.log10_coverage (
bool) – Whether to log10-transform coverage when applying the coverage model. Default is True.
- Return type:
- Returns:
StructExpression with “mu”, “predicted_proportion_observed”, “expected_variants”, and optionally “coverage_correction”.