gnomad.utils.constraint.ranking
Functions to rank transcripts by a metric and assign percentile bins.
Rank rows by a numeric expression and assign bin labels. |
|
|
Compute approximate percentile thresholds for a metric expression. |
|
Annotate rows with bin assignments using pre-computed thresholds. |
|
Rank metrics within array elements and annotate rank structs back. |
Functions to rank transcripts by a metric and assign percentile bins.
- gnomad.utils.constraint.ranking.rank_and_assign_bins(value_expr, bin_granularities=None, prefix='')[source]
Rank rows by a numeric expression and assign bin labels.
Rank-based binning: every row receives a unique position in the sorted order, and bins are derived from that position. This differs from threshold-based binning (see
annotate_bins_by_threshold()), where bins are assigned by comparing values against pre-computed boundary values.Rows are ordered ascending by
value_expr. Each row is assigned a 0-based{prefix}rankand a{prefix}bin_{name}field for every entry inbin_granularities, computed ashl.int(rank * multiplier / n_rows).Rows where
value_expris missing are excluded from the ranking and receive missing rank and bin annotations.n_rowsis therefore the number of rows with a definedvalue_expr, so bins are evenly sized and the full range of bins is populated.Used by
rank_array_element_metrics()to rank metrics within array elements.Note
This function evaluates its input when it is called. It needs the number of rows with a defined
value_exprfor the bin denominator, so it counts them, and it caches the filtered and the ranked rows so the filter and the sort are not recomputed. In Hail,Table.cache()ignores the storage level and writes a checkpoint to apersist_Table*path under the temporary directory set byhl.init(tmp_dir=...), so with a GCStmp_dirthese Tables are written to GCS. They hold onlyvalue_exprand the key, and are not removed when the function returns.- Parameters:
value_expr (
Float64Expression) – Numeric expression to rank by (ascending).bin_granularities (
Optional[Dict[str,int]]) – Mapping of bin name to multiplier. Each entry produces a{prefix}bin_{name}field. Default is{"percentile": 100, "decile": 10, "sextile": 6}.prefix (
str) – String prepended torankandbin_{name}field names. Default is""(no prefix).
- Return type:
- Returns:
Struct with
{prefix}rankand{prefix}bin_{name}fields for each entry inbin_granularities.
- gnomad.utils.constraint.ranking.compute_percentile_thresholds(ht, metric_expr, outlier_expr=None, transcript_filter_expr=None, percentiles=(1, 5, 10, 15, 25, 50, 75), quantile_k=1000)[source]
Compute approximate percentile thresholds for a metric expression.
Threshold-based binning, step 1: computes the boundary values that define bin edges. Its values, keyed by
(granularity, metric)instead of percentile, form thethresholdsargument ofannotate_bins_by_threshold()(step 2), which assigns each row to a bin; see that function for an example.This two-step approach differs from rank-based binning (see
rank_and_assign_bins()), where every row receives a unique position in the sorted order. Threshold-based binning allows thresholds to be computed on a filtered subset (e.g., representative transcripts) and then applied to all rows, so multiple rows can share the same bin.Optionally filters to a subset of rows and excludes outliers, then computes approximate quantile thresholds at the requested percentiles in a single aggregation pass.
Note
Uses
hl.agg.approx_quantiles, so results are approximate. Increasequantile_kfor higher accuracy.- Parameters:
ht (
Table) – Input Table.metric_expr (
Float64Expression) – Float expression to compute thresholds for. Must be defined onht.outlier_expr (
Optional[BooleanExpression]) – Optional boolean expression that isTruefor rows to exclude. When None (default), no outlier filtering is applied.transcript_filter_expr (
Optional[BooleanExpression]) – Optional boolean expression that isTruefor rows to include. When None (default), all rows are included.percentiles (
Tuple[float,...]) – Percentile values (0-100) at which to compute thresholds. Default is (1, 5, 10, 15, 25, 50, 75).quantile_k (
int) – Accuracy parameter forhail.expr.aggregators.approx_quantiles(). Default is 1000.
- Return type:
Dict[float,float]- Returns:
Dict mapping each percentile to its threshold value.
- gnomad.utils.constraint.ranking.annotate_bins_by_threshold(ht, metric_exprs, thresholds, granularities, field_name='constraint_bins')[source]
Annotate rows with bin assignments using pre-computed thresholds.
Threshold-based binning, step 2: assigns each row to a bin by comparing its metric value against boundary values produced by
compute_percentile_thresholds()(step 1). For each(granularity, metric)pair, the bin equals the number of threshold boundaries the value exceeds. Bin 0 is the most constrained (below all thresholds); bin N means the value exceeds all N boundaries.This differs from rank-based binning (see
rank_and_assign_bins()), where every row gets a unique position. Here, multiple rows can share a bin, and the thresholds may have been derived from a different subset of rows than those being annotated.compute_percentile_thresholds()returns a dict of percentile to value, sothresholdsis built from it by selecting the percentiles for each granularity. Example use, with decile bins for two metrics:deciles = [10, 20, 30, 40, 50, 60, 70, 80, 90] metric_exprs = {"lof": ht.lof_oe_upper, "mis": ht.mis_oe_upper} thresholds = {} for metric, expr in metric_exprs.items(): pct_to_value = compute_percentile_thresholds(ht, expr, percentiles=deciles) thresholds[("decile", metric)] = [pct_to_value[p] for p in deciles] ht = annotate_bins_by_threshold(ht, metric_exprs, thresholds, ["decile"])
- Parameters:
ht (
Table) – Input Table.metric_exprs (
Dict[str,Float64Expression]) – Mapping of metric name to the Float64Expression to bin (e.g.{"lof": ht.lof_oe_upper, "mis": ht.mis_oe_upper}).thresholds (
Dict[Tuple[str,str],List[float]]) – Mapping of(granularity, metric)to a list of threshold values in ascending order, built fromcompute_percentile_thresholds()output (see the example above).granularities (
Union[List[str],Tuple[str,...]]) – Granularity names to iterate over (e.g.["decile", "ventile"]). Each must appear as the first element of at least one key inthresholds.field_name (
str) – Name of the struct field to annotate onht. Default is"constraint_bins".
- Return type:
- Returns:
Table with an added struct field containing per-granularity, per-metric bin assignments.
- gnomad.utils.constraint.ranking.rank_array_element_metrics(ht, array_field, element_value_fn, filter_fn=None, bin_granularities=None, rank_field_prefix='')[source]
Rank metrics within array elements and annotate rank structs back.
Rank-based binning for array fields: applies
rank_and_assign_bins()independently to each element of an array field. For each element,element_value_fnextracts named metric values, which are ranked across rows (optionally on a filtered subset viafilter_fn). Each array element is then annotated with{metric_name}_rankstructs containing rank and bin fields.Rows not matching
filter_fnget missing rank annotations.This differs from threshold-based binning (see
compute_percentile_thresholds()andannotate_bins_by_threshold()), where bins are assigned by comparing values against pre-computed boundaries rather than sorted position.Note
This function evaluates its input when it is called. It caches the full input Table (with an added row index) and the extracted metric values, reads one row to get the array length, and each
rank_and_assign_bins()call it makes caches its own ranked rows. Caching avoids recomputinghtandfilter_fnfor every metric, but in HailTable.cache()writes a checkpoint under the temporary directory (seerank_and_assign_bins()), so this writes a copy ofhtthere. The copies are not removed when the function returns.- Parameters:
ht (
Table) – Input Table.array_field (
str) – Name of the array field onht.element_value_fn (
Callable[[StructExpression],Dict[str,Float64Expression]]) – Function that takes an array element (StructExpression) and returns a dict mapping metric names to Float64Expressions to rank. Applied identically to every element.filter_fn (
Optional[Callable[[Table],BooleanExpression]]) – Optional function that takes a Table and returns a BooleanExpression to filter rows before ranking. When None, all rows are ranked.bin_granularities (
Optional[Dict[str,int]]) – Bin granularities passed torank_and_assign_bins().rank_field_prefix (
str) – Prefix for therankandbin_{name}sub-fields within each rank struct. Passed through torank_and_assign_bins(). Default is""(no prefix).
- Return type:
- Returns:
Table with
{metric_name}_rankstructs added to each array element. The table is returned with its original key restored.