gnomad.utils.constraint.ranking

Functions to rank transcripts by a metric and assign percentile bins.

gnomad.utils.constraint.ranking.rank_and_assign_bins(...)

Rank rows by a numeric expression and assign bin labels.

gnomad.utils.constraint.ranking.compute_percentile_thresholds(ht, ...)

Compute approximate percentile thresholds for a metric expression.

gnomad.utils.constraint.ranking.annotate_bins_by_threshold(ht, ...)

Annotate rows with bin assignments using pre-computed thresholds.

gnomad.utils.constraint.ranking.rank_array_element_metrics(ht, ...)

Rank metrics within array elements and annotate rank structs back.

Functions to rank transcripts by a metric and assign percentile bins.

gnomad.utils.constraint.ranking.rank_and_assign_bins(value_expr, bin_granularities=None, prefix='')[source]

Rank rows by a numeric expression and assign bin labels.

Rank-based binning: every row receives a unique position in the sorted order, and bins are derived from that position. This differs from threshold-based binning (see annotate_bins_by_threshold()), where bins are assigned by comparing values against pre-computed boundary values.

Rows are ordered ascending by value_expr. Each row is assigned a 0-based {prefix}rank and a {prefix}bin_{name} field for every entry in bin_granularities, computed as hl.int(rank * multiplier / n_rows).

Rows where value_expr is missing are excluded from the ranking and receive missing rank and bin annotations. n_rows is therefore the number of rows with a defined value_expr, so bins are evenly sized and the full range of bins is populated.

Used by rank_array_element_metrics() to rank metrics within array elements.

Note

This function evaluates its input when it is called. It needs the number of rows with a defined value_expr for the bin denominator, so it counts them, and it caches the filtered and the ranked rows so the filter and the sort are not recomputed. In Hail, Table.cache() ignores the storage level and writes a checkpoint to a persist_Table* path under the temporary directory set by hl.init(tmp_dir=...), so with a GCS tmp_dir these Tables are written to GCS. They hold only value_expr and the key, and are not removed when the function returns.

Parameters:
  • value_expr (Float64Expression) – Numeric expression to rank by (ascending).

  • bin_granularities (Optional[Dict[str, int]]) – Mapping of bin name to multiplier. Each entry produces a {prefix}bin_{name} field. Default is {"percentile": 100, "decile": 10, "sextile": 6}.

  • prefix (str) – String prepended to rank and bin_{name} field names. Default is "" (no prefix).

Return type:

StructExpression

Returns:

Struct with {prefix}rank and {prefix}bin_{name} fields for each entry in bin_granularities.

gnomad.utils.constraint.ranking.compute_percentile_thresholds(ht, metric_expr, outlier_expr=None, transcript_filter_expr=None, percentiles=(1, 5, 10, 15, 25, 50, 75), quantile_k=1000)[source]

Compute approximate percentile thresholds for a metric expression.

Threshold-based binning, step 1: computes the boundary values that define bin edges. Its values, keyed by (granularity, metric) instead of percentile, form the thresholds argument of annotate_bins_by_threshold() (step 2), which assigns each row to a bin; see that function for an example.

This two-step approach differs from rank-based binning (see rank_and_assign_bins()), where every row receives a unique position in the sorted order. Threshold-based binning allows thresholds to be computed on a filtered subset (e.g., representative transcripts) and then applied to all rows, so multiple rows can share the same bin.

Optionally filters to a subset of rows and excludes outliers, then computes approximate quantile thresholds at the requested percentiles in a single aggregation pass.

Note

Uses hl.agg.approx_quantiles, so results are approximate. Increase quantile_k for higher accuracy.

Parameters:
  • ht (Table) – Input Table.

  • metric_expr (Float64Expression) – Float expression to compute thresholds for. Must be defined on ht.

  • outlier_expr (Optional[BooleanExpression]) – Optional boolean expression that is True for rows to exclude. When None (default), no outlier filtering is applied.

  • transcript_filter_expr (Optional[BooleanExpression]) – Optional boolean expression that is True for rows to include. When None (default), all rows are included.

  • percentiles (Tuple[float, ...]) – Percentile values (0-100) at which to compute thresholds. Default is (1, 5, 10, 15, 25, 50, 75).

  • quantile_k (int) – Accuracy parameter for hail.expr.aggregators.approx_quantiles(). Default is 1000.

Return type:

Dict[float, float]

Returns:

Dict mapping each percentile to its threshold value.

gnomad.utils.constraint.ranking.annotate_bins_by_threshold(ht, metric_exprs, thresholds, granularities, field_name='constraint_bins')[source]

Annotate rows with bin assignments using pre-computed thresholds.

Threshold-based binning, step 2: assigns each row to a bin by comparing its metric value against boundary values produced by compute_percentile_thresholds() (step 1). For each (granularity, metric) pair, the bin equals the number of threshold boundaries the value exceeds. Bin 0 is the most constrained (below all thresholds); bin N means the value exceeds all N boundaries.

This differs from rank-based binning (see rank_and_assign_bins()), where every row gets a unique position. Here, multiple rows can share a bin, and the thresholds may have been derived from a different subset of rows than those being annotated.

compute_percentile_thresholds() returns a dict of percentile to value, so thresholds is built from it by selecting the percentiles for each granularity. Example use, with decile bins for two metrics:

deciles = [10, 20, 30, 40, 50, 60, 70, 80, 90]
metric_exprs = {"lof": ht.lof_oe_upper, "mis": ht.mis_oe_upper}
thresholds = {}
for metric, expr in metric_exprs.items():
    pct_to_value = compute_percentile_thresholds(ht, expr, percentiles=deciles)
    thresholds[("decile", metric)] = [pct_to_value[p] for p in deciles]

ht = annotate_bins_by_threshold(ht, metric_exprs, thresholds, ["decile"])
Parameters:
  • ht (Table) – Input Table.

  • metric_exprs (Dict[str, Float64Expression]) – Mapping of metric name to the Float64Expression to bin (e.g. {"lof": ht.lof_oe_upper, "mis": ht.mis_oe_upper}).

  • thresholds (Dict[Tuple[str, str], List[float]]) – Mapping of (granularity, metric) to a list of threshold values in ascending order, built from compute_percentile_thresholds() output (see the example above).

  • granularities (Union[List[str], Tuple[str, ...]]) – Granularity names to iterate over (e.g. ["decile", "ventile"]). Each must appear as the first element of at least one key in thresholds.

  • field_name (str) – Name of the struct field to annotate on ht. Default is "constraint_bins".

Return type:

Table

Returns:

Table with an added struct field containing per-granularity, per-metric bin assignments.

gnomad.utils.constraint.ranking.rank_array_element_metrics(ht, array_field, element_value_fn, filter_fn=None, bin_granularities=None, rank_field_prefix='')[source]

Rank metrics within array elements and annotate rank structs back.

Rank-based binning for array fields: applies rank_and_assign_bins() independently to each element of an array field. For each element, element_value_fn extracts named metric values, which are ranked across rows (optionally on a filtered subset via filter_fn). Each array element is then annotated with {metric_name}_rank structs containing rank and bin fields.

Rows not matching filter_fn get missing rank annotations.

This differs from threshold-based binning (see compute_percentile_thresholds() and annotate_bins_by_threshold()), where bins are assigned by comparing values against pre-computed boundaries rather than sorted position.

Note

This function evaluates its input when it is called. It caches the full input Table (with an added row index) and the extracted metric values, reads one row to get the array length, and each rank_and_assign_bins() call it makes caches its own ranked rows. Caching avoids recomputing ht and filter_fn for every metric, but in Hail Table.cache() writes a checkpoint under the temporary directory (see rank_and_assign_bins()), so this writes a copy of ht there. The copies are not removed when the function returns.

Parameters:
  • ht (Table) – Input Table.

  • array_field (str) – Name of the array field on ht.

  • element_value_fn (Callable[[StructExpression], Dict[str, Float64Expression]]) – Function that takes an array element (StructExpression) and returns a dict mapping metric names to Float64Expressions to rank. Applied identically to every element.

  • filter_fn (Optional[Callable[[Table], BooleanExpression]]) – Optional function that takes a Table and returns a BooleanExpression to filter rows before ranking. When None, all rows are ranked.

  • bin_granularities (Optional[Dict[str, int]]) – Bin granularities passed to rank_and_assign_bins().

  • rank_field_prefix (str) – Prefix for the rank and bin_{name} sub-fields within each rank struct. Passed through to rank_and_assign_bins(). Default is "" (no prefix).

Return type:

Table

Returns:

Table with {metric_name}_rank structs added to each array element. The table is returned with its original key restored.