Log of breaking changes in 0.2 beta

[jbloom]

This thread documents breaking changes as we work toward stabilizing the development branch (i.e., master, 0.2 beta) as Hail 0.2 proper.

[jbloom]

Removed as_array parameter from PCA

pca and hwe_normalized_pca no longer take an as_array parameter. They now always return scores and loadings as arrays (formerly the as_array=True option).

See the overview tutorial for example usage in GWAS, where PC1 becomes scores[0].

https://github.com/hail-is/hail/pull/3280

[jbloom]

Removed dataset parameter from eight methods

All methods that took a dataset and at least one required expression on that dataset no longer take a dataset parameter at all (the dataset is implicitly the source of the expression):

grm
linear_regression
logistic_regression
linear_mixed_regression
pc_relate
pca
rrm
skat

https://github.com/hail-is/hail/pull/3211
https://github.com/hail-is/hail/pull/3262

[jbloom]

Changed ys to y and schema in linear regression

Consistent with the other statistics methods, the parameter ys on linear_regression is now y, and when y is an expression the linreg fields all have type float64. This is consistent with the other regression methods.

When y is a list of expressions (even a list of one expression) the behavior is the same as before: the the five y-dependent linreg fields have type array[float64].

The field n_complete_samples is now just n.

See the overview tutorial for example usage of the case where y is an expression. In particular, linear_regression_results.linreg.p_value[0].collect() no longer takes [0].

https://github.com/hail-is/hail/pull/3295

[wang]

See:

[Breaking change] New changes to select, annotate, key_by interface for 0.2

https://github.com/hail-is/hail/pull/3297 This PR will change the way keys on Tables and MatrixTables are dealt with in certain functions. (The doc links will be updated once this pull request lands.) key_by(*expr, **named_exprs) (and key_rows_by and key_cols_by) is the only method that can modify key fields. The interface is identical to the current select interface, where non-named exprs must be field references (not necessarily top-level), but non-field-reference expressions can still be u…

[tpoterba]

Oops. See:

[Breaking change] Filter alleles

Though we have reason to believe nobody has been using this method, we are committed to announcing all breaking changes! The interface for filter_alleles has changed. Take a look at the new methods: hl.filter_alleles hl.filter_alleles_hts

[Meredith_Accum]

ld_prune has changed to take a CallExpression instead of a matrix table. The new signature is ld_prune(call_expr, r2=0.2, window=1000000, memory_per_core=256).

See: https://github.com/hail-is/hail/pull/3518

[jbloom]

ld_prune no longer requires unphased genotypes (though it still makes no use of phasing information). And the parameter window has been renamed bp_window_size.

See: https://github.com/hail-is/hail/pull/3575

[konradjk]

While we’re at it, it also returns a Table with just ('locus', 'alleles') that is the set of independent variants at that threshold (rather than previously returning the MatrixTable filtered to that set).

[cseed]

See:

[Breaking change] Hail 0.2 no longer supports BGEN 1.1

As of the PR support only bgen12, Hail 0.2 no longer supports BGEN 1.1 and only supports BGEN 1.2 with the 8-bit genotype probability encoding. If you have BGEN 1.1 data (or 1.2 with non-8-bit encoding), you can use Hail 0.1 or use qctool to convert your data to BGEN 1.2.

[danking]

see:

[Breaking Change] Hail 0.2 import_bgen should be passed a min_partitions parameter

Previously, some Spark heuristics chose a reasonable number of partitions for import_bgen. This logic has been removed while we improve bgen performance (we may add the logic back in before 0.2 release). In the meantime, you will notice that import_bgen uses far fewer partitions than is appropriate. You can use the min_partitions argument to force a higher number of partitions. For the UKBB dataset, try using around 18,000 partitions.

[jbloom]

see:

[Breaking Change] Hail 0.2 removed KinshipMatrix, linear_mixed_regression

(Update: Hail 0.2 has a function linear_mixed_regression_rows which most closely implements the old linear_mixed_regression function) The KinshipMatrix class is gone. Instead, realized_relationship_matrix and genetic_relatedness_matrix both return a BlockMatrix with improved performance. The gcta_grm and gcta_grm_bin on KinshipMatrix are therefore gone as well. BlockMatrix supports export, as well as and conversion to_numpy for further processing if small enough (less than 46k samples, assuming…

[jbloom]

see:

[Breaking Change] Rename of methods/fields: ctt, chisq, hardy_weinberg, hardy_weinberg_p, variant_qc, transition_disequilibrium_test

For clarity and consistency, we’ve renamed the following: Functions: ctt => contingency_table_test chisq => chi_squared_test hardy_weinberg => hardy_weinberg_test Aggregator: hardy_weinberg_p => hardy_weinberg_test Fields in hardy_weinberg_test: p_hwe => p_value r_expected_het_freq => het_freq_hwe (the expected frequency of het under HWE) Fields in variant_qc result: p_hwe => p_value_hwe r_expected_het_freq => het_freq_hwe Field in transition_disequilibrium_test result: chi2 => chi…

[jbloom]

see:

[Breaking Change] sample_qc

The following changes have been made to sample_qc: The fields gq_mean and gq_stdev have been replaced with gq_stats that is a struct with four fields: mean, stdev, min, and max. The fields dp_mean and dp_stdev have been replaced with dp_stats that is a struct with four fields: mean, stdev, min, and max. n_singleton now includes singleton alleles that are the reference allele.

[jbloom]

see:

[Breaking Change] overwrite parameter on BlockMatrix write and write_from_entry_expr

BlockMatrix write and write_from_entry_expr now take an overwrite parameter, with default value False.

[jbloom]

see:

[Breaking Change] Intercept is now optional in regression and SKAT

To give users more control, we’ve changed inclusion of an intercept from implicit to explicit in linear regression, logistic regression, and SKAT. Consider the linear regression model y = b*x + b0 + b1*c1 + b2*c2 + e where we are interested the effect size b on x per row of a matrix table mt, and b0 represents an intercept. BEFORE the intercept was implicitly added so you would write: hl.linear_regression(y=mt.y, x=mt.x, covariates=[mt.c1, mt.c2]). NOW the intercept must be included explici…

[konradjk]

Minor breaking change: hl.min_rep() now returns struct of locus (a LocusExpression) and alleles (an ArrayExpression of type str). This makes min_rep and re-key much easier as in:

mt = mt.key_rows_by(**hl.min_rep(mt.locus, mt.alleles))

[wang]

minor change:

the parameter names of hl.rand_unif(min, max) are changing to lower and upper.

https://github.com/hail-is/hail/pull/4145