ArrayIndexOutOfBoundsException error in hail 0.2.40

Hi Hail team,

I am running the following code

from gnomad.sample_qc.relatedness import filter_mt_to_trios
from gnomad.utils.filtering import filter_to_autosomes
from gnomad.variant_qc.pipeline import generate_trio_stats
from ukbb_qc.resources.basics import get_ukbb_data
from ukbb_qc.resources.sample_qc import (
    array_concordance_results_path,
    array_concordance_sites_path,
    inferred_ped_path,
    relatedness_ht_path,
)
from ukbb_qc.resources.variant_qc import var_annotations_ht_path


mt = get_ukbb_data(data_source, freeze)

ped_fp = inferred_ped_path(data_source, freeze)
ped = hl.Pedigree.read(ped_fp, delimiter="\t")
fam_ht = hl.import_fam(ped_fp, delimiter="\t")

mt = filter_to_autosomes(mt)
mt = filter_mt_to_trios(mt, fam_ht)
mt = hl.experimental.densify(mt)

mt = hl.trio_matrix(mt, pedigree=ped, complete_trios=True)
trio_stats_ht = generate_trio_stats(mt)
trio_stats_ht.naive_coalesce(n_partitions).write(
    var_annotations_ht_path("trio_stats", data_source, freeze),
     overwrite=True,
)

And I get a ArrayIndexOutOfBoundsException: 1 error. I am using hail 0.2.40 and the full log is attached.

I have previously run this code without error and at the time I was using this whl that Tim gave me:
https://storage.googleapis.com/hail-common/hailctl/dataproc/tpoterba-dev/0.2.33-d1759a52836e/hail-0.2.33-py3-none-any.whl

Thanks in advance for your help!

-Julia

Sorry you’re running into this problem. I don’t see where the log is attached though.

Yeah sorry I don’t think it finished uploading let me try again

It’s not letting me upload, will this work?

https://drive.google.com/open?id=1N8EM1qQxE0LxL1vpT79HSCpwEKZgsYU7

Yes, that’s helpful, thanks. Investigating.

Alright, Arcturus looked into this, and this is a bug we introduced in 0.2.40. There’s a fix here: https://github.com/hail-is/hail/pull/8786 , will be included in 0.2.41.

Probably the easiest thing for you to do is to use 0.2.39 until 0.2.41 comes out. If there’s some reason you specifically need 0.2.40, we’ll get a custom wheel going.

Great, thank you! I just got it rerunning with 0.2.39, I will let you know how it goes

Seems to still be a problem with 0.2.39. So maybe something changed about the code and there is an error on my end. Here is the log from the run on 0.2.39.

https://drive.google.com/open?id=1cf3zM9_16xcIzHIY96nFGQFo_dBAIJan

This is actually a different, related problem. I’ll dig into it more tomorrow, but hopefully won’t be too hard to fix / get you a patched wheel.

That is great to hear, thank you!

Figured out the problem, working on a fix currently.

Thank you!

This should fix: https://github.com/hail-is/hail/pull/8799

Our plan is just to release 0.2.41 after it goes in since other people will likely hit this bug too.

Slightly more complicated than expected, but we will plan on releasing 0.2.41 today

Sorry to hear it’s more complicated, and thank you for working so hard on it!

Ok, hail version 0.2.41 has been released, this should hopefully fix your problem.

Thank you so much for how quickly you put this in! I will check it now.

Small test worked perfectly! So I can kick off the larger job

Great! Glad it worked out. Thanks for the bug reports.