I just ran a large job, finishing with a matrix_table.write() command after splitting multiallelics in a VDS file and producing the matrix table, and doing minor QC (removing failed tests in columns, without calling variant_qc). After the write command (which I verified the correct stage by seeing when the file began to write to the bucket), it proceeded to do a computationally intense stage that took longer than the entire pipeline and is ongoing. I am trying to understand what is going on, so resources are not wasted.
Code:
vds = hl.vds.read_vds(vds_path)
mt = hl.vds.to_dense_mt(hl.vds.VariantDataset(vds.reference_data, mt))
mt_plink = hl.split_multi(mt)
out_path = f'{my_bucket}/data/hail_{curr_chr}_pre_variant_qc.mt'
mt_plink.write(out_path, overwrite = True)