Hi,
Hopefully a quick question from a relatively new Hail user.
I’m using Hail locally to merge datasets of approximately 50,000 single-sample VCFs. These are unfortunately not gVCFs. To aggregate these into one large Hail MatrixTable, I am using Hail’s import_vcf and a function I found via this forum, multi_way_union_mts, for this purpose. However, while the Spark cluster is initiated when Hail starts (locally over 12 compute nodes), multi_way_union_mts only seems to run on a single CPU. Is there a way to distribute this function across the Spark worker nodes?