I’ve been working on some code for computing PRSes in AllOfUs, and was hitting warnings like: WARN TaskSetManager: Stage 2 contains a task of very large size (1528 KiB). The maximum recommended task size is 1000 KiB.
After some time debugging with claude, I tracked this down to jobs being launched across MTs with very many partitions (a single chromosome’s worth of AoU SNP calls, via hl.filter_intervals). The size of the task grew with the size of the chromosome being processed.
A fairly minor change (moving RDDPartition our of RDD) seems to have fixed this:
I ran make -C hail jvm-test - there were some failures but no new ones compared to before I changed this.