No change with partition_hint

Hail Query & hailctl
1

Hi Hail Team,

Under what conditions does a supplied partition_hint get used to repartition a Table?

With the below code, “new_ht” has the same number of partitions as “ht” after supplying a partition_hint (although “new_ht” is only slightly smaller than “ht”).

ht = hl.read_table(ht_path)

grouping = hl.struct(context=ht.context, 
                      ref=ht.ref, 
                      alt=ht.alt, 
                      methylation_level=ht.methylation_level,
                      exome_coverage=ht.exome_coverage)

agg = {"variant_count": hl.agg.count()}

new_ht = ht.group_by(**grouping).partition_hint(n=100).aggregate(**agg)

ht.n_partitions()
new_ht.n_partitions()
2

Hi @klaricch,

Great question! This looks like a bug to me, and I think should be an easy fix.

Just to be clear on the context, this isn’t about repartitioning tables, it’s about how many partitions to use for the result of a group_by and aggregate operation, which potentially can have many fewer rows.

3

This fix will land in 0.2.120

4

OK, thank you!

5

I’ve updated to the most recent version of Hail, but I’m still having the same issues with partition hint