Get Spark query execution plans for Hail commands

Hail Query & hailctl
1

Hi, are there any utilities built into Hail for gathering Spark execution plans after the intermediate representation is translated into Spark commands? Or is the Spark UI the best place to go for figuring out what goes on when a command is run?

I would imagine some Hail commands correspond to multiple Actions so getting a single plan is no doubt hard, but I thought I’d see if there were any tricks built in for getting that kind of visibility.

1 Like
2

There are currently no such utilities, so monitoring Hail pipelines remains an arcane craft. I think we’ll need to build better monitoring tools, but we’ll probably wait until whole stage code generation infrastructure is ready, which should be several months.

A shorter-term stop-gap could be to add more log output that describes when actions are run.

3

Fair enough, thanks Tim.