Estimating computational cost for a hail task

Hail Query & hailctl
1

Objective:
We need to find compute power(amount of ram and cores used) for hail tasks executed.

Challenges:

  • hail tasks running in parallel make it difficult to find computational cost(we are using a workflow manager - luigi on top of it
  • task information is lost at spark interface i.e. spark job, stages and executor UI do not contain information about which hail task was executed

Mechanisms:

  • Slurm provides info about computation but at macro level. It won’t work in case of parallel tasks
  • Spark logs don’t contains information about computation cost. They only give status for success and failure
  • Pyspark python library doesn’t provide information about amount of ram or cores used. It only displayes success or failure for a task.
  • Spark REST api(Monitoring and Instrumentation - Spark 3.1.2 Documentation) provides various information about executor task time etc. Also, we need to map spark job to hail task.

PS: I have seen this thread Link but it discusses the topic in terms of money

2

Hi @danking and @tpoterba,
Your inputs on this would be appreciated!.

3

This is not an easy thing to do. You can get an approximation of memory usage by checking the Hail log. We have messages like:

2021-07-05 23:06:05 root: INFO: TaskReport: stage=54, partition=0, attempt=0, peakBytes=196864, peakBytesReadable=192.25 KiB

Which should be the maximum memory used by Hail per Spark task. It only tracks memory managed by Hail directly though, not any Java objects that might be created. There are methods like linear_regression_rows that currently make some large Java objects, and their memory usage will undercounted here. We are working on this.

CPU use is also hard to estimate. Roughly it should be total job time multiplied by number of cores, but if a pipeline does a lot of work on the driver node at some point, this will overestimate. You may be able to get more precise task start and stop times from Spark.

4

I did look into hail logs but I couldn’t find any term like TaskReport.

These were the common logs

2021-07-08 01:02:57 root: INFO: Timer: Time taken for LowerMatrixToTable -- Verify : 0.090ms, total 23.506ms
2021-07-08 01:02:57 root: INFO: Timer: Time taken for LowerMatrixToTable : 3.748ms, total 23.566ms, tagged coverage 93.2
2021-07-08 01:02:57 root: INFO: Timer: Time taken for Optimize -- FoldConstants : 0.817ms, total 29.661ms
2021-07-08 01:02:57 root: INFO: Timer: Time taken for Optimize -- ExtractIntervalFilters : 0.283ms, total

So, the higher-level goal here is to find tasks that are taking more than a day and see if they can be parallelized further by scaling or redefining the pipeline.

I have been looking at spark logs. I am not sure how to relate them to hail tasks.

Should I check spark jobs between specified times and get cpu and ram usage across all executors?

5

In case, there is no out of the box solution, I will try to provide rough estimate to my team. I can look at the spark UI for the jobs or task started and filter them by time brackets.

E.g. Computation done between 11:25 PM to 11:45 PM must sample_qc only. However, it may be drastically incorrect in case there are parallel tasks running.

This can be achieved via two ways:

  1. Spark REST API - It exposes the values of the Task Metrics collected by Spark executors with the granularity of task execution.
    This is not possible for us since we do not have a REST API setup. Our cluster is managed by a third party source.

  2. Selenium automation: I can scrape Spark UI and extract parameters.

6

We don’t currently have any way to determine how long a particular operation (like sample_qc) takes. By the time execution begins, the context of what python commands created the Spark jobs is lost.

This is hard to do because we have an optimizing compiler that will fuse operations together. Like, if you do a sample_qc, then an annotate_cols, we will recognize that those are both column annotation operations and fuse them together into one bulk column annotation, so it no longer makes sense to ask how long one or the other took, since they became one operation.

1 Like