Hail massively leaking memory

The post you’re looking for is: How do I increase the memory or RAM available to the JVM when I start Hail through Python?. Spark through Python is a bit more complicated than running a plain old java app.

On your second point, you correctly understand how Hail / Spark works.

I can’t gainfully comment on the 200 MB of RAM usage without the Hail log files, the script, and a profiler hooked up to the JVM. When using our hailctl utility, a Hail driver node has 16 cores and ~60GB of RAM. It’s entirely possible that Hail or Spark is doing something silly and holding ~200MB of RAM around because we usually have an abundance of memory available.

I empathize with the pain of diagnosing performance issues in a JVM controlled via Python. We’re actively moving away from Spark & the JVM for that reason but its a multi-year project.