Running hail locally - number of cores

[ag14774]

It is unclear from the docs currently how to correctly initialize hail to maximize performance. In my case I am running Hail locally on a large machine with 1TB of memory and 96 cores. 1) Does hail automatically use all the available memory? 2) Does hail automatically use all available cores? What do I need to adjust to make hail use all resources? Any particular configurations to pass to hail.init()?

Thank you

[tpoterba]

Hail running on Apache Spark in local mode will use all available cores by default, but you can load the Spark WebUI (the URL is printed in initialization) and look at the Executors pane to ensure everything is getting used. Using hl.init(master='local[96]') can explicitly request 96 local cores.

You should also make sure that you set memory with PYSPARK_SUBMIT_ARGS:

How do I increase the memory or RAM available to the JVM when I start Hail through Python?

You can set the memory using an environment variable: PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"

[ag14774]

Thanks for the reply!

In the “Environment” tab it also says this under “Resource Profiles”:

Executor Reqs:
	cores: [amount: 1]
Task Reqs:
	cpus: [amount: 1.0]

In the Executors pane though it shows all cores. Does that mean it is all ok and it is using all cores?

[tpoterba]

Should be, yes!

Spark sometimes has issues with resource contention when running with such a high degree of multithreading in a single java process, so post here if anything else looks funky as you start running computations.