Running hail locally - number of cores

[tpoterba]

Hail running on Apache Spark in local mode will use all available cores by default, but you can load the Spark WebUI (the URL is printed in initialization) and look at the Executors pane to ensure everything is getting used. Using hl.init(master='local[96]') can explicitly request 96 local cores.

You should also make sure that you set memory with PYSPARK_SUBMIT_ARGS:

How do I increase the memory or RAM available to the JVM when I start Hail through Python?

You can set the memory using an environment variable: PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"