Possible vcf_combiner issue

[tpoterba]

Ah, when Spark starts in local mode, it allocates a very small amount of memory (<1G) by default. You can set this using an env variable, instructions here:

How do I increase the memory or RAM available to the JVM when I start Hail through Python?

You can set the memory using an environment variable: PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"