Hail configuration

Hail Query & hailctl
1

Hi,

I am using Hail 0.2 and I am trying to change configuration parameters. So, first using python3 in the PYSPARK_SUBMIT_ARGS variables I add a --driver-memory, --driver-cores and --executor-cores. The --driver-memory works but not the limitation of cores and Hail import_vcf uses all my cores. I also tried with spark-submit but hail uses all cores. With the same configuration of spark-submit reading a csv file with spark only uses the cores of the parameters… so hail is not using correctly the parameters.
I am using Spark 2.4.

2

Try:

  --master local[N_CORES]

in your PYSPARK_SUBMIT_ARGS

3

Thank you, works