I’m having trouble setting a limit to the memory Hail uses when running it locally on a server. I have set spark.driver.memory, spark.executor.pyspark.memory and spark.executor.memory:
export PYSPARK_SUBMIT_ARGS="--conf spark.driver.memory=48G --conf spark.executor.pyspark.memory=48G --conf spark.executor.memory=48G --conf spark.task.maxFailures=5 pyspark-shell"
But the job still eats up >500G memory before getting killed. Could anyone suggest what I’m doing wrong?