Tell Hail to make different use of Spark processing

Hail Query & hailctl
1

Hello,

I would like to know if there is any option to tell Spark to run just on disk or use a different ratio of in-memory processing. Any way to easily perform benchmarking ?

Thanks,
Pedro

2

Hi Pedro,
Spark configuration is pretty tough, and we’re not especially good at it (we’re also not sure who is). You can change the memory settings here: https://spark.apache.org/docs/2.2.0/configuration.html

Regarding benchmarking, we’re building a benchmarking suite but it’s not at a point where it’s especially useful beyond our development work.