Has anyone used Slurm as a submission system for hail to a local cluster? I’m in the middle of setting it up and wanted to make sure I understood how to let hail know it has X many cpus available with Y amount of memory. Thanks!
Hi Nate,
The real question is setting up Spark – running Hail is pretty easy when you’ve got a Spark cluster running.
I found a page here: https://www.princeton.edu/researchcomputing/faq/spark-via-slurm/
Unfortunately, setting up Spark on top of HPC submission engines isn’t super simple, and I think this is going to be a problem for a lot of people.
It seems like the approach would be to write a script in python importing hail, and submit that as described in the link. Or, use a jupyter notebook attached to a running spark instance. I don’t see any particular roadblock here, though I have yet to try it. If the cluster can run spark, what other roadblocks are you anticipating?
Nope! Hail is really just a java jar and a python library. If you submit a Hail python script with spark-submit and supply the jar, everything should work. Let us know if things do go well! You’re not the first to have trouble with HPC systems.
Great, thank you! I’ll try it out today and report back with what I find.