I am running hail locally and seem to be getting BrokenPipeError and ConnectionRefusedError. I haven’t experienced these errors before, any suggestions for how to resolve? Currently, I’ve just been restarting the kernel in my Jupyter notebook.
Hi @Trisha_Karani ,
Sounds like you’re running out of memory on the driver. Our error messages are really bad in this case and we have outstanding work to improve this. Sorry about that.
Are you default-initialising hail in your notebook? Hail uses spark by default and you can configure how much memory the driver has with the spark_conf parameter:
Hail | Hail Query Python API (see spark_conf)
Here’s a link to various spark config paramters: Configuration - Spark 3.5.2 Documentation
You probably want to increase spark.driver.memory, perhaps start with 8G and go from there.
Hope this helps! Let me know if you’re still having this issue.
Ed