# Spark 2.4.4 gets stuck in initialization phase **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [November 19, 2019, 7:44pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178 "2019-11-19T19:44:01Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [November 19, 2019, 7:44pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178/1 "2019-11-19T19:44:01Z")
Hi, I am working on ‘RHEL Server release 7.7’. I installed the recommended ‘miniconda3’, then created python 3.7 virtual environment: conda create -n py37 python=3.7 Then I installed hail using pip the way it is recommended: pip install hail py-spark and ipython: conda install -c conda-forge pyspark conda install -c anaconda ipython I have also downloaded ‘Spark2.4.4’, started one master and one slave. Then, I tried to run the basic script in various ways: > > > import hail as hl > > > mt = hl.balding\_nichols\_model(n\_populations=3, n\_samples=50, n\_variants=100) > > > mt.count() But it just gets stuck on the second line in Hail and Spark init and does not go anywhere further. No log output, no error. Whether I set $SPARK\_HOME or not, it does not fix it. I also set the path to HAIL jar directly when running it with spark-submit but the result is the same: spark-submit --master spark://ai-grisnodedev1:7077 --verbose --conf spark.driver.port=40065 --driver-memory 4g --conf spark.driver.extraClassPath=/opt/seqr/.conda/envs/py37/lib/python3.7/site-packages/hail/hail-all-spark.jar --conf spark.executor.extraClassPath=./hail-all-spark.jar test\_hail.py Or spark-submit --master spark://ai-grisnodedev1:7077 --verbose --conf spark.driver.port=40065 --driver-memory 4g --conf spark.driver.extraClassPath=/opt/seqr/.conda/envs/py37/lib/python3.7/site-packages/hail/hail-all-spark.jar --conf spark.executor.extraClassPath=/opt/seqr/.conda/envs/py37/lib/python3.7/site-packages/hail/hail-all-spark.jar test\_hail.py test\_hail.py just contains the 3 lines of the sample code. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [November 19, 2019, 7:50pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178/2 "2019-11-19T19:50:31Z")
Can you run Spark pipelines that don’t involve Hail? ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [November 19, 2019, 8:42pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178/3 "2019-11-19T20:42:05Z")
I launched spark-shell, loaded a file from Hadoop and counted the number of lines in it, working fine, or how else could I test it? ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [November 19, 2019, 10:29pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178/4 "2019-11-19T22:29:41Z")
Ok, we figured it out. The issue was that we do not have that much space on the partition where Hadoop and spark logs were written. After redirection of the logs to a different place on the node, it started to work. There was also a ‘java cp’ process that was triggered by the ‘spark-submit’ and it was filling up Hadoop log with one error over and over again: > Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE: If you turn off safe mode before adding resource s, the NN will immediately return to safe mode. Use “hdfs dfsadmin -safemode leave” to turn safe mode off. > 1808697 2019-11-19 16:22:20,204 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs fro m 137.187.60.61:44398 Call#33705121 Retry#0: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hail.wDXS6L3AD3Ta. Name node is in safe mode. It is still rather vague what was the issue, but that is just most relevant. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [November 19, 2019, 11:55pm UTC](https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178/5 "2019-11-19T23:55:33Z")
huh, weird! Glad you’re unblocked.