# Pip-installed Hail requires additional configuration options in Spark referring to the path to the Hail Python module directory HAIL\_DIR

**URL:** <https://discuss.hail.is/t/pip-installed-hail-requires-additional-configuration-options-in-spark-referring-to-the-path-to-the-hail-python-module-directory-hail-dir/2685>\
**Category:** Hail Batch & General Cloud\
**Created:** [May 31, 2022, 10:09am UTC](https://discuss.hail.is/t/pip-installed-hail-requires-additional-configuration-options-in-spark-referring-to-the-path-to-the-hail-python-module-directory-hail-dir/2685 "2022-05-31T10:09:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bpark](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/bpark/32/563_2.png) [@bpark](https://discuss.hail.is/u/bpark)\
**Post date:** [May 31, 2022, 10:09am UTC](https://discuss.hail.is/t/pip-installed-hail-requires-additional-configuration-options-in-spark-referring-to-the-path-to-the-hail-python-module-directory-hail-dir/2685/1 "2022-05-31T10:09:02Z")

</div>

Hi

I build hail 0.2.95, and it is OK.

I run pyspark with jupyter notebook

hl.init(sc=sc)

The message is out this.

pip-installed Hail requires additional configuration options in Spark referring  
to the path to the Hail Python module directory HAIL\_DIR,  
e.g. /path/to/python/site-packages/hail:  
spark.jars=HAIL\_DIR/backend/hail-all-spark.jar  
spark.driver.extraClassPath=HAIL\_DIR/backend/hail-all-spark.jar  
spark.executor.extraClassPath=./hail-all-spark.jarTraceback (most recent call last):

This is pyspark cmd that I run.

> export SPARK\_HOME=/home1/sshuser/spark-3.2.0-bin-without-hadoop  
> export SPARK\_CONF\_DIR=/home1/sshuser/spark-3.2.0-bin-without-hadoop/conf  
> export SPARK\_SUBMIT\_OPTS=“-Dhdp.version=3.1.0.0-78”  
> export PATH=$SPARK\_HOME/bin:$PATH  
> export SPARK\_DIST\_CLASSPATH=`$HADOOP_COMMON_HOME/bin/hadoop classpath`  
> export PYSPARK\_PYTHON=python3  
> export PYSPARK\_DRIVER\_PYTHON=jupyter  
> export PYSPARK\_DRIVER\_PYTHON\_OPTS=‘notebook --no-browser --ip=0.0.0.0 --port 8888’

> pyspark --conf spark.jars=/home1/sshuser/.local/lib/python3.7/site-packages/hail/backend/hail-all-spark.jar   
> –conf spark.driver.extraClassPath=/home1/sshuser/.local/lib/python3.7/site-packages/hail/backend/hail-all-spark.jar   
> –conf spark.executor.extraClassPath=./hail-all-spark.jarTraceback   
> –conf spark.driver.extraJavaOptions=-Dhdp.version=3.1.0.0-78   
> –conf spark.yarn.am.extraJavaOptions=-Dhdp.version=3.1.0.0-78   
> –conf spark.yarn.appMasterEnv.YARN\_CONTAINER\_RUNTIME\_DOCKER\_MOUNTS=/usr/hdp:/usr/hdp:ro   
> –conf spark.executorEnv.YARN\_CONTAINER\_RUNTIME\_DOCKER\_MOUNTS=/usr/hdp:/usr/hdp:ro

Thanks

---

<div class="post-metadata">

**Author:** ![bpark](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/bpark/32/563_2.png) [@bpark](https://discuss.hail.is/u/bpark)\
**Post date:** [June 3, 2022, 6:48am UTC](https://discuss.hail.is/t/pip-installed-hail-requires-additional-configuration-options-in-spark-referring-to-the-path-to-the-hail-python-module-directory-hail-dir/2685/2 "2022-06-03T06:48:05Z")

</div>

In CentOS7.8

sudo yum install python3

is Python 3.6.

And then I built Python 3.7 source.

After then, something was wrong with a client for the Spark cluster.

I rebuild the Spark cluster and client.

Skip sudo yum install python3. Just make install Python 3.7 from source.

hail initialize OK!

Thanks!