# \[Hail on apache spark\] Using pyspark, py4j.protocol.Py4JError

**URL:** <https://discuss.hail.is/t/hail-on-apache-spark-using-pyspark-py4j-protocol-py4jerror/2082>\
**Category:** Hail Query & hailctl\
**Created:** [June 14, 2021, 4:38am UTC](https://discuss.hail.is/t/hail-on-apache-spark-using-pyspark-py4j-protocol-py4jerror/2082 "2021-06-14T04:38:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dong](https://avatars.discourse-cdn.com/v4/letter/d/3e96dc/32.png) [@Dong](https://discuss.hail.is/u/Dong)\
**Post date:** [June 14, 2021, 4:38am UTC](https://discuss.hail.is/t/hail-on-apache-spark-using-pyspark-py4j-protocol-py4jerror/2082/1 "2021-06-14T04:38:26Z")

</div>

Hi, I’m studying Hail and installing Hail on spark.

I have plan that run GWAS about 1000 genomes. So, I install and set up hail on spark.

Linux: Centos 7.8  
Python: 3.7.3 (anaconda)  
Apache spark: spark-2.2.0-bin-hadoop2.6  
Hadoop: hadoop-2.6.0  
Java -version (info. I’m using linux server by korea Institution, So i can’t use root permission)  
openjdk version “1.8.0\_262”  
OpenJDK Runtime Environment (build 1.8.0\_262-b10)  
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)  
Hail version: 0.2.68

1. Run start-master.sh and start-slaves.sh in spark sbin directory.
2. (bash) pyspark

I got message below.

 ![1](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/2320654daed36e6cfe2c932cb96f7bc7afbb3688.jpeg)  
 ![2](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/c9ee0a4bc303af7bef905c1da8505dc0a8ab087d.jpeg)  
 ![3](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/3683dc23892339ea75e8b98ea89a76ec542dd4a2.jpeg)

How can i set up hail on spark?  
Do i need to change java version?

Thank you for your services.

My , \<conf/spark-defaults.conf\> and \<./spark-env.sh\> are below.

**\<.bashrc\>**

```auto
#SPARK
export SPARK_HOME=/home/edu1/tools/spark-2.2.0-bin-hadoop2.6
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/python:$PATH
export PYTHONPATH=$HAIL_HOME/python:$SPARK_HOME/python:$(echo ${SPARK_HOME}/python/lib/py4j-*-src.zip):$PYTHONPATH

# Hail
export HAIL_HOME=/home/edu1/miniconda2/envs/Hail-on-spark/lib/python3.7/site-packages/hail
export PATH=$PATH:$HAIL_HOME/bin
export PYTHONPATH=$PYTHONPATH:$HAIL_HOME/python
export SPARK_CLASSPATH=$HAIL_HOME/backend/hail-all-spark.jar

# JAVA (I just can modify .bashrc, so This would not apply to java path.)
export JAVA_HOME=/home/edu1/tools/jdk-1.8.0_231
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/lib/tools.jar

# Hadoop
export HADOOP_INSTALL=/home/edu1/tools/hadoop-2.6.0
export PAHT=$PATH:$HADOOP_INSTALL/bin
export LD_LIBRARY_PATH=$HADOOP_INSTALL/lib/native

```

**\</spark/conf/spark-defaults.conf\>**

```auto
spark.master spark://training.server:7077

spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator is.hail.kryo.HailKryoRegistrator
spark.speculation True

spark.driver.memory 37414m
spark.executor.memory 37414m
spark.executor.instances 1

spark.driver.extraClassPath /home/edu1/miniconda2/envs/Hail-on-spark/lib/python3.7/site-packages/hail/backend/hail-all-spark.jar
spark.executor.extraClassPath /home/edu1/miniconda2/envs/Hail-on-spark/lib/python3.7/site-packages/hail/hail-all-spark.jar
spark.jars /home/edu1/miniconda2/envs/Hail-on-spark/lib/python3.7/site-packages/hail/backend/hail-all-spark.jar

spark.eventLog.enabled true
spark.history.fs.logDirectory file:/tmp/spark-events
spark.enevtLog.dir file:/tmp/spark-events

spark.ui.reverseProxy true
spark.ui.reverseProxyUrl spark://training.server/spark
spark.executor.extraJavaOptions -Dlog4j.debug=true

```

**\</spark/conf/spark-env.sh\>**

```auto
export SPARK_WORKER_INSTANCES=1

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [June 15, 2021, 12:07pm UTC](https://discuss.hail.is/t/hail-on-apache-spark-using-pyspark-py4j-protocol-py4jerror/2082/2 "2021-06-15T12:07:18Z")

</div>

I think the problem here is that your Spark version is not the same version Hail is compiled against. Hail can be compiled against. I’m assuming that you installed Hail with pip?

Are you actually running a cluster, or just running Spark to multithread on a single server? If you are running on a single server, running in local mode will be much easier – just pip install Hail and its pyspark dependency, unset all your spark variables like `SPARK_HOME`, and it should work.

---

<div class="post-metadata">

**Author:** ![Dong](https://avatars.discourse-cdn.com/v4/letter/d/3e96dc/32.png) [@Dong](https://discuss.hail.is/u/Dong)\
**Post date:** [July 14, 2021, 4:27am UTC](https://discuss.hail.is/t/hail-on-apache-spark-using-pyspark-py4j-protocol-py4jerror/2082/3 "2021-07-14T04:27:01Z")

</div>

Hi !  
I did install hail by using pip

Thank you for your help!