# Unable to initialize hail 0.2.72 with spark3

**URL:** <https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185>\
**Category:** Hail Batch & General Cloud\
**Created:** [August 16, 2021, 8:31am UTC](https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185 "2021-08-16T08:31:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhishek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/abhishek/32/570_2.png) [@Abhishek](https://discuss.hail.is/u/Abhishek)\
**Post date:** [August 16, 2021, 8:31am UTC](https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185/1 "2021-08-16T08:31:46Z")

</div>

We are trying to setup hail `0.2.72` on spark `3.1.2` version. However, we are facing error.

- java version: `OpenJDK 64-Bit Server VM, 1.8.0_242`
- scala version: `2.12.10`
- py4j: `0.10.9`
- Python: `3.7.10`

> **Stacktrace**
>
> ```auto
> Py4JJavaError: An error occurred while calling o126.exists.
> : java.lang.NoClassDefFoundError: com/amazonaws/AmazonClientException
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2532)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2497)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2593)
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at is.hail.io.fs.HadoopFS.fileStatus(HadoopFS.scala:164)
> at is.hail.io.fs.FS.exists(FS.scala:183)
> at is.hail.io.fs.FS.exists$(FS.scala:181)
> at is.hail.io.fs.HadoopFS.exists(HadoopFS.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:282)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonClientException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> ... 27 more
> 
> During handling of the above exception, another exception occurred:
> 
> Py4JJavaError Traceback (most recent call last)
> /bmrn/apps/hail/0.2.72-spark-3.1.2/python/hail-0.2.72-py3-none-any.egg/hail/backend/py4j_backend.py in deco(*args, **kwargs)
> 15 try:
> ---> 16 return f(*args, **kwargs)
> 17 except py4j.protocol.Py4JJavaError as e:
> 
> /bmrn/apps/python/miniconda/64/3.7/envs/piranha_0.2.0_20210812/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value)
> 329 else:
> 
> Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
> : java.lang.IllegalArgumentException: requirement failed
> at scala.Predef$.require(Predef.scala:268)
> at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:218)
> at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:282)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> 
> During handling of the above exception, another exception occurred:
> 
> Py4JJavaError Traceback (most recent call last)
> /bmrn/apps/hail/0.2.72-spark-3.1.2/python/hail-0.2.72-py3-none-any.egg/hail/backend/py4j_backend.py in deco(*args, **kwargs)
> 15 try:
> ---> 16 return f(*args, **kwargs)
> 17 except py4j.protocol.Py4JJavaError as e:
> 
> /bmrn/apps/python/miniconda/64/3.7/envs/piranha_0.2.0_20210812/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value)
> 329 else:
> 
> Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
> : java.lang.IllegalArgumentException: requirement failed
> at scala.Predef$.require(Predef.scala:268)
> at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:218)
> at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:282)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Thread.java:748)
> 
> During handling of the above exception, another exception occurred:
> 
> ```

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [August 16, 2021, 6:23pm UTC](https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185/2 "2021-08-16T18:23:25Z")

</div>

How did you build/install hail?

---

<div class="post-metadata">

**Author:** ![Abhishek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/abhishek/32/570_2.png) [@Abhishek](https://discuss.hail.is/u/Abhishek)\
**Post date:** [August 26, 2021, 10:29am UTC](https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185/3 "2021-08-26T10:29:59Z")

</div>

We built hail from source as we running on cluster.

---

<div class="post-metadata">

**Author:** ![Abhishek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/abhishek/32/570_2.png) [@Abhishek](https://discuss.hail.is/u/Abhishek)\
**Post date:** [August 26, 2021, 10:31am UTC](https://discuss.hail.is/t/unable-to-initialize-hail-0-2-72-with-spark3/2185/4 "2021-08-26T10:31:28Z")

</div>

It is running now. The error was in external shuffle service not getting started.

We copied `start-shuffle-service.sh` and `stop-shuffle-service.sh` from 2.4.5 version to 3.1.2.