# DNANexus: hail installation on custom spark cluster

**URL:** https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527
**Category:** Development
**Created:** [February 24, 2022, 8:20am UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527 "2022-02-24T08:20:04Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Abhishek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/abhishek/32/570_2.png) [@Abhishek](https://discuss.hail.is/u/Abhishek)
#### Post date: [February 24, 2022, 8:20am UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527/1 "2022-02-24T08:20:04Z")

</div>

Hi all,

_Context:_ I am working on DNANexus platform. I am trying to install hail version 0.2.83 with spark 3.2.0 version.

```auto
WARNING: Hail primarily tested with Spark 3.1.2, use other versions at your own risk.
* FAILURE: Build failed with an exception.
* * Where:
* Build file '/builds/hail/hail/build.gradle' line: 196
* * What went wrong:
* A problem occurred evaluating root project 'hail'.
* > Couldn't pick a valid elasticsearch.
* * Try:
* Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* * Get more help at https://help.gradle.org
* BUILD FAILED in 14s
* make: *** [Makefile:87: build/libs/hail-all-spark.jar] Error 1

```

These are my installation instructions:

```auto
HAIL_VERSION=0.2.83
git clone https://github.com/hail-is/hail.git
cd hail/hail
git checkout $HAIL_VERSION

make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.2.0 HAIL_PYTHON3=python3

```

I have gone through this thread: [Link](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216)

PS: I tried other versions but got below error

```auto
Error: The specified software version is not supported for APACHESPARK. Supported versions are: 2.4.0,2.4.4,3.0.1,3.2.0,3.2.0_alpha_h331, code 422. Request Time=1645689251.5551283, Request ID=1645689251599-716677

```

---

<div class="post-metadata">

### Author: ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)
#### Post date: [February 24, 2022, 6:27pm UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527/2 "2022-02-24T18:27:57Z")

</div>

This pull request should add support for Spark 3.2.0. I also expect a new release sometime this week.

> <https://github.com/hail-is/hail/pull/11410#pullrequestreview-892827517>
>
> This PR drastically simplifies the hail/elasticsearch story. We no longer requir…e any special custom built JAR from google cloud (was necessary because there was a time when Spark 3 compatible ES wasn't on Maven). Also allow ES version 7 or 8 when on Spark 3, and don't require that it's Spark 3.1

---

<div class="post-metadata">

### Author: ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [February 28, 2022, 6:39pm UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527/3 "2022-02-28T18:39:24Z")

</div>

This has merged now, would be curious to know if the latest main allows you to build on DNANexus

---

<div class="post-metadata">

### Author: ![Abhishek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/abhishek/32/570_2.png) [@Abhishek](https://discuss.hail.is/u/Abhishek)
#### Post date: [March 3, 2022, 7:14am UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527/4 "2022-03-03T07:14:07Z")

</div>

Yes, I was able to successfully build it. Thanks for the patch fix John(@johnc1231) 🙂
