DNANexus: hail installation on custom spark cluster

Development
1

Hi all,

Context: I am working on DNANexus platform. I am trying to install hail version 0.2.83 with spark 3.2.0 version.

WARNING: Hail primarily tested with Spark 3.1.2, use other versions at your own risk.
* FAILURE: Build failed with an exception.
* * Where:
* Build file '/builds/hail/hail/build.gradle' line: 196
* * What went wrong:
* A problem occurred evaluating root project 'hail'.
* > Couldn't pick a valid elasticsearch.
* * Try:
* Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* * Get more help at https://help.gradle.org
* BUILD FAILED in 14s
* make: *** [Makefile:87: build/libs/hail-all-spark.jar] Error 1

These are my installation instructions:

HAIL_VERSION=0.2.83
git clone https://github.com/hail-is/hail.git
cd hail/hail
git checkout $HAIL_VERSION

make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.2.0 HAIL_PYTHON3=python3

I have gone through this thread: Link

PS: I tried other versions but got below error

Error: The specified software version is not supported for APACHESPARK. Supported versions are: 2.4.0,2.4.4,3.0.1,3.2.0,3.2.0_alpha_h331, code 422. Request Time=1645689251.5551283, Request ID=1645689251599-716677
2

This pull request should add support for Spark 3.2.0. I also expect a new release sometime this week.

1 Like
3

This has merged now, would be curious to know if the latest main allows you to build on DNANexus

4

Yes, I was able to successfully build it. Thanks for the patch fix John(@johnc1231) :slight_smile: