Unable to deploy/compile latest Hail on aws

Development
1

Hi, I am trying to update Hail from 0.1 to the latest version but kept failing. Here is what I did (from this site: Hail | For Software Developers)


echo 'Installing pre-reqs'

          yum install -y g++ cmake git

          yum install -y lz4

          yum install -y lz4-devel

          git clone $HAIL_REPO

          cd hail/hail && git fetch && git checkout main

          ./gradlew clean

          make install HAIL_COMPILE_NATIVES=1

          make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=3.1.1

          cd build

          pip download decorator==4.2.1

          aws s3 cp decorator-4.2.1-py2.py3-none-any.whl s3://${RESOURCES_BUCKET}/artifacts/decorator.zip

          aws s3 cp distributions/hail-python.zip s3://${RESOURCES_BUCKET}/artifacts/

          aws s3 cp libs/hail-all-spark.jar s3://${RESOURCES_BUCKET}/artifacts/


And this is what I got


> Task :compileScala FAILED

 FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileScala'.
 > javac returned non-zero exit code


make: *** [build/libs/hail-all-spark.jar] Error 1

A problem occurred evaluating root project 'hail'.
286 > Could not resolve all dependencies for configuration ':justSpark'.
287 > Could not find org.apache.spark:spark-mllib_2.11:3.1.1.

Following by other errors too, but I am assuming it could be caused by previous errors.

How should I change my template?

Thank you!

2

Hi @yc000000 !

I’m sorry you’re having trouble installing Hail! The key issue is that Spark 3.1.1 is not compatible with Scala 2.11. That’s why you get this error:

287 > Could not find org.apache.spark:spark-mllib_2.11:3.1.1.

This command:

make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=3.1.1

Attempts to install Hail with Scala 2.11.12 and Spark 3.1.1. If you need to use Scala 2.11, you must compile with Spark version 2.4.5. If you need to use Spark version 3.1.1, then you must compile with Scala version 2.12.13.

Moreover, if you are installing Hail on a Spark Cluster, you don’t need to use the “For Software Developers” installation guide. You can use the simpler “Install Hail on a Spark Cluster” installation guide. Assuming you’ve already installed the yum packages for BLAS, LAPACK, Java, lz4-devel, python3, and g++, you can just run this:

make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1

You should not need to upload any files into S3 buckets. For more information see “Install Hail on a Spark Cluster”.

3

I tried the following (exactly the same from the doc) before and it didn’t work. Error logs said used Spark 3.1.1, that’s why I changed it to 3.1.1

make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=2.4.5

4

Can you share the output verbatim? I expect this to print:

WARNING: Hail primarily tested with Spark 3.1.1, use other versions at your own risk.

But that is only a warning. You should still be able to try using Spark 2.4.5.

Are you unable to use Scala 2.12?

5

I got

BUILD FAILED in 2m 52s
1 actionable task: 1 executed
make: *** [build/libs/hail-all-spark.jar] Error 1

when running with make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=2.4.5

There is no hail-python.zip in the destination bucket

6

Usually the BUILD FAILED message is preceded by an explanation of the failure. Was anything else printed? If there is a lot of output you can upload it to pastebin and share the link here.

7

Ooops, missed the important part…

* What went wrong:
Execution failed for task ':compileScala'.
> Failed to run Gradle Worker Daemon
   > Process 'Gradle Worker Daemon 2' finished with non-zero exit value 137
8

The template itself finished just fine but still had error messages and no hail-python.zip available.

9

Hi @yc000000,

I saw your new thread here, but let’s keep it all in one thread.

It’s difficult for us to diagnose what is going wrong without more of the error output. Was there more printed that you could share, possibly using pastebin if it’s too long? If not, you could try running gradle with the --stacktrace option.

Also, is there a reason you’re unable to use Scala 2.12? We test with Scala 2.12 and Spark 3.1.1.

10

hi @patrick-schultz,

Thank you for getting back to me. I simply followed the doc (Hail | For Software Developers) for deployment.

Can I just run ./gradlew directly instead of using make install HAIL_COMPILE_NATIVES=1 ?

I read somewhere to run ./gradlew --no-deamon option but I only used the make command to deploy so it’s confusing how to fix this issue.

11

Ah, okay, good to know you’re not prevented from using Scala 2.12!

We noticed recently that some of our installation docs are out of data. We made some fixes, but they won’t push to the website until our next release. You can see the updated “Install Hail on a Spark Cluster” page on our github here. I would suggest trying the updated instructions, in partitcular

make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1

And you’re right about make making it difficult to pass flags to gradle. It’s pretty easy to modify the makefile to add options to pass to gradle. But try with the right scala and spark versions first.

12

@patrick-schultz ,

would that install-on-cluster works on aws? or just the version?

13

Sorry, yeah, just the version:

make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1
14

@patrick-schultz The versions seemed to work fine! Thank you! But something else is failing…

Where is hail-python.zip for 0.2? We had

          aws s3 cp decorator-4.2.1-py2.py3-none-any.whl s3://${RESOURCES_BUCKET}/artifacts/decorator.zip
          aws s3 cp distributions/hail-python.zip s3://${RESOURCES_BUCKET}/artifacts/
          aws s3 cp libs/hail-all-spark.jar s3://${RESOURCES_BUCKET}/artifacts/

for Hail 0.1 and it worked but not for this version, we are missing hail-python.zip

15

This conversation was continued here: hail-python.zip location · Issue #10844 · hail-is/hail · GitHub . In short, the zip archive has been replaced with a proper Python wheel file.