# Unable to deploy/compile latest Hail on aws **URL:** \ **Category:** Development\ **Created:** [August 31, 2021, 1:04pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216 "2021-08-31T13:04:18Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [August 31, 2021, 1:04pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/1 "2021-08-31T13:04:18Z")
Hi, I am trying to update Hail from 0.1 to the latest version but kept failing. Here is what I did (from this site: [Hail | For Software Developers](https://hail.is/docs/0.2/getting_started_developing.html)) ```auto echo 'Installing pre-reqs' yum install -y g++ cmake git yum install -y lz4 yum install -y lz4-devel git clone $HAIL_REPO cd hail/hail && git fetch && git checkout main ./gradlew clean make install HAIL_COMPILE_NATIVES=1 make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=3.1.1 cd build pip download decorator==4.2.1 aws s3 cp decorator-4.2.1-py2.py3-none-any.whl s3://${RESOURCES_BUCKET}/artifacts/decorator.zip aws s3 cp distributions/hail-python.zip s3://${RESOURCES_BUCKET}/artifacts/ aws s3 cp libs/hail-all-spark.jar s3://${RESOURCES_BUCKET}/artifacts/ ``` ```auto And this is what I got > Task :compileScala FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':compileScala'. > javac returned non-zero exit code ``` ```auto make: *** [build/libs/hail-all-spark.jar] Error 1 ``` ```auto A problem occurred evaluating root project 'hail'. 286 > Could not resolve all dependencies for configuration ':justSpark'. 287 > Could not find org.apache.spark:spark-mllib_2.11:3.1.1. ``` Following by other errors too, but I am assuming it could be caused by previous errors. How should I change my template? Thank you! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [August 31, 2021, 4:45pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/2 "2021-08-31T16:45:27Z")
Hi @yc000000 ! I’m sorry you’re having trouble installing Hail! The key issue is that Spark 3.1.1 is not compatible with Scala 2.11. That’s why you get this error: ```auto 287 > Could not find org.apache.spark:spark-mllib_2.11:3.1.1. ``` * * * This command: ```auto make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=3.1.1 ``` Attempts to install Hail with Scala 2.11.12 and Spark 3.1.1. If you need to use Scala 2.11, you must compile with Spark version 2.4.5. If you need to use Spark version 3.1.1, then you must compile with Scala version 2.12.13. * * * Moreover, if you are installing Hail on a Spark Cluster, you don’t need to use the “For Software Developers” installation guide. You can use the simpler [“Install Hail on a Spark Cluster”](https://hail.is/docs/0.2/install/other-cluster.html) installation guide. Assuming you’ve already installed the yum packages for BLAS, LAPACK, Java, lz4-devel, python3, and g++, you can just run this: ```auto make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1 ``` You should not need to upload any files into S3 buckets. For more information see [“Install Hail on a Spark Cluster”](https://hail.is/docs/0.2/install/other-cluster.html). ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [August 31, 2021, 4:49pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/3 "2021-08-31T16:49:53Z")
I tried the following (exactly the same from the doc) before and it didn’t work. Error logs said used Spark 3.1.1, that’s why I changed it to 3.1.1 `make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=2.4.5` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [August 31, 2021, 5:21pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/4 "2021-08-31T17:21:49Z")
Can you share the output verbatim? I expect this to print: ```auto WARNING: Hail primarily tested with Spark 3.1.1, use other versions at your own risk. ``` But that is only a warning. You should still be able to try using Spark 2.4.5. Are you unable to use Scala 2.12? ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [August 31, 2021, 6:02pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/5 "2021-08-31T18:02:09Z")
I got ```auto BUILD FAILED in 2m 52s 1 actionable task: 1 executed make: *** [build/libs/hail-all-spark.jar] Error 1 ``` when running with `make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=2.4.5` There is no hail-python.zip in the destination bucket ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [August 31, 2021, 6:21pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/6 "2021-08-31T18:21:01Z")
Usually the `BUILD FAILED` message is preceded by an explanation of the failure. Was anything else printed? If there is a lot of output you can upload it to [pastebin](https://pastebin.com) and share the link here. ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [August 31, 2021, 6:25pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/7 "2021-08-31T18:25:46Z")
Ooops, missed the important part… ```auto * What went wrong: Execution failed for task ':compileScala'. > Failed to run Gradle Worker Daemon > Process 'Gradle Worker Daemon 2' finished with non-zero exit value 137 ``` ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [September 1, 2021, 2:55pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/8 "2021-09-01T14:55:55Z")
The template itself finished just fine but still had error messages and no hail-python.zip available. ---
**Author:** ![patrick-schultz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/patrick-schultz/32/265_2.png) [@patrick-schultz](https://discuss.hail.is/u/patrick-schultz)\ **Post date:** [September 2, 2021, 5:26pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/9 "2021-09-02T17:26:33Z")
Hi @yc000000, I saw your new thread [here](https://discuss.hail.is/t/failed-to-use-hail-on-cloud-aws/2226), but let’s keep it all in one thread. It’s difficult for us to diagnose what is going wrong without more of the error output. Was there more printed that you could share, possibly using pastebin if it’s too long? If not, you could try running gradle with the `--stacktrace` option. Also, is there a reason you’re unable to use Scala 2.12? We test with Scala 2.12 and Spark 3.1.1. ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [September 2, 2021, 5:47pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/10 "2021-09-02T17:47:00Z")
hi @patrick-schultz, Thank you for getting back to me. I simply followed the doc ([Hail | For Software Developers](https://hail.is/docs/0.2/getting_started_developing.html)) for deployment. Can I just run ./gradlew directly instead of using `make install HAIL_COMPILE_NATIVES=1` ? I read somewhere to run `./gradlew --no-deamon` option but I only used the make command to deploy so it’s confusing how to fix this issue. ---
**Author:** ![patrick-schultz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/patrick-schultz/32/265_2.png) [@patrick-schultz](https://discuss.hail.is/u/patrick-schultz)\ **Post date:** [September 2, 2021, 6:07pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/11 "2021-09-02T18:07:48Z")
Ah, okay, good to know you’re not prevented from using Scala 2.12! We noticed recently that some of our installation docs are out of data. We made some fixes, but they won’t push to the website until our next release. You can see the updated “Install Hail on a Spark Cluster” page on our github [here](https://github.com/hail-is/hail/blob/main/hail/python/hail/docs/install/other-cluster.rst). I would suggest trying the updated instructions, in partitcular ```auto make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1 ``` And you’re right about `make` making it difficult to pass flags to gradle. It’s pretty easy to modify the makefile to add options to pass to gradle. But try with the right scala and spark versions first. ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [September 2, 2021, 6:22pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/12 "2021-09-02T18:22:01Z")
@patrick-schultz , would that `install-on-cluster` works on aws? or just the version? ---
**Author:** ![patrick-schultz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/patrick-schultz/32/265_2.png) [@patrick-schultz](https://discuss.hail.is/u/patrick-schultz)\ **Post date:** [September 2, 2021, 6:38pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/13 "2021-09-02T18:38:55Z")
Sorry, yeah, just the version: ```auto make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1 ``` ---
**Author:** ![yc000000](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/yc000000/32/640_2.png) [@yc000000](https://discuss.hail.is/u/yc000000)\ **Post date:** [September 3, 2021, 1:50pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/14 "2021-09-03T13:50:18Z")
> [@patrick-schultz](#): > > `make install HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.13 SPARK_VERSION=3.1.1` @patrick-schultz The versions seemed to work fine! Thank you! But something else is failing… Where is hail-python.zip for 0.2? We had ```python aws s3 cp decorator-4.2.1-py2.py3-none-any.whl s3://${RESOURCES_BUCKET}/artifacts/decorator.zip aws s3 cp distributions/hail-python.zip s3://${RESOURCES_BUCKET}/artifacts/ aws s3 cp libs/hail-all-spark.jar s3://${RESOURCES_BUCKET}/artifacts/ ``` for Hail 0.1 and it worked but not for this version, we are missing hail-python.zip ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [September 10, 2021, 8:41pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216/15 "2021-09-10T20:41:06Z")
This conversation was continued here: [hail-python.zip location · Issue #10844 · hail-is/hail · GitHub](https://github.com/hail-is/hail/issues/10844) . In short, the zip archive has been replaced with a proper Python wheel file.