# Hail 0.2 on glue **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [June 29, 2021, 9:55pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113 "2021-06-29T21:55:16Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [June 29, 2021, 9:55pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/1 "2021-06-29T21:55:16Z")
I am facing below error when trying to import hail on glue (spark3, python2, glue version 1.0). Any help is much appreciated: 21/06/29 21:47:11 ERROR ApplicationMaster: User application exited with status 1 import sys, os from pyspark import SparkContext, SparkConf from awsglue.transforms import \* from awsglue.utils import getResolvedOptions from awsglue.context import GlueContext from awsglue.job import Job from hail import \* #import hail as hl conf = SparkConf() conf.set(‘spark.app.name’, u’Running Hail on Glue’) conf.set(‘spark.sql.files.maxPartitionBytes’, ‘1099511627776’) conf.set(‘spark.sql.files.openCostInBytes’, ‘1099511627776’) conf.set(‘spark.kryo.registrator’, ‘is.hail.kryo.HailKryoRegistrator’) conf.set(‘spark.serializer’, ‘org.apache.spark.serializer.KryoSerializer’) sc = SparkContext(conf=conf) sc.\_jsc.hadoopConfiguration().set(“mapred.output.committer.class”, “org.apache.hadoop.mapred.FileOutputCommitter”) sc.getConf().getAll() glueContext = GlueContext(sc) spark = glueContext.spark\_session job = Job(glueContext) #job.init(args[‘JOB\_NAME’], args) hc = HailContext(sc) #hl.init(sc) print(“Hello World!!!”) when I change the import statements as shown below I see this error: Py4JError: An error occurred while calling z:is.hail.HailContext.apply. Trace: #from hail import \* import hail as hl #hc = HailContext(sc) hl.init(sc) ---
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [June 29, 2021, 10:03pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/2 "2021-06-29T22:03:26Z")
also want to add that when using python 3 I see this error: ModuleNotFoundError: No module named ‘SocketServer’ ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 29, 2021, 11:07pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/3 "2021-06-29T23:07:08Z")
What version of Hail are you using? There’s no mention of “SocketServer” anywhere in the code base right now. ---
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [June 29, 2021, 11:45pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/4 "2021-06-29T23:45:30Z")
I am using hail 2.0 ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 30, 2021, 12:14am UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/5 "2021-06-30T00:14:01Z")
What’s the full version? `pip show hail`? ---
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [August 2, 2021, 2:16am UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/6 "2021-08-02T02:16:42Z")
I am trying to rebuild (steps below) on an ec2 with amzn linux and seeing this error : FAILURE: Build failed with an exception. - Where: Build file ‘/home/ec2-user/hail/hail/build.gradle’ line: 194 - What went wrong: A problem occurred evaluating root project ‘hail’. > assert(scalaMajorVersion == “2.11”) > | | > ‘2.12’ false sudo yum install -y g++ cmake git sudo yum install -y lz4 sudo yum install -y lz4-devel git clone [GitHub - hail-is/hail: Cloud-native genomic dataframes and batch computing](https://github.com/hail-is/hail.git) cd hail/hail && git fetch && git checkout sudo yum groupinstall ‘Development Tools’ sudo yum install java-1.8.0 sudo alternatives --config java sudo yum search java | grep openjdk sudo yum install java-1.8.0-openjdk-headless.x86\_64 sudo yum install java-1.8.0-openjdk-devel.x86\_64 sudo update-alternatives --config java sudo update-alternatives --config javac make install HAIL\_COMPILE\_NATIVES=1 SPARK\_VERSION=2.4.4 any help please ? I am basically stuck big time getting hail 0.2 working on glue ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [August 2, 2021, 12:33pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/7 "2021-08-02T12:33:50Z")
Try this: ```auto make install HAIL_COMPILE_NATIVES=1 SPARK_VERSION=2.4.4 SCALA_VERSION=2.11.12 ``` ---
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [August 2, 2021, 4:27pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/8 "2021-08-02T16:27:37Z")
> [@tpoterba](#): > > pip show hail Thanks, I was able to build on ec2 (amazo linux). But when I try to use these files on aws glue I see this error: import hail as hl ModuleNotFoundError: No module named ‘hail’ [ec2-user@ip-172-31-50-72 ~]$ pip show hail Name: hail Version: 0.2.74 Summary: Scalable library for exploring and analyzing genomic data. Home-page: [https://hail.is](https://hail.is) Author: Hail Team Author-email: [hail@broadinstitute.org](mailto:hail@broadinstitute.org) License: UNKNOWN Location: /home/ec2-user/.local/lib/python3.7/site-packages Requires: aiohttp, humanize, janus, pandas, asyncinit, decorator, aiohttp-session, pyspark, google-cloud-storage, tabulate, python-json-logger, scipy, bokeh, tqdm, dill, botocore, gcsfs, boto3, PyJWT, nest-asyncio, Deprecated, fsspec, requests, numpy, parsimonious, hurry.filesize ---
**Author:** ![rahulch](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@rahulch](https://discuss.hail.is/u/rahulch)\ **Post date:** [August 3, 2021, 8:26pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/10 "2021-08-03T20:26:57Z")
I got past the previous error but currently stuck at the one below. Please help if you can: File “/tmp/test\_08\_02”, line 8, in import hail as hl File “/tmp/hail-python.zip/hail/ **init**.py”, line 44, in from .table import Table, GroupedTable, asc, desc # noqa: E402 File “/tmp/hail-python.zip/hail/table.py”, line 7, in from hail.expr.expressions import Expression, StructExpression, File “/tmp/hail-python.zip/hail/expr/ **init**.py”, line 1, in from .types import dtype, HailType, hail\_type, is\_container, is\_compound, File “/tmp/hail-python.zip/hail/expr/types.py”, line 10, in from hail import genetics File “/tmp/hail-python.zip/hail/genetics/ **init**.py”, line 1, in from .call import Call File “/tmp/hail-python.zip/hail/genetics/call.py”, line 2, in from hail.utils import FatalError File “/tmp/hail-python.zip/hail/utils/ **init**.py”, line 8, in from .tutorial import get\_1kg, get\_hgdp, get\_movie\_lens File “/tmp/hail-python.zip/hail/utils/tutorial.py”, line 7, in from hailtop.utils import sync\_retry\_transient\_errors File “/tmp/hail-python.zip/hailtop/utils/ **init**.py”, line 2, in from .utils import ( File “/tmp/hail-python.zip/hailtop/utils/utils.py”, line 19, in import google.auth.exceptions ModuleNotFoundError: No module named ‘google’ **pip** install --upgrade --target=/home/ec2-user/hail/hail/python/ google import hail as hl File “/tmp/hail-python.zip/hail/ **init**.py”, line 44, in from .table import Table, GroupedTable, asc, desc # noqa: E402 File “/tmp/hail-python.zip/hail/table.py”, line 7, in from hail.expr.expressions import Expression, StructExpression, File “/tmp/hail-python.zip/hail/expr/ **init**.py”, line 1, in from .types import dtype, HailType, hail\_type, is\_container, is\_compound, File “/tmp/hail-python.zip/hail/expr/types.py”, line 10, in from hail import genetics File “/tmp/hail-python.zip/hail/genetics/ **init**.py”, line 1, in from .call import Call File “/tmp/hail-python.zip/hail/genetics/call.py”, line 2, in from hail.utils import FatalError File “/tmp/hail-python.zip/hail/utils/ **init**.py”, line 8, in from .tutorial import get\_1kg, get\_hgdp, get\_movie\_lens File “/tmp/hail-python.zip/hail/utils/tutorial.py”, line 7, in from hailtop.utils import sync\_retry\_transient\_errors File “/tmp/hail-python.zip/hailtop/utils/ **init**.py”, line 2, in from .utils import ( File “/tmp/hail-python.zip/hailtop/utils/utils.py”, line 19, in import google.auth.exceptions ModuleNotFoundError: No module named ‘google’ **pip** install --upgrade google-auth google-auth-httplib2 google-api-python-client import hail as hl File “/tmp/hail-python.zip/hail/ **init**.py”, line 44, in from .table import Table, GroupedTable, asc, desc # noqa: E402 File “/tmp/hail-python.zip/hail/table.py”, line 7, in from hail.expr.expressions import Expression, StructExpression, File “/tmp/hail-python.zip/hail/expr/ **init**.py”, line 1, in from .types import dtype, HailType, hail\_type, is\_container, is\_compound, File “/tmp/hail-python.zip/hail/expr/types.py”, line 10, in from hail import genetics File “/tmp/hail-python.zip/hail/genetics/ **init**.py”, line 1, in from .call import Call File “/tmp/hail-python.zip/hail/genetics/call.py”, line 2, in from hail.utils import FatalError File “/tmp/hail-python.zip/hail/utils/ **init**.py”, line 8, in from .tutorial import get\_1kg, get\_hgdp, get\_movie\_lens File “/tmp/hail-python.zip/hail/utils/tutorial.py”, line 7, in from hailtop.utils import sync\_retry\_transient\_errors File “/tmp/hail-python.zip/hailtop/utils/ **init**.py”, line 2, in from .utils import ( File “/tmp/hail-python.zip/hailtop/utils/utils.py”, line 19, in import google.auth.exceptions ModuleNotFoundError: No module named 'google ---
**Author:** ![a-li](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/a-li/32/929_2.png) [@a-li](https://discuss.hail.is/u/a-li)\ **Post date:** [May 2, 2023, 11:22pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/11 "2023-05-02T23:22:04Z")
Just bumping up this thread as I’m trying to achieve the same thing. Is there a recommended way to run Hail in AWS Glue? I tried the following - - I pulled down the main branch from Github and ran the ./gradlew to bundle up the JAR. Added “hail-all-spark.jar” in “Python library path” → This gave me a “ModuleNotFoundError: No module named ‘hail’” error - “–additional-python-modules” with “hail==0.2.78” but this complained about “ImportError: cannot import name ‘Markup’ from ‘jinja2’ (/home/spark/.local/lib/python3.7/site-packages/jinja2/ **init**.py)” so I tried to add “jinja2==3.0.3” but then it gave me “TypeError: ‘JavaPackage’ object is not callable” error. What is the recommended approach to add Hail library to AWS Glue? Similarly for EMR? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 3, 2023, 1:28pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/12 "2023-05-03T13:28:07Z")
Hmm. There was an AWS EMR supported solution but it appears AWS has dropped support: [Partner Solution not available—Amazon Web Services (AWS)](https://aws.amazon.com/quickstart/architecture/hail/) ---
**Author:** ![a-li](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/a-li/32/929_2.png) [@a-li](https://discuss.hail.is/u/a-li)\ **Post date:** [May 3, 2023, 6:23pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/13 "2023-05-03T18:23:34Z")
Do you happen to know if AWS Glue with Hail was ever supported? From documentation, it seems like Glue supports adding Python libraries through using a JAR > > Only pure Python libraries can be used. Libraries that rely on C extensions, such as `pandas` (Python data analysis) library, are not yet supported. Ref: See #6 in [Providing your own custom scripts - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/console-custom-created.html) However, I don’t think this would work because Hail is exposed as a Python library but it’s built in Scala, Spark and C++. So it’s not a pure Python library. The other alternative way I found is to include the python modules through pip install with --additional-python-modules (ref: [Using Python libraries with AWS Glue - AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html)). Is there any way to find a complete list of dependency libraries that Hail requires? Perhaps I can try including all the libraries needed to see if the installation will be successful. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 3, 2023, 6:27pm UTC](https://discuss.hail.is/t/hail-0-2-on-glue/2113/14 "2023-05-03T18:27:55Z")
We never supported AWS Glue. I don’t know if Amazon / AWS Glue ever supported Hail. You’re correct,. Hail is not a pure Python library. It needs a lot of special configuration. We describe how to install Hail on an arbitrary Spark cluster [here](https://hail.is/docs/0.2/install/other-cluster.html).