Hi, I apologize if this is a dumb question but I’m a Hail newbie. I am trying to simply run Hail on Amazon EMR. I have setup a small EMR cluster using emr-5.13.0 which has Spark 2.3.0. Following some scripts built by Amazon I logged into the master node and ran the following HAIL_VERSION=“0.1” …

Just made a post about this: [image] ImportError: cannot import name getargspec Help [0.1] The author of the decorator module pushed a new version to PyPI a few days ago which removed functionality from the module without updating the major version. This means that decorat…

however, I don’t think Hail 0.1 is compatible with Spark 2.3.0 anyway. You should use the 0.2 beta version if you’re just getting started!

[image] tpoterba: pip install decorator==4.2.1 Thanks so much, yes I just tried it but am realizing that pyspark on EMR uses python 2.7, however you guys use Python 3.6. Do you know how one would change that?

here’s something that might help: https://medium.com/@datitran/quickstart-pyspark-with-anaconda-on-aws-660252b88c9a

If you use EMR release 5.12.1 you can install python 3.6 as a bootstrap action on the cluster.

Running hail on AWS EMR

Help [0.1]

tpoterba April 17, 2018, 6:00pm 2

Just made a post about this:

Topic		Replies	Views
Hail 0.2 on EMR? Help [0.1]	17	2643	June 18, 2018
Install Hail using Spark Hail Query & hailctl	15	1417	April 13, 2018
Hail 0.2.27 not running on Cloudera spark 2.4.0 Hail Query & hailctl	8	672	November 19, 2019
[Hail on apache spark] Using pyspark, py4j.protocol.Py4JError Hail Query & hailctl	2	536	July 14, 2021
Pip-installed Hail requires additional configuration options in Spark referring to the path to the Hail Python module directory HAIL_DIR Hail Batch & General Cloud	1	675	June 3, 2022

Running hail on AWS EMR

Related topics