So I’m on CDH 5.10 which has Spark 2.0.0 as a side parcel install, and I’m simply walking through the “Getting Started” tutorial here: https://hail.is/hail/getting_started.html
Everything below is fine, but I hit an error in the python/hail shell when using the HailContext. Any ideas about what may be going on ?
$ ls $HAIL_HOME
acknowledgements.txt build changes.md derby.log gradle gradlew.bat python settings.gradle style-guide.md www
AUTHORS build.gradle code_style.xml docs gradlew LICENSE README.md src testng.xml
$ ls $SPARK_HOME
bin etc lib meta
$ alias hail="PYTHONPATH=$SPARK_HOME/lib/spark2/python:$SPARK_HOME/lib/spark2/python/lib/py4j-0.10.3-src.zip:$HAIL_HOME/python SPARK_CLASSPATH=$HAIL_HOME/build/libs/hail-all-spark.jar python"
$ hail
Python 2.7.5 (default, Sep 15 2016, 22:37:39)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hail
>>> hc = hail.HailContext()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/keebler/hail/python/hail/context.py", line 37, in __init__
SparkContext._ensure_initialized()
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/context.py", line 251, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/java_gateway.py", line 85, in launch_gateway
proc = Popen(command, stdin=PIPE, preexec_fn=preexec_func, env=env)
File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory