Hail-overview:error in running the first two lines in the tutorial

Help [0.1]
1

from hail import *
hc = HailContext()

ImportError Traceback (most recent call last)
in ()
----> 1 from hail import *
2 hc = HailContext()

/Users/AleRodriguez/bin/hail/python/hail/init.py in ()
----> 1 import hail.expr
2 from hail.representation import *
3 from hail.context import HailContext
4 from hail.dataset import VariantDataset
5 from hail.expr import *

/Users/AleRodriguez/bin/hail/python/hail/expr.py in ()
1 import abc
----> 2 from hail.java import scala_object, Env, jset
3 from hail.representation import Variant, AltAllele, Genotype, Locus, Interval, Struct, Call
4
5

/Users/AleRodriguez/bin/hail/python/hail/java.py in ()
----> 1 import SocketServer
2 import socket
3 import sys
4 from threading import Thread
5

ImportError: No module named ‘SocketServer’

2

Are you possibly using Python 3? Hail currently only supports Python 2, and it looks like SocketServer is one module that was renamed in Python 3.

3

I changed my default to python 2 and got the same error

4

Which Python 2? Maybe SocketServer was a late addition.

5

Python 2.7.10

I was originally running python 3 and if I type python -V on my terminal I am running Python 2.7.10
However i am Not sure if this is also happening in the Jupyter note book.
This is how I was able to set python 2 as default in my computer:
set python 2 as default to my computer*********
Use this to create your temporary python “install”

(Assuming that is the correct path to the python interpreter you want to use.)

virtualenv -p /usr/bin/python2.7 --distribute temp-python

Type this command when you want to use your temporary python.

While you are using your temporary python you will also have access to a temporary pip,

which will keep all packages installed with it separate from your main python install.

A shorter version of this command would be “. temp-python/bin/activate”

source temp-python/bin/activate

When you no longer wish to use you temporary python type

deactivate

Thanks for your help

6

inside the jupyter notebook, can you give us the output of:

import sys
print(sys.version_info)
7

sys.version_info(major=3, minor=5, micro=2, releaselevel=‘final’, serial=0)

8

Yep, your notebook still has python 3. The below link has instructions for now to set up multiple ipython kernels for jupyter, which will let you make a python 2 version for hail and python 3 version for other projects (if necessary)

9

I fixed that :
sys.version_info(major=2, minor=7, micro=14, releaselevel=‘final’, serial=0)

but there is still a different error:

ImportError Traceback (most recent call last)
in ()
----> 1 from hail import *
2 hc = HailContext()

/Users/AleRodriguez/bin/hail/python/hail/init.py in ()
----> 1 import hail.expr
2 from hail.representation import *
3 from hail.context import HailContext
4 from hail.dataset import VariantDataset
5 from hail.expr import *

/Users/AleRodriguez/bin/hail/python/hail/expr.py in ()
1 import abc
----> 2 from hail.java import scala_object, Env, jset
3 from hail.representation import Variant, AltAllele, Genotype, Locus, Interval, Struct, Call
4
5

/Users/AleRodriguez/bin/hail/python/hail/java.py in ()
4 from threading import Thread
5
----> 6 import py4j
7 from decorator import decorator
8

ImportError: No module named py4j

10

I did used the following to build kernels:
conda create -n py27 python=2.7 ipykernel
conda create -n py36 python=3.6 ipykernel

11

OK, this is good, we’ve gotten to a new error!

How did you install Hail? Are you using the distribution? If so, then you should be able to follow the instructions on the getting started page to use the jhail script.

12

I installed Hail by following the instructions on the page you are suggesting, to make sure I also went back and did this again. There is a serious problem with this instructions because I noticed that many people are having trouble with the installations and somehow their always instructed to go back to the getting started page.

First
Here, fill in the path to the unzipped Hail distribution.
#this is the location where Hail-0.1-0ab38b4c77e7-Spark-2.0.2.zip is located.
also where the unzipped folder is located hail so I have edited these lines to include the path to these two:
export HAIL_HOME=/Users/AleRodriguez/
I am not sure if the instruction is to keep this line intact but that is what I did:
export PATH=$PATH:$HAIL_HOME/bin/

13

also
java -version
java version "9.0.1"
Java™ SE Runtime Environment (build 9.0.1+11)
Java HotSpot™ 64-Bit Server VM (build 9.0.1+11, mixed mode)

14

Thanks for the feedback, it’s good to know what’s not clear! I’ll update the doc to be more specific about using the path to the unzipped distribution folder, not the directory containing it.

I think for you the first line should be:

export HAIL_HOME=/Users/AleRodriguez/Hail-0.1-0ab38b4c77e7-Spark-2.0.2

I also just saw your latest post about your java version. Hail isn’t tested or built against java 9, so you’ll need to install java 8. Sorry for the strict dependencies, we’re working on making things more flexible!

15

Here is what I have
I both typed these into the command line and also edited my ~/.bash_profile (added all 3 lines)

export SPARK_HOME=/Users/AleRodriguez/spark-2.2.0-bin-hadoop2.7.tgz
export HAIL_HOME=/Users/AleRodriguez/Hail-0.1-0ab38b4c77e7-Spark-2.0.2.zip
export PATH=$PATH:$HAIL_HOME/bin/

I have also set Java 8 as default:
java version "1.8.0_151"
Java™ SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot™ 64-Bit Server VM (build 25.151-b12, mixed mode)

still not working very frustrating

16

Before anything else, I notice that you have:

/Users/AleRodriguez/spark-2.2.0-bin-hadoop2.7.tgz

But the getting started page notes:

  • Spark 2.0.2. Hail should work with other versions of Spark 2, see below.

followed by:

Hail uploads distributions to Google Storage as part of our continuous
integration suite. You can download a pre-built distribution from the below
links. Make sure you download the distribution that matches your Spark
version!

But it looks like the distribution you downloaded is

/Users/AleRodriguez/Hail-0.1-0ab38b4c77e7-Spark-2.0.2.zip

Note the Spark-2.0.2. Your hail distribution must be compatible with your
Spark version, as noted above. Try downloading Spark 2.0.2 instead:

Now, let’s take this from the top:

# cd /Users/AleRodriguez
# ls

I assume this prints a list of files including: spark-2.0.2-bin-hadoop2.7.tgz
and Hail-0.1-0ab38b4c77e7-Spark-2.0.2.zip.

Unzip the distribution after you download it.

# unzip Hail-0.1-0ab38b4c77e7-Spark-2.0.2.zip
# ls

Now there should be a folder /Users/AleRodriguez/hail that contains five
sub-folders:

# ls /Users/AleRodriguez/hail
bin       docs      jars      python    tutorials

Next, edit and copy the below bash commands to set up the Hail environment
variables. You may want to add these to your bash dot-file (~/.bash_profile,
~/.bashrc, etc) so that you don’t need to rerun these commands in each new
session.

Here, fill in the path to the un-tarred Spark package.

export SPARK_HOME=???

At this point, if you have not yet un-tarred the Spark package, do that:

# pwd
/Users/AleRodriguez
# tar -xvf spark-2.0.2-bin-hadoop2.7.tgz

You can read more about tar at it’s man page:

man tar

Back to the issue of editing ~/.bashrc, let’s fill in the “path to the
un-tarred Spark package”:

export SPARK_HOME=/Users/AleRodriguez/spark-2.0.2-bin-hadoop2.7

Here, fill in the path to the unzipped Hail distribution.

export HAIL_HOME=???
export PATH=$PATH:$HAIL_HOME/bin/

Let’s enter the unzipped Hail distribution:

export HAIL_HOME=/Users/AleRodriguez/hail
export PATH=$PATH:$HAIL_HOME/bin/

Now let’s save ~/.bashrc. Back at the terminal, we need to load the changes we
just made:

# source ~/.bashrc

Now let’s get hailing:

# ihail
17

great this works, I was not loading the changes using source ~/.bashrc
and yes my spark version was not compatible with hail
Thank you very much!!!