Sorry, my bad. That should be spark_conf={... not conf={....
Hi Dan,
Same issue with spark_conf:
In [2]: hl.init(spark_conf={"spark.driver.bindAddress", "127.0.0.1"})
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-78c62c840b08> in <module>
----> 1 hl.init(spark_conf={"spark.driver.bindAddress", "127.0.0.1"})
TypeError: init() got an unexpected keyword argument 'spark_conf'
E
What version of Hail do you have? This has been present since Hail 0.2.31.
pip3 show hail
Ahh I have 0.2.14. Though I installed it only the other day so not sure why I have an old version?!
Did you install through pip? You need to figure out how to get newer version. Do
pip install -U hail
(or pip3 if that’s what you normally use)
and then pip show hail (or pip3) again to make sure it worked.
It’s possible there’s some environment mix ups as well. Are you using conda/Anaconda/minconda?
I’m using conda.
So I reinstalled hail 0.2.43 and followed your instructions from scratch. Please see output from stages 4-6 (where warnings first appeared).
(hailenv) wm06b-5c7:~ eseaby$ python3 -c 'from pyspark import SparkContext, SparkConf
> sc = SparkContext()'
2020-06-05 19:05:52 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2020-06-05 19:05:55 WARN Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2020-06-05 19:05:55 WARN Utils:66 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
2020-06-05 19:05:55 WARN Utils:66 - Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
(hailenv) wm06b-5c7:~ eseaby$ sudo /bin/sh -c 'echo "127.0.0.1\t$(hostname)" >>/etc/hosts'
Password:
(hailenv) wm06b-5c7:~ eseaby$ python3 -c 'from pyspark import SparkContext, SparkConf
> sc = SparkContext()'
2020-06-05 19:06:17 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2020-06-05 19:06:18 WARN Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2020-06-05 19:06:18 WARN Utils:66 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
2020-06-05 19:06:18 WARN Utils:66 - Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
(hailenv) wm06b-5c7:~ eseaby$ python3
Python 3.7.7 (default, May 6 2020, 04:59:01)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hail as hl
mt = hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
mt.show()
>>> mt = hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
Initializing Hail with default parameters...
2020-06-05 19:07:13 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2020-06-05 19:07:14 WARN Hail:37 - This Hail JAR was compiled for Spark 2.4.5, running with Spark 2.4.0.
Compatibility is not guaranteed.
2020-06-05 19:07:14 WARN Utils:66 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2020-06-05 19:07:14 WARN Utils:66 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
2020-06-05 19:07:14 WARN Utils:66 - Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
Running on Apache Spark version 2.4.0
SparkUI available at http://localhost:4043
Welcome to
__ __ <>__
/ /_/ /__ __/ /
/ __ / _ `/ / /
/_/ /_/\_,_/_/_/ version 0.2.43-d7da1fd14f3c
LOGGING: writing to /Users/eseaby/hail-20200605-1907-0.2.43-d7da1fd14f3c.log
2020-06-05 19:07:15 Hail: INFO: balding_nichols_model: generating genotypes for 3 populations, 50 samples, and 100 variants...
I then skipped 7 as I didn’t get the same binding address error as before. I then installed jupyter and the tutorials are working!
However, I opened a new terminal window and tried to run the tutorials without your instructions (since I’d like to avoid having to install everything everytime I want to run hail) and sadly the notebook tutorials don’t work 
- there is just no output. Not sure if I’m doing something stupid… and I need to activate something before running jupyter notebook tutorials?
Thanks again for all your help…
Ahh wait, I’m an idiot! I forgot to activate the hail environment! Now it works!
Seems like the whole issue was simply outdated version of hail!!
1 Like