EnvironmentError: no Hail context initialized, create one first

Help [0.1]
1

Trying to run Hail locally, and followed the instruction in “Get Started” documentation at https://hail.is/hail/getting_started.html. Unfortunately I got the following EnvironmentError. Any comment would be greatly appreciated.

from hail import *
hc = HailContext()

Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

EnvironmentError Traceback (most recent call last)
in ()
----> 1 hc = HailContext()

in init(self, sc, app_name, master, local, log, quiet, append, parquet_compression, min_block_size, branching_factor, tmp_dir)

/Users/kbchoi/Projects/hail/python/hail/java.pyc in handle_py4j(func, *args, **kwargs)
110 raise FatalError(’%s\n\nJava stack trace:\n%s\n’
111 ‘Hail version: %s\n’
–> 112 ‘Error summary: %s’ % (deepest, full, Env.hc().version, deepest))
113 except py4j.protocol.Py4JError as e:
114 if e.args[0].startswith(‘An error occurred while calling’):

/Users/kbchoi/Projects/hail/python/hail/java.pyc in hc()
40 def hc():
41 if not Env._hc:
—> 42 raise EnvironmentError(‘no Hail context initialized, create one first’)
43 return Env._hc
44

EnvironmentError: no Hail context initialized, create one first

2

Hah! This is a terrible error message. The real error was something else, but it was hidden when we tried to get the version out.

Will fix this ASAP and post the pull request number here so that you can track it.

3

here’s the patch: https://github.com/hail-is/hail/pull/1849

I’ll merge it as soon as the tests pass. Sorry about this!

4

The fix is live, if you pull the latest master it should work.

And by ‘work’, I mean give you a better error message that we can actually work with :slight_smile:

5

Thanks. Now it says I have a conflict in spark versions. It is a bit weird because I have only one spark installation.

$ spark-shell
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel).
17/05/18 16:24:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
17/05/18 16:24:54 WARN SparkConf:
SPARK_CLASSPATH was detected (set to ‘/home/kbchoi/hail//build/libs/hail-all-spark.jar’).
This is deprecated in Spark 1.0+.

Please instead use:

  • ./spark-submit with --driver-class-path to augment the driver classpath
  • spark.executor.extraClassPath to augment the executor classpath

17/05/18 16:24:54 WARN SparkConf: Setting ‘spark.executor.extraClassPath’ to ‘/home/kbchoi/hail//build/libs/hail-all-spark.jar’ as a work-around.
17/05/18 16:24:54 WARN SparkConf: Setting ‘spark.driver.extraClassPath’ to ‘/home/kbchoi/hail//build/libs/hail-all-spark.jar’ as a work-around.
17/05/18 16:24:55 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
Spark context Web UI available at http://x.x.x.x:4040
Spark context available as ‘sc’ (master = local[*], app id = local-1495139095129).
Spark session available as ‘spark’.
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
// .__/_,// //_\ version 2.0.0
//

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.

scala> :quit

Whereas in ipython,

In [1]: from hail import *
In [2]: hc = HailContext()
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel).

Py4JJavaError Traceback (most recent call last)
in ()
----> 1 hc = HailContext()

in init(self, sc, app_name, master, local, log, quiet, append, parquet_compression, min_block_size, branching_factor, tmp_dir)

/home/kbchoi/hail/build/distributions/hail-python.zip/hail/typecheck/check.pyc in _typecheck(f, *args, **kwargs)
190 def _typecheck(f, *args, **kwargs):
191 check_all(f, args, kwargs, checkers, is_method=True)
–> 192 return f(*args, **kwargs)
193
194 return decorator(_typecheck)

/home/kbchoi/hail/build/distributions/hail-python.zip/hail/context.pyc in init(self, sc, app_name, master, local, log, quiet, append, parquet_compression, min_block_size, branching_factor, tmp_dir)
81 self._jhc = scala_object(self._hail, ‘HailContext’).apply(
82 jsc, app_name, joption(master), local, log, quiet, append,
—> 83 parquet_compression, min_block_size, branching_factor, tmp_dir)
84
85 self._jsc = self._jhc.sc()

/usr/local/spark/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py in call(self, *args)
931 answer = self.gateway_client.send_command(command)
932 return_value = get_return_value(
–> 933 answer, self.gateway_client, self.target_id, self.name)
934
935 for temp_arg in temp_args:

/usr/local/spark/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
310 raise Py4JJavaError(
311 “An error occurred while calling {0}{1}{2}.\n”.
–> 312 format(target_id, “.”, name), value)
313 else:
314 raise Py4JError(

Py4JJavaError: An error occurred while calling o0.apply.
: java.lang.IllegalArgumentException: requirement failed: This Hail JAR was compiled for Spark 2.0.2,
but the version of Spark available at runtime is 2.0.0.
at scala.Predef$.require(Predef.scala:224)
at is.hail.HailContext$.configureAndCreateSparkContext(HailContext.scala:33)
at is.hail.HailContext$.apply(HailContext.scala:163)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:748)

6

We’ve instituted a check at HailContext creation to make sure that the version of Spark that Hail was compiled for matches the version it’s running against – in most cases it would be okay to have minor version differences (2.0.2 vs 2.0.0) but we’ve had issues before.

I assume that you compiled this yourself with ./gradlew shadowJar, right? You can add a Spark version like so:

./gradlew -Dspark.version=2.0.0 shadowJar
7

Works great now. Thank you for your help.

8

Sure, and sorry about that first error!