Error in calling SparkBackend

Hello,
I’ve recently installed hail in Mac and tried to use it.
But when I run :

import hail as hl
hl.init() 

I have an error message

Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
: java.lang.IllegalArgumentException: requirement failed
	at scala.Predef$.require(Predef.scala:212)
	at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:191)
	at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:830)

I’ve already checked my python and java version, and I also have spark in recent version.

could you provide the Python stack trace as well, and the version you’re running?

Dear tpoterba,
This is the whole error message including traceback.

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-5-aca8dac9a9a8> in <module>
  1 import hail as hl
----> 2 hl.init(default_reference='GRCh38')

</Users/dahaelee/anaconda3/lib/python3.7/site-packages/decorator.py:decorator-gen-1751> in init(sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, spark_conf, skip_logging_configuration, local_tmpdir, _optimizer_iterations)

~/anaconda3/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
612     def wrapper(__original_func, *args, **kwargs):
613         args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
--> 614         return __original_func(*args_, **kwargs_)
615 
616     return wrapper

~/anaconda3/lib/python3.7/site-packages/hail/context.py in init(sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, spark_conf, skip_logging_configuration, local_tmpdir, _optimizer_iterations)
229         idempotent, sc, spark_conf, app_name, master, local, log,
230         quiet, append, min_block_size, branching_factor, tmpdir, local_tmpdir,
--> 231         skip_logging_configuration, optimizer_iterations)
232 
233     HailContext(

~/anaconda3/lib/python3.7/site-packages/hail/backend/spark_backend.py in __init__(self, idempotent, sc, spark_conf, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmpdir, local_tmpdir, skip_logging_configuration, optimizer_iterations)
191         else:
192             self._jbackend = hail_package.backend.spark.SparkBackend.apply(
--> 193                 jsc, app_name, master, local, True, min_block_size, tmpdir, local_tmpdir)
194             self._jhc = hail_package.HailContext.apply(
195                 self._jbackend, log, True, append, branching_factor, skip_logging_configuration, optimizer_iterations)

~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

~/anaconda3/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
326                 raise Py4JJavaError(
327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
329             else:
330                 raise Py4JError(
Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
: java.lang.IllegalArgumentException: requirement failed
	at scala.Predef$.require(Predef.scala:212)
	at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:191)
	at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:830)

And this is my python version checked in terminal.

$ python --version

Python 3.7.6

This is super weird, never seen this! Did you install Hail using pip install hail?

And what version are you running? (pip show hail)

Dear tpoterba,
I’ve installed hail using pip install hail.
Below is my version of hail.

    $ pip show hail
    Name: hail
    Version: 0.2.54
    Summary: Scalable library for exploring and analyzing genomic data.
    Home-page: https://hail.is
    Author: Hail Team
    Author-email: hail@broadinstitute.org
    License: UNKNOWN
    Location: /Users/dahaelee/anaconda3/lib/python3.7/site-packages
    Requires: hurry.filesize, asyncinit, bokeh, scipy, decorator, requests, pandas, tqdm, numpy, dill, humanize, parsimonious, PyJWT, aiohttp-session, python-json-logger, nest-asyncio, aiohttp, pyspark, gcsfs, tabulate, google-cloud-storage
    Required-by: