Potential installation issues with Hail update

Hail Query & hailctl
1

Hello!

I just updated my Hail with pip with no issues (to version 0.2.110), and am having some issues working with code that previously worked a month ago that seem to be related to Hail installation.

Specifically, I’m just trying to pull in the public pan-ukb phenotype mt path -
mt = hl.read_matrix_table(get_ukb_pheno_mt_path())
and get the following error message:

Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
: java.lang.NullPointerException
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:680)
	at is.hail.backend.spark.SparkBackend$.configureAndCreateSparkContext(SparkBackend.scala:149)
	at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:231)
	at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:829)

If useful, I fired up a cluster with hailctl with the following command

$ hailctl dataproc start ega --region us-central1 --requester-pays-allow-buckets ukb-diverse-pops-public,gnomad-public --packages ukbb_common --num-worker-local-ssds=1

Any suggestions for troubleshooting?
Thanks so much!
Elizabeth

2

Update - this does seem to be an issue with the new version, 2.1.110. I just back-graded to 2.1.09 and did not encounter this issue any more.

3

Hmm. We just updated to Spark 3.3.0 and the new dataproc version. This line indicates that something went wrong with Spark initialization. We test against dataproc before releasing, but we don’t test with the ukbb_common package. Does it also fail if you start the cluster without ukbb_common? Does that package install any Spark plugins?

Can you post the full stack trace including the hail version information?