py4j.protocol.Py4JNetworkError: Answer from Java side is empty

Warm greetings to Hail community.

gwas = hl.linear_regression_rows(y=mt.pheno.CaffeineConsumption,
x=mt.GT.n_alt_alleles(),
covariates=[1.0])

While I was running the above command in jupyter notebook in centos 7 , I got the following error:-

**019-04-29 11:38:35 Hail: INFO: linear_regression_rows: running on 250 samples for 1 response variable y,
with input variable x, and 1 additional covariate…
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File “/home/aby/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1159, in send_command
raise Py4JNetworkError(“Answer from Java side is empty”)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/aby/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 985, in send_command
response = connection.send_command(command)
File “/home/aby/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1164, in send_command
“Error while receiving”, e, proto.ERROR_ON_RECEIVE)
4j.protocol.Py4JNetworkError: Error while receiving

Initially I thought this might be because of Java home is not available, but I checked and set my JAVA_HOME variable , still I am getting the same error.

Below is my JAVA_HOME variable’s Output.

**[root@localhost ~]# echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/
**

Kindly help me in resolving the issue.

what is the full version (with git SHA) of Hail you’re using?

import hail
import hail as hl
hl.init()
19/05/01 10:33:12 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 10.149.240.149 instead (on interface wlp8s0)
19/05/01 10:33:12 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/05/01 10:33:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Running on Apache Spark version 2.4.1
SparkUI available at http://10.149.240.149:4040
Welcome to
__ __ <>__
/ // /__ __/ /
/ __ / _ `/ / /
/
/ //_,/// version 0.2.12-2571917c39c6
LOGGING: writing to /root/hail-20190501-1033-0.2.12-2571917c39c6.log

The hail version is 0.2.12-2571917c39c6.

Is there any java stack trace?

I am attaching the backend terminal output of jupyter notebook apart from this I didnt find any stack trace for java.

[I 14:26:23.396 NotebookApp] 302 GET /?token=20c3fe169bcd06b8b53b7da8a5f35092f8c03297a929411b (::1) 0.74ms
[I 14:26:30.069 NotebookApp] Kernel started: d75dedfd-1718-44d9-be5c-4cde788f528e
[I 14:26:32.806 NotebookApp] Adapting to protocol v5.1 for kernel d75dedfd-1718-44d9-be5c-4cde788f528e
19/05/03 14:26:44 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 10.149.240.149 instead (on interface wlp8s0)
19/05/03 14:26:44 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
19/05/03 14:26:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
[Stage 32:> (0 + 2) / 2]/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre//bin/java: symbol lookup error: /tmp/jniloader8933272701730098438netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dgemm
[I 14:28:29.992 NotebookApp] Saving file at /gwas.ipynb

I think you need to install openblas

This error can occur on linux systems for which Hail doesn’t precomile native libraries, like centos. The solution is to run:

./gradlew releaseJar instead of ./gradlew shadowJar to build the jar – this will compile the native libraries as well.

Note that the mechanisms for building Hail are subject to change.

Hello tpoterba,

I tried using ./gradlew releaseJar instead of ./gradlew shadowJar and got the following error:-

g++ -o build/ibs.o -march=sandybridge -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I…/resources/include -I/usr/lib/jvm/java/include -I/usr/lib/jvm/java/include/linux -MD -MF build/ibs.d -MT build/ibs.o -c ibs.cpp
In file included from ibs.cpp:5:0:
ibs.h:17:10: fatal error: simdpp/simd.h: No such file or directory
*** #include “simdpp/simd.h”***
*** ^~~~~~~~~~~~~~~***
compilation terminated.
make: *** [build/ibs.o] Error 1

FAILURE: Build failed with an exception.

**** What went wrong:***
Execution failed for task ‘:nativeLibPrebuilt’.
> Process ‘command ‘make’’ finished with non-zero exit value 2

One more point is that I have succesfully installed hail in my machine without compiling it from scratch, still I am facing py4j.protocol.py4jNetworkError issue when trying to run linear regression model.

So if you can point out how to solve this python and java connectivity issue, that will be really helpful.

Thank you

Regards
aby

The build system should addressed this for you, but you’re missing libsimdpp, try:

cd src/main/c
make libsimdpp-2.1

Then retry ./gradlew releaseJar

Can you share the full make log? I want to understand why the build system didn’t automatically download libsimdpp.

With respect to the PY4J issue, please share the full error message and the hail log file.

Dear danking,

I am attaching the complete py4j error:-

2019-07-12 12:16:01 Hail: INFO: linear_regression_rows: running on 284 samples for 1 response variable y,
with input variable x, and 1 additional covariate…
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/py4j/java_gateway.py”, line 1159, in send_command
raise Py4JNetworkError(“Answer from Java side is empty”)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.6/site-packages/py4j/java_gateway.py”, line 985, in send_command
response = connection.send_command(command)
File “/usr/lib/python3.6/site-packages/py4j/java_gateway.py”, line 1164, in send_command
“Error while receiving”, e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving

Py4JError Traceback (most recent call last)
in
1 gwas = hl.linear_regression_rows(y=mt.pheno.CaffeineConsumption,
2 x=mt.GT.n_alt_alleles(),
----> 3 covariates=[1.0])

</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-1379> in linear_regression_rows(y, x, covariates, block_size, pass_through)

/usr/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
559 def wrapper(original_func, *args, **kwargs):
560 args
, kwargs
= check_all(__original_func, args, kwargs, checkers, is_method=is_method)
→ 561 return original_func(*args, **kwargs)
562
563 return wrapper

/usr/lib/python3.6/site-packages/hail/methods/statgen.py in linear_regression_rows(y, x, covariates, block_size, pass_through)
435 ht_result = ht_result.annotate(**{f: ht_result[f][0] for f in fields})
436
→ 437 return ht_result.persist()
438
439

</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-945> in persist(self, storage_level)

/usr/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
559 def wrapper(original_func, *args, **kwargs):
560 args
, kwargs
= check_all(__original_func, args, kwargs, checkers, is_method=is_method)
→ 561 return original_func(*args, **kwargs)
562
563 return wrapper

/usr/lib/python3.6/site-packages/hail/table.py in persist(self, storage_level)
1757 Persisted table.
1758 “”"
→ 1759 return Env.backend().persist_table(self, storage_level)
1760
1761 def unpersist(self) → ‘Table’:

/usr/lib/python3.6/site-packages/hail/backend/backend.py in persist_table(self, t, storage_level)
123
124 def persist_table(self, t, storage_level):
→ 125 return Table._from_java(self._to_java_ir(t._tir).pyPersist(storage_level))
126
127 def unpersist_table(self, t):

/usr/lib/python3.6/site-packages/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
→ 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

/usr/lib/python3.6/site-packages/hail/utils/java.py in deco(*args, **kwargs)
226 import pyspark
227 try:
→ 228 return f(*args, **kwargs)
229 except py4j.protocol.Py4JJavaError as e:
230 s = e.java_exception.toString()

/usr/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 “An error occurred while calling {0}{1}{2}”.
→ 336 format(target_id, “.”, name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o149.pyPersist

Dear danking,

The hail log file is available in the following link:-

https://drive.google.com/file/d/1HhwLI0Ay8JlziJlCqHZ7Cg0BoCcyJ9eJ/view?usp=sharing

I meet the same error. Have you solved it and how?

You’re getting this error from linear regression rows in Hail as well?

I suspect this is a BLAS issue. @wkl1990, what is your compute environment? Do you use Google Dataproc, Amazon EMR, an on-premises cluster, or something else? Do you have BLAS installed on your compute nodes? What is the OS and CPU architecture on your compute nodes and on your leader node?

I installed hail on my local server. The OS is Linux version 3.10.0-514.10.2.el7.x86_64 (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ). The CPU is CPU0: Intel® Xeon® CPU E7-8895 v3 @ 2.60GHz. I found the blas installed (blas-3.4.2-4.el7.x86_64).

You probably need to compile hail from source. I think we build the pip hail for newer CPUs.

Are you able to download the source and compile from source?

I downloaded the source of hail from github and installed successfully using “make install”. However, I still got the same error when “run linear_regression_rows”.

Hi All,

Could anyone point out the solution for the error discussed (question asked by @aby: py4j.protocol.Py4JNetworkError: Answer from Java side is empty? Still, we are stuck in this error, we had tried all the possibilities, but still it persists.

Does anyone have a solution for this?
Thanks a lot!