Py4JError: An error occurred while calling o201.pyPersist

Hail Query & hailctl
1

Hi I’m running the tutorial. All well until the actual gwas analysis call:

gwas = hl.linear_regression_rows(y=mt.pheno.CaffeineConsumption, x=mt.GT.n_alt_alleles(), covariates=[1.0])

When I get a stream of error output - I think it may be a memory issue but Im on a HPC and I am very surprised its happening on a small tutorial dataset. Output:

`
2019-12-26 16:20:57 Hail: INFO: linear_regression_rows: running on 250 samples for 1 response variable y,
with input variable x, and 1 additional covariate…
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1159, in send_command
raise Py4JNetworkError(“Answer from Java side is empty”)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 985, in send_command
response = connection.send_command(command)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1164, in send_command
“Error while receiving”, e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:42828)
Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 3319, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File “”, line 3, in
covariates=[1.0])
File “</fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/decorator.py:decorator-gen-1497>”, line 2, in linear_regression_rows
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/methods/statgen.py”, line 447, in linear_regression_rows
return ht_result.persist()
File “</fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/decorator.py:decorator-gen-1057>”, line 2, in persist
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/table.py”, line 1788, in persist
return Env.backend().persist_table(self, storage_level)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/backend/backend.py”, line 128, in persist_table
return Table._from_java(self._to_java_ir(t._tir).pyPersist(storage_level))
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/hail/utils/java.py”, line 213, in deco
return f(*args, **kwargs)
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/protocol.py”, line 336, in get_return_value
format(target_id, “.”, name))
py4j.protocol.Py4JError: An error occurred while calling o201.pyPersist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 2034, in showtraceback
stb = value.render_traceback()
AttributeError: ‘Py4JError’ object has no attribute ‘render_traceback’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused

Py4JError Traceback (most recent call last)
in
1 gwas = hl.linear_regression_rows(y=mt.pheno.CaffeineConsumption,
2 x=mt.GT.n_alt_alleles(),
----> 3 covariates=[1.0])
4 gwas.row.describe()

</fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/decorator.py:decorator-gen-1497> in linear_regression_rows(y, x, covariates, block_size, pass_through)

~/.conda/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
583 def wrapper(original_func, *args, **kwargs):
584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 585 return original_func(*args, **kwargs)
586
587 return wrapper

~/.conda/envs/hail/lib/python3.7/site-packages/hail/methods/statgen.py in linear_regression_rows(y, x, covariates, block_size, pass_through)
445 ht_result = ht_result.annotate(**{f: ht_result[f][0] for f in fields})
446
–> 447 return ht_result.persist()
448
449

</fs/home/cliveterence.dar/.conda/envs/hail/lib/python3.7/site-packages/decorator.py:decorator-gen-1057> in persist(self, storage_level)

~/.conda/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
583 def wrapper(original_func, *args, **kwargs):
584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 585 return original_func(*args, **kwargs)
586
587 return wrapper

~/.conda/envs/hail/lib/python3.7/site-packages/hail/table.py in persist(self, storage_level)
1786 Persisted table.
1787 “”"
-> 1788 return Env.backend().persist_table(self, storage_level)
1789
1790 def unpersist(self) -> ‘Table’:

~/.conda/envs/hail/lib/python3.7/site-packages/hail/backend/backend.py in persist_table(self, t, storage_level)
126
127 def persist_table(self, t, storage_level):
–> 128 return Table._from_java(self._to_java_ir(t._tir).pyPersist(storage_level))
129
130 def unpersist_table(self, t):

~/.conda/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/.conda/envs/hail/lib/python3.7/site-packages/hail/utils/java.py in deco(*args, **kwargs)
211 import pyspark
212 try:
–> 213 return f(*args, **kwargs)
214 except py4j.protocol.Py4JJavaError as e:
215 s = e.java_exception.toString()

~/.conda/envs/hail/lib/python3.7/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 “An error occurred while calling {0}{1}{2}”.
–> 336 format(target_id, “.”, name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o201.pyPersist
`

2

What system are you running on? This could potentially be a native library mismatch.

3

Hi sorry for late response

I’m on GNU/Linux:

Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Thanks

C

4

@cliveno1 Hail requires GCC 5.0 (or LLVM 3.4) or later, which will install a compatible libstdc++. We also require a recent version of glibc, at least 2.14. I suspect your glibc is sufficient because you have a relatively recent version of the kernel. You probably just need to install gcc or llvm on the leader node and every worker node.

6

Hi there

So, when I type: gcc --version, I get: gcc (GCC) 7.3.0

So, I assume my gcc is compatible. However, when I type: glibc --version, I get: command not found

So, I’m guessing glibc is the issue. Can you give any advice to install this, I’m struggling to find anything I can understand on the web

Thanks

7

glibc is not a tool like gcc, it’s a shared library file on your system. glibc is distributed with ldd, a tool for inspecting compiled files, as such, they share a version. You can check glibc's version with ldd --version. Anyway, I suspect this is not the issue.

My next guess is a BLAS/LAPACK issue. Can you attach the log from a session where linear_regression_rows fails? The log location should be printed when Hail initializes. Within this log is hopefully the true error message.

8

Hi I’ll email attach this log due to internet issues (even for a 1Mb doc) using the forum pages here in Thailand

Thanks

hail-20200107-2145-0.2.30-2ae07d872f43.log (1.03 MB)

9

It appears there is an issue with BLAS/LAPACK. My guess is that the implementations of BLAS or LAPACK provided by your distribution of GNU/Linux is binary-incompatible with the normal BLAS and LAPACK interfaces. A similar problem is described at the netlib forum. hntd187 ultimately fixed their issue by installing/compiling OpenBLAS and ensuring that libblas.so on their machine was a symlink to OpenBLAS (see hntd187’s comment on fommil/netlib-java#87). If you’re on an HPC cluster, it’s possible you can fix this by useing an appropriate version of OpenBLAS or MKL.

10

Thanks for your input