JAVA version problem

Hail Query & hailctl
1

Hello,

I installed the hail according to the guideline, JAVA version 11.
java-11-openjdk.x86_64 (/usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el7_8.x86_64/bin/java)
However, some of functions are not working as example below.
I would like to ask you how to solve this problem.

Thanks

logistic = hl.logistic_regression_rows(x=mt_pheno.DS, y=mt_pheno.pheno.CASE, covariates=[1.0, mt_pheno.pheno.AGE], test=‘wald’)

2024-11-09 02:34:27.866 Hail: INFO: logistic_regression_rows: running wald on 504 samples for response variable y,
with input variable x, and 2 additional covariates…
/home/joonho345/.jenv/versions/11.0.8/bin/java: symbol lookup error: /tmp/jniloader6572155367366312789netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dgemv
Traceback (most recent call last):
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 716, in urlopen
httplib_response = self._make_request(
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 468, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 463, in _make_request
httplib_response = conn.getresponse()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 1377, in getresponse
response.begin()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 320, in begin
version, status, reason = self._read_status()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 289, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/requests/adapters.py”, line 667, in send
resp = conn.urlopen(
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 802, in urlopen
retries = retries.increment(
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/util/retry.py”, line 552, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/packages/six.py”, line 769, in reraise
raise value.with_traceback(tb)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 716, in urlopen
httplib_response = self._make_request(
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 468, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/urllib3/connectionpool.py”, line 463, in _make_request
httplib_response = conn.getresponse()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 1377, in getresponse
response.begin()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 320, in begin
version, status, reason = self._read_status()
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/http/client.py”, line 289, in _read_status
raise RemoteDisconnected(“Remote end closed connection without”
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “”, line 1, in
File “”, line 2, in logistic_regression_rows
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/methods/statgen.py”, line 1025, in logistic_regression_rows
return result.persist()
File “”, line 2, in persist
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/table.py”, line 2761, in persist
return Env.backend().persist(self)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/backend/backend.py”, line 292, in persist
persisted = dataset.checkpoint(tempfile.enter())
File “”, line 2, in checkpoint
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/table.py”, line 1963, in checkpoint
self.write(output=output, overwrite=overwrite, stage_locally=stage_locally, _codec_spec=_codec_spec)
File “”, line 2, in write
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/table.py”, line 2005, in write
Env.backend().execute(
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/backend/spark_backend.py”, line 217, in execute
raise err
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/backend/spark_backend.py”, line 209, in execute
return super().execute(ir, timed)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/backend/backend.py”, line 179, in execute
result, timings = self._rpc(ActionTag.EXECUTE, payload)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/hail/backend/py4j_backend.py”, line 218, in _rpc
resp = self._requests_session.post(f’http://localhost:{port}{path}', data=data)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/requests/sessions.py”, line 637, in post
return self.request(“POST”, url, data=data, json=json, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/requests/sessions.py”, line 589, in request
resp = self.send(prep, **send_kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/requests/sessions.py”, line 703, in send
r = adapter.send(request, **kwargs)
File “/home/joonho345/miniconda3/envs/hail_env/lib/python3.9/site-packages/requests/adapters.py”, line 682, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))

2

Hi @Joonho_Kim,
Sadly that error doesn’t tell us much (and by no fault of your own).
After installing jdk-11, are you setting the environment variable JAVA_HOME?
Do you have any other java installations on your computer? (what’s the output of java --version?)
Can you run any simple hail commands, such as:

hl.init()
hl.utils.range_table(10)._force_count()
3

Hi ehigham,

I appreciate your answer. This is the verision of java
(base) [joonho345@compute04 ~]$ java --version
openjdk 11.0.8 2020-07-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)

I created the conda env for hail and these are the packages installed

(hail_env) [joonho345@compute04 ~]$ conda list

packages in environment at /home/joonho345/miniconda3/envs/hail_env:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
_sysroot_linux-64_curr_repodata_hack 3 h69a702a_16 conda-forge
aiodns 2.0.0 pypi_0 pypi
aiohappyeyeballs 2.4.0 pypi_0 pypi
aiohttp 3.10.5 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
alsa-lib 1.2.12 h4ab18f5_0 conda-forge
async-timeout 4.0.3 pypi_0 pypi
attrs 24.2.0 pypi_0 pypi
avro 1.11.3 pypi_0 pypi
azure-common 1.1.28 pypi_0 pypi
azure-core 1.30.2 pypi_0 pypi
azure-identity 1.17.1 pypi_0 pypi
azure-mgmt-core 1.4.0 pypi_0 pypi
azure-mgmt-storage 20.1.0 pypi_0 pypi
azure-storage-blob 12.22.0 pypi_0 pypi
binutils_impl_linux-64 2.40 ha1999f0_7 conda-forge
binutils_linux-64 2.40 hb3c18ed_4 conda-forge
blas 2.123 openblas conda-forge
blas-devel 3.9.0 23_linux64_openblas conda-forge
bokeh 3.3.4 pypi_0 pypi
boto3 1.35.10 pypi_0 pypi
botocore 1.35.10 pypi_0 pypi
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2024.8.30 hbcca054_0 conda-forge
cachetools 5.5.0 pypi_0 pypi
cairo 1.18.0 hebfffa5_3 conda-forge
certifi 2024.8.30 pypi_0 pypi
cffi 1.17.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
contourpy 1.3.0 pypi_0 pypi
cryptography 43.0.0 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
deprecated 1.2.14 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
expat 2.6.2 h59595ed_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_2 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
frozenlist 1.4.1 pypi_0 pypi
gcc_impl_linux-64 11.4.0 h00c12a0_13 conda-forge
gcc_linux-64 11.4.0 ha077dfb_4 conda-forge
giflib 5.2.2 hd590300_0 conda-forge
google-auth 2.34.0 pypi_0 pypi
google-auth-oauthlib 0.8.0 pypi_0 pypi
graphite2 1.3.13 h59595ed_1003 conda-forge
gxx_impl_linux-64 11.4.0 h634f3ee_13 conda-forge
gxx_linux-64 11.4.0 h35bfe5d_4 conda-forge
hail 0.2.132 pypi_0 pypi
harfbuzz 9.0.0 hda332d3_1 conda-forge
humanize 1.1.0 pypi_0 pypi
icu 75.1 he02047a_0 conda-forge
idna 3.8 pypi_0 pypi
isodate 0.6.1 pypi_0 pypi
janus 1.0.0 pypi_0 pypi
jinja2 3.1.4 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
jproperties 2.1.2 pypi_0 pypi
kernel-headers_linux-64 3.10.0 h4a8ded7_16 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
lapack 3.9.0 netlib conda-forge
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libblas 3.9.0 23_linux64_openblas conda-forge
libcblas 3.9.0 23_linux64_openblas conda-forge
libcups 2.3.3 h4637d8d_4 conda-forge
libdeflate 1.21 h4bc722e_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libexpat 2.6.2 h59595ed_0 conda-forge
libffi 3.4.4 h6a678d5_1
libgcc 14.1.0 h77fa898_1 conda-forge
libgcc-devel_linux-64 11.4.0 h8f596e0_113 conda-forge
libgcc-ng 14.1.0 h69a702a_1 conda-forge
libgfortran 14.1.0 h69a702a_1 conda-forge
libgfortran-ng 14.1.0 h69a702a_1 conda-forge
libgfortran5 14.1.0 hc5f4f2c_1 conda-forge
libglib 2.80.3 h315aac3_2 conda-forge
libgomp 14.1.0 h77fa898_1 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
liblapack 3.9.0 23_linux64_openblas conda-forge
liblapacke 3.9.0 23_linux64_openblas conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.27 pthreads_hac2b453_1 conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libsanitizer 11.4.0 h5763a12_13 conda-forge
libsqlite 3.45.2 h2797004_0 conda-forge
libstdcxx 14.1.0 hc0a3c3a_1 conda-forge
libstdcxx-devel_linux-64 11.4.0 h8f596e0_113 conda-forge
libstdcxx-ng 14.1.0 h4852527_1 conda-forge
libtiff 4.6.0 h46a8edc_4 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.4.0 hd590300_0 conda-forge
libxcb 1.16 hb9d3cd8_1 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
llvm-openmp 18.1.8 hf5423f3_1 conda-forge
markupsafe 2.1.5 pypi_0 pypi
msal 1.30.0 pypi_0 pypi
msal-extensions 1.2.0 pypi_0 pypi
msrest 0.7.1 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openblas 0.3.27 pthreads_h9eca1d5_1 conda-forge
openjdk 11.0.23 h38bd402_1 conda-forge
openssl 3.3.1 hb9d3cd8_3 conda-forge
orjson 3.10.7 pypi_0 pypi
packaging 24.1 pypi_0 pypi
pandas 2.2.2 pypi_0 pypi
parsimonious 0.10.0 pypi_0 pypi
pcre2 10.44 hba22ea6_2 conda-forge
pillow 10.4.0 pypi_0 pypi
pip 24.2 py39h06a4308_0
pixman 0.43.2 h59595ed_0 conda-forge
plotly 5.24.0 pypi_0 pypi
portalocker 2.10.1 pypi_0 pypi
protobuf 3.20.2 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
py4j 0.10.9.7 pypi_0 pypi
pyasn1 0.6.0 pypi_0 pypi
pyasn1-modules 0.4.0 pypi_0 pypi
pycares 4.4.0 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
pyjwt 2.9.0 pypi_0 pypi
pyspark 3.5.2 pypi_0 pypi
python 3.9.18 h0755675_1_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python-json-logger 2.0.7 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2024.7.24 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
requests-oauthlib 2.0.0 pypi_0 pypi
rich 12.6.0 pypi_0 pypi
rsa 4.9 pypi_0 pypi
s3transfer 0.10.2 pypi_0 pypi
scipy 1.11.4 pypi_0 pypi
setuptools 72.1.0 py39h06a4308_0
shellingham 1.5.4 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
sqlite 3.45.2 h2c6b66d_0 conda-forge
sysroot_linux-64 2.17 h4a8ded7_16 conda-forge
tabulate 0.9.0 pypi_0 pypi
tenacity 9.0.0 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tornado 6.4.1 pypi_0 pypi
typer 0.12.5 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
urllib3 1.26.20 pypi_0 pypi
uvloop 0.20.0 pypi_0 pypi
wheel 0.43.0 py39h06a4308_0
wrapt 1.16.0 pypi_0 pypi
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.1.1 hd590300_0 conda-forge
xorg-libsm 1.2.4 h7391055_0 conda-forge
xorg-libx11 1.8.9 hb711507_1 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge
xorg-libxi 1.7.10 h4bc722e_1 conda-forge
xorg-libxrender 0.9.11 hd590300_0 conda-forge
xorg-libxt 1.3.0 hd590300_1 conda-forge
xorg-libxtst 1.2.5 h4bc722e_0 conda-forge
xorg-recordproto 1.14.2 h7f98852_1002 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xyzservices 2024.6.0 pypi_0 pypi
xz 5.4.6 h5eee18b_1
yarl 1.9.7 pypi_0 pypi
zlib 1.3.1 h4ab18f5_1 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge

those commands seems runnable, but some functions are not working like the one that I uploaded.

import hail as hl
hl.init()
SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.
SLF4J: Defaulting to no-operation (NOP) logger implementation
Running on Apache Spark version 3.5.2
SparkUI available at http://compute04:4040
Welcome to
__ __ <>__
/ // /__ __/ /
/ __ / _ `/ / /
// //_,/// version 0.2.132-678e1f52b999
LOGGING: writing to /home/joonho345/hail-20241109-1013-0.2.132-678e1f52b999.log
hl.utils.range_table(10)._force_count()
SLF4J: Failed to load class “org.slf4j.impl.StaticMDCBinder”.
SLF4J: Defaulting to no-operation MDCAdapter implementation.
10tage 0:> (0 + 10) / 10]

4

Thanks for your detailed reply. That all looks fine so I think this isn’t an installation issue.
It’s hard to know what’s the issue without seeing the logfile hail writes on your current directory. The error message isn’t particularly insightful. If you have the log, could you provide it?
That said, I’ve seen those kind of errors when the spark driver runs out of memory and dies silently. You could try increasing your JVM heap memory and seeing if that helps.

5

Thanks for you considerable answer. I provide the log file for the first command with error. ‘logistic = hl.logistic_regression_rows(x=mt_pheno.DS, y=mt_pheno.pheno.CASE, covariates=[1.0, mt_pheno.pheno.AGE], test=‘wald’)’
hail-20241109-0218-0.2.132-678e1f52b999.log (2.1 MB)

I also tried to set spark memory but still not working

hl.init(spark_conf={‘spark.driver.memory’: ‘100g’})
hl.init(spark_conf={‘spark.driver.memory’: ‘20g’, ‘spark.executor.memory’ : ‘20g’})

Do I have to increase the memory more?
Or is there other way to increase JVM heap memory?