Hi there,
I’m trying to run logistic regression with Hail, and I experienced similar error as in this post:
I tried the suggestion in that post, but the same error still occured. Here is my script:
import hail as hl
hl.init(spark_conf={"spark.executor.extraClassPath": "/usr/lib64/libblas.so:/usr/lib64/liblapack.so"})
mt = hl.import_plink(bed='chr22_sexl1_40_40.bed',bim='chr22_sexl1_40_40.bim',fam='chr22_sexl1_40_40.fam',quant_pheno=True)
covar = (hl.import_table('covar_l1.txt',types={'IID':hl.tstr},impute=True).key_by('IID'))
mt = mt.annotate_cols(covar=covar[mt.s])
result_ht = hl.logistic_regression_rows(test='wald', y=mt.covar.pheno, x=mt.GT.n_alt_alleles(), covariates=[1, mt.covar.isAxiom])
And here is my full stack trace:
2020-10-01 13:22:19 Hail: INFO: Found 37652 samples in fam file.
2020-10-01 13:22:19 Hail: INFO: Found 2302 variants in bim file.
2020-10-01 13:22:20 Hail: INFO: Reading table to impute column types
[Stage 0:> (0 + 1) / 1]2020-10-01 13:22:22 Hail: INFO: Finished type imputation
Loading column 'IID' as type 'str' (user-specified)
Loading column 'isMale' as type 'int32' (imputed)
Loading column 'isAxiom' as type 'bool' (imputed)
Loading column 'PC1' as type 'float64' (imputed)
Loading column 'PC2' as type 'float64' (imputed)
Loading column 'PC3' as type 'float64' (imputed)
Loading column 'PC4' as type 'float64' (imputed)
Loading column 'PC5' as type 'float64' (imputed)
Loading column 'PC6' as type 'float64' (imputed)
Loading column 'PC7' as type 'float64' (imputed)
Loading column 'PC8' as type 'float64' (imputed)
Loading column 'PC9' as type 'float64' (imputed)
Loading column 'PC10' as type 'float64' (imputed)
Loading column 'PC11' as type 'float64' (imputed)
Loading column 'PC12' as type 'float64' (imputed)
Loading column 'PC13' as type 'float64' (imputed)
Loading column 'PC14' as type 'float64' (imputed)
Loading column 'PC15' as type 'float64' (imputed)
Loading column 'PC16' as type 'float64' (imputed)
Loading column 'PC17' as type 'float64' (imputed)
Loading column 'PC18' as type 'float64' (imputed)
Loading column 'PC19' as type 'float64' (imputed)
Loading column 'PC20' as type 'float64' (imputed)
Loading column 'pheno' as type 'bool' (imputed)
[Stage 1:> (0 + 1) / 1]2020-10-01 13:22:27 Hail: INFO: logistic_regression_rows: running wald on 37652 samples for response variable y,
with input variable x, and 2 additional covariates...
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java: symbol lookup error: /tmp/jniloader41302633382189116netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dgemv
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1159, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 985, in send_command
response = connection.send_command(command)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1164, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "test_logistic.py", line 7, in <module>
result_ht = hl.logistic_regression_rows(test='wald', y=mt.covar.pheno, x=mt.GT.n_alt_alleles(), covariates=[1, mt.covar.isAxiom])
File "<decorator-gen-1525>", line 2, in logistic_regression_rows
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py", line 614, in wrapper
return __original_func(*args_, **kwargs_)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/methods/statgen.py", line 865, in logistic_regression_rows
return result.persist()
File "<decorator-gen-1085>", line 2, in persist
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py", line 614, in wrapper
return __original_func(*args_, **kwargs_)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/table.py", line 1836, in persist
return Env.backend().persist_table(self, storage_level)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark_backend.py", line 315, in persist_table
return Table._from_java(self._jbackend.pyPersistTable(storage_level, self._to_java_table_ir(t._tir)))
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark_backend.py", line 29, in deco
return f(*args, **kwargs)
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1.pyPersistTable
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:8792)
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:8792)
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 929, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1067, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
May I ask if there’s anything else I can do to resolve this problem? Thanks!