# When running Logistic Regression Rows I get a Py4JNetworkError

**URL:** <https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704>\
**Category:** Hail Query & hailctl\
**Created:** [September 23, 2020, 6:20pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704 "2020-09-23T18:20:54Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![fengyi](https://avatars.discourse-cdn.com/v4/letter/f/977dab/32.png) [@fengyi](https://discuss.hail.is/u/fengyi)\
**Post date:** [September 23, 2020, 6:20pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/1 "2020-09-23T18:20:54Z")

</div>

Like some other topics, I have also received errors relating to py4j.protocol.Py4JNetworkError: Answer from Java side is empty when I am trying to run

result\_ht =hl.logistic\_regression\_rows()

 ![Screen Shot 2020-09-23 at 1.17.57 PM](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/7f91b6fb48fd7209b0a32c4df0eac7f2d2755dac.png)

I am not sure if their solution apply to me and should I build hail from [https://hail.is/docs/0.2/getting\_started\_developing.html#building-hail](https://hail.is/docs/0.2/getting_started_developing.html#building-hail)?

Thanks!

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [September 23, 2020, 6:27pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/2 "2020-09-23T18:27:28Z")

</div>

I’d try this: [Undefined symbol: cblas\_dgemm](https://discuss.hail.is/t/undefined-symbol-cblas-dgemm/1488)

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [September 23, 2020, 6:49pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/3 "2020-09-23T18:49:44Z")

</div>

You might also simply not have the BLAS libraries installed. Verify that you’ve followed all the steps under [installing Hail for Linux](https://hail.is/docs/0.2/install/linux.html).

---

<div class="post-metadata">

**Author:** ![Stephen](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Stephen](https://discuss.hail.is/u/Stephen)\
**Post date:** [September 27, 2020, 10:31pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/4 "2020-09-27T22:31:11Z")

</div>

Hi @danking ,

If I would like to install Hail on remote linux servers (cluster), I do not have the permission to run

> apt-get install -y   
> openjdk-8-jre-headless   
> g++   
> python3.6 python3-pip   
> libopenblas-base liblapack3

What should I do if I want to install these packages for Hail?

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [September 28, 2020, 1:11am UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/5 "2020-09-28T01:11:45Z")

</div>

You’ll need to ask the system administrator or IT department to install these packages for you. Every compute cluster has a different system for doing this. Some are called “dot kits”.

---

<div class="post-metadata">

**Author:** ![fengyi](https://avatars.discourse-cdn.com/v4/letter/f/977dab/32.png) [@fengyi](https://discuss.hail.is/u/fengyi)\
**Post date:** [October 1, 2020, 8:42pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/6 "2020-10-01T20:42:31Z")

</div>

Hi Dan,

I have confirmed with the IT department and we have all those packages installed. Do you know what could be other problems/solutions for this?

Thanks,  
-Fengyi

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [October 1, 2020, 8:50pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/7 "2020-10-01T20:50:48Z")

</div>

Hey @fengyi ,

You probably need to explicitly tell Spark (a library that Hail uses) where BLAS and LAPACK are. We explain how to do that in this thread: [Undefined symbol: cblas\_dgemm](https://discuss.hail.is/t/undefined-symbol-cblas-dgemm/1488/2).

---

<div class="post-metadata">

**Author:** ![fengyi](https://avatars.discourse-cdn.com/v4/letter/f/977dab/32.png) [@fengyi](https://discuss.hail.is/u/fengyi)\
**Post date:** [October 3, 2020, 6:53am UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/8 "2020-10-03T06:53:50Z")

</div>

Hi Dan,

I follow the explanation in the solution you provided. But this error occurred again, can you help me to take a look at what else could go wrong? I have attached my code below.

 ![Screen Shot 2020-10-03 at 2.47.23 AM](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/753d35444e705c69b5b32cecb33154d0624917b4.png)

I used:

hl.init(spark\_conf={

“spark.executor.extraClassPath”: “/usr/lib64/libblas.so.3.4.2.so:/usr/lib64/liblapack.so.3.4.2”})

mt=hl.import\_plink(bed= ‘,bim=’ ‘,fam=’/ ',quant\_pheno=True,types={‘quant\_pheno’:hl.tfloat64})

mt= mt.annotate\_cols(is\_case =

hl.case()

.when(mt.quant\_pheno == -9, **False** )

.when(mt.quant\_pheno == 1, **True** )

.or\_missing())

mt = mt.drop(‘quant\_pheno’)

covar = (hl.import\_table(‘covar.txt’,types={‘IID’:hl.tstr},impute= **True** ).key\_by(‘IID’))

mt = mt.annotate\_cols(covar=covar[mt.s])

result\_ht =hl.logistic\_regression\_rows(test=‘wald’,

y=mt.is\_case,

x=mt.GT.n\_alt\_alleles() ,

covariates = [1,mt.covar.X1, mt.covar.X2]) # this line is where the code has an error.

There are a few versions of liblapack.so and libblas.so. So I just used the most recent I think.

Please let me know what else I can provide to clarify my problem!

Thanks!  
-Fengyi

---

<div class="post-metadata">

**Author:** ![wsa](https://avatars.discourse-cdn.com/v4/letter/w/a9a28c/32.png) [@wsa](https://discuss.hail.is/u/wsa)\
**Post date:** [October 15, 2020, 6:31pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/9 "2020-10-15T18:31:45Z")

</div>

I am helping fengyi try to get hail to work. I’ve gotten past the BLAS errors, and have run into some other failure. It’s not clear from the error message what exactly spark is unhappy about. Are there settings I can use to get much more verbose error messages from the spark/java side?

```
2020-10-13 14:21:11 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
2020-10-13 14:21:12 WARN Hail:37 - This Hail JAR was compiled for Spark 2.4.5, running with Spark 2.4.1. 
  Compatibility is not guaranteed. 
Running on Apache Spark version 2.4.1 
SparkUI available at http://nebula-3:4040 
Welcome to 
     ____ <>__ 
    / /_/ / ____ / / 
   / __ / _ `/ / / 
  /_/ /_/\_,_/_/_/ version 0.2.57-582b2e31b8bd 
LOGGING: writing to /ua/annis/hail/hail-20201013-1421-0.2.57-582b2e31b8bd.log 
2020-10-13 14:21:17 Hail: WARN: Hail has already been initialized. If this call was intended to change configuration, close the session with hl.stop() first. 
Traceback (most recent call last): 
  File "orig.py", line 8, in <module> 
    hl.init() 
  File "<decorator-gen-1758>", line 2, in init 
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/hail/typecheck/check.py", line 614, in wrapper 
    return __original_func(*args_, **kwargs_) 
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/hail/context.py", line 231, in init 
    skip_logging_configuration, optimizer_iterations) 
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/hail/backend/spark_backend.py", line 194, in __init__ 
    jsc, app_name, master, local, True, min_block_size, tmpdir, local_tmpdir) 
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/py4j/java_gateway.py", line 1257, in __call__ 
    answer, self.gateway_client, self.target_id, self.name) 
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/hail/backend/spark_backend.py", line 42, in deco 
    'Error summary: %s' % (deepest, full, hail. __version__ , deepest)) from None 
hail.utils.java.FatalError: IllegalArgumentException: requirement failed 

Java stack trace: 
java.lang.IllegalArgumentException: requirement failed 
        at scala.Predef$.require(Predef.scala:212) 
        at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:191) 
        at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:498) 
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) 
        at py4j.Gateway.invoke(Gateway.java:282) 
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 
        at py4j.commands.CallCommand.execute(CallCommand.java:79) 
        at py4j.GatewayConnection.run(GatewayConnection.java:238) 
        at java.lang.Thread.run(Thread.java:748) 

Hail version: 0.2.57-582b2e31b8bd 
Error summary: IllegalArgumentException: requirement failed
```

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [October 29, 2020, 2:32pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/10 "2020-10-29T14:32:15Z")

</div>

> [@wsa](#):
>
> ```python
> 2020-10-13 14:21:17 Hail: WARN: Hail has already been initialized. If this call was intended to change configuration, close the session with hl.stop() first. 
> 
> ```

Hi @wsa,

Sorry for the huge delay, I forgot about this thread.

It looks you call `hl.init` twice? Can you share `orig.py`? What happens on lines 1 to 8?

EDIT: There is also more debugging information in the hail log file.

---

<div class="post-metadata">

**Author:** ![wsa](https://avatars.discourse-cdn.com/v4/letter/w/a9a28c/32.png) [@wsa](https://discuss.hail.is/u/wsa)\
**Post date:** [October 29, 2020, 7:54pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/11 "2020-10-29T19:54:39Z")

</div>

The hail logs don’t give any more information than the screen message gives.

The first 8 lines of `orig.py`, omitting the actual path names:

```
#!/s/bin/python3.6

import hail as hl

hl.stop()
hl.init(spark_conf={"spark.executor.extraClassPath": "/usr/lib64/libopenblas.so:/usr/lib64/liblapack.so"})

hl.init()
mt = hl.import_plink(bed='.bed',bim='.bim',fam='.fam',quant_pheno=True)
```

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [October 29, 2020, 8:04pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/12 "2020-10-29T20:04:25Z")

</div>

You shouldn’t have that second `hl.init()`. You can only initialize hail once.

---

<div class="post-metadata">

**Author:** ![wsa](https://avatars.discourse-cdn.com/v4/letter/w/a9a28c/32.png) [@wsa](https://discuss.hail.is/u/wsa)\
**Post date:** [October 29, 2020, 8:20pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/13 "2020-10-29T20:20:52Z")

</div>

I can’t believe I missed that. I’ve been staring at the code for too long.

Now we’ve moved on to new errors, but I suspect this is a problem in the data files rather than hail itself:

```
2020-10-29 15:17:10 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:233
2020-10-29 15:17:10 Hail: WARN: Interpreting value '-9' as a valid quantitative phenotype, which differs from default PLINK behavior. Use missing='-9' to interpret '-9' as a missing value.
2020-10-29 15:17:10 Hail: WARN: Interpreting value '-9' as a valid quantitative phenotype, which differs from default PLINK behavior. Use missing='-9' to interpret '-9' as a missing value.
```

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [October 29, 2020, 8:25pm UTC](https://discuss.hail.is/t/when-running-logistic-regression-rows-i-get-a-py4jnetworkerror/1704/14 "2020-10-29T20:25:28Z")

</div>

So these are warnings, not errors. You just have to decide what you intended. If you want -9 to mean “missing”, which you probably do I’m guessing, you can add `missing='-9'` as an option to `import_plink`.