Java error in Jupyter notebook on google cloud

Hello,

I am getting an error when I run the following command:

ukbb.write('file_path', overwrite=True)

The error is posted below. When I ran the same code yesterday with the same set up I didn’t get an error, but I am now working with a larger dataset. Thanks!

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1159, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
    response = connection.send_command(command)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1164, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-fbd959e44a2b>", line 8, in <module>
    metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py", line 2426, in count
    return Env.backend().execute(count_ir)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 74, in execute
    result = json.loads(self._jhc.backend().executeJSON(jir))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py", line 16, in deco
    return f(*args, **kwargs)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
    format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o164.executeJSON

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'Py4JError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
<ipython-input-7-fbd959e44a2b> in <module>
      6 while(True):
      7     if not path.exists("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt"):
----> 8         metadata.to_csv("gs://ukb-gt/ukbQC_"+str(ukbb.count()[0])+"_x_"+str(ukbb.count()[1])+"_metadata"+str(imeta)+".txt", index=False)
      9         break
     10 

/opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py in count(self)
   2424         """
   2425         count_ir = ir.MatrixCount(self._mir)
-> 2426         return Env.backend().execute(count_ir)
   2427 
   2428     @typecheck_method(output=str,

/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py in execute(self, ir, timed)
     72         # print(self._hail_package.expr.ir.Pretty.apply(jir, True, -1))
     73         try:
---> 74             result = json.loads(self._jhc.backend().executeJSON(jir))
     75             value = ir.typ._from_json(result['value'])
     76             timings = result['timings']

/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j_backend.py in deco(*args, **kwargs)
     14         import pyspark
     15         try:
---> 16             return f(*args, **kwargs)
     17         except py4j.protocol.Py4JJavaError as e:
     18             s = e.java_exception.toString()

/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    334             raise Py4JError(
    335                 "An error occurred while calling {0}{1}{2}".
--> 336                 format(target_id, ".", name))
    337     else:
    338         type = answer[1]

Py4JError: An error occurred while calling o164.executeJSON
'''

I tried updating hail and gcloud using:

pip3 install -U hail
gcloud components update

Now I get the following error whenever I try to read or write to any file:

ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:35125)
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _get_connection(self)
    928         try:
--> 929             connection = self.deque.pop()
    930         except IndexError:
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
ConnectionRefusedError                    Traceback (most recent call last)
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in start(self)
   1066         try:
-> 1067             self.socket.connect((self.address, self.port))
   1068             self.stream = self.socket.makefile("rb")
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Py4JNetworkError                          Traceback (most recent call last)
<ipython-input-23-586fb829c89f> in <module>
      1 # Define phenotype and assessment centre classification:
----> 2 withdrawn = hl.read_table(withdrawn_file)
      3 gwassamples = hl.read_table(gwassamples_file)
<decorator-gen-1439> in read_table(path, _intervals, _filter_intervals)
/opt/conda/miniconda3/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
    612     def wrapper(__original_func, *args, **kwargs):
    613         args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
--> 614         return __original_func(*args_, **kwargs_)
    615 
    616     return wrapper
/opt/conda/miniconda3/lib/python3.6/site-packages/hail/methods/impex.py in read_table(path, _intervals, _filter_intervals)
   2436     :class:`.Table`
   2437     """
-> 2438     for rg_config in Env.backend().load_references_from_dataset(path):
   2439         hl.ReferenceGenome._from_config(rg_config)
   2440 
/opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/spark_backend.py in load_references_from_dataset(self, path)
    320 
    321     def load_references_from_dataset(self, path):
--> 322         return json.loads(Env.hail().variant.ReferenceGenome.fromHailDataset(self.fs._jfs, path))
    323 
    324     def from_fasta_file(self, name, fasta_file, index_file, x_contigs, y_contigs, mt_contigs, par):
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __getattr__(self, name)
   1589             self._jvm_id + "\n" +\
   1590             proto.END_COMMAND_PART
-> 1591         answer = self._gateway_client.send_command(command)
   1592         if answer == proto.SUCCESS_PACKAGE:
   1593             return JavaPackage(new_fqn, self._gateway_client, self._jvm_id)
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in send_command(self, command, retry, binary)
    981          if `binary` is `True`.
    982         """
--> 983         connection = self._get_connection()
    984         try:
    985             response = connection.send_command(command)
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _get_connection(self)
    929             connection = self.deque.pop()
    930         except IndexError:
--> 931             connection = self._create_connection()
    932         return connection
    933 
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in _create_connection(self)
    935         connection = GatewayConnection(
    936             self.gateway_parameters, self.gateway_property)
--> 937         connection.start()
    938         return connection
    939 
/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in start(self)
   1077                 "server ({0}:{1})".format(self.address, self.port)
   1078             logger.exception(msg)
-> 1079             raise Py4JNetworkError(msg, e)
   1080 
   1081     def _authenticate_connection(self):
Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:35125)

Can you share the hail log file? That has the true error.

First guess: missing linear algebra libraries. Did you apt get install openblas?

I don’t think I can create a log file without getting the error.

I tried the following:

import hail as hl
from hail.plot import show
from os import path
import os
import pandas as pd
hl.plot.output_notebook()

os.system('apt get install openblas')

hl.init(log='gs://ukb-gt/log.log')

But I still get the same error when I get to the last line of setting the log file. I also cannot find any log file in gs://ukb-gt/

You can’t log to a google cloud storage bucket, you have to log to a local file. A log will automatically be created by Hail when it initializes. It will print the log location after the Hail logo. It should look like this:

Welcome to
     __  __     <>__
    / /_/ /__  __/ /
   / __  / _ `/ / /
  /_/ /_/\_,_/_/_/   version 0.2.60-de1845e1c2f6
LOGGING: writing to /Users/dking/projects/hail/hail-20201130-0957-0.2.60-de1845e1c2f6.log

EDIT: Also, does apt get actually work? It’s been a while since I was a daily linux user but I thought it had to be apt-get. (Typo mine in my previous post, apologies)