Hi
I’m trying to run a script that includes the pc_rel function:
# Repartition
mt = mt.repartition(300).persist()
# Run kinship analysis
pc_rel = hl.pc_relate(mt.GT, 0.001, k=2, min_kinship=0.1)
# Export pc_rel to tsv
pc_rel_filename = output_prefix + '.hail_kinship.tsv'
pc_rel_path = os.path.join(qc_dir, pc_rel_filename)
pc_rel.flatten().export(pc_rel_path, delimiter = "\t")
However the script fails with the following trackback:
2024-02-08 10:03:11,290 An error occurred while trying to connect to the Java server (127.0.0.1:41254)
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
2024-02-08 10:03:11,292 An error occurred while trying to connect to the Java server (127.0.0.1:41254)
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
2024-02-08 10:03:11,292 An error occurred while trying to connect to the Java server (127.0.0.1:41254)
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
2024-02-08 10:03:11,292 An error occurred while trying to connect to the Java server (127.0.0.1:41254)
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
I believe this is memory issue, but it occurs even when I allocate very large amounts of memory:
PYSPARK_SUBMIT_ARGS=" --driver-memory 500g --executor-memory 500g pyspark-shell"
Does anyone have suggestions how I might resolve this?
Thanks