Hi Hail team,
I got an error when running hail.stop(), as below. Is there anyway to avoid this error? Thanks!
$ python hail_jointcalling_run_combiner.py -i snakemake_results/subsetGvcf/chr2 -o hail_runCombiner_result
2021-11-19 00:10:58 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
pip-installed Hail requires additional configuration options in Spark referring
to the path to the Hail Python module directory HAIL_DIR,
e.g. /path/to/python/site-packages/hail:
spark.jars=HAIL_DIR/hail-all-spark.jar
spark.driver.extraClassPath=HAIL_DIR/hail-all-spark.jar
spark.executor.extraClassPath=./hail-all-spark.jarRunning on Apache Spark version 2.4.0
SparkUI available at http://compute1-exec-131.ris.wustl.edu:4040
Welcome to
__ __ <>__
/ /_/ /__ __/ /
/ __ / _ `/ / /
/_/ /_/\_,_/_/_/ version 0.2.61-3c86d3ba497a
LOGGING: writing to hail_run_combiner_chr2_20211119.log
Temp Folder: hail_chr2_temp
Output MT: hail_runCombiner_result/neuropathy_batch1_hail_runCombiner_chr2.mt
Input folder: snakemake_results/subsetGvcf/chr2
CHR name: chr2
START RUNNING HAIL...
2021-11-19 00:11:00 Hail: INFO: Using 2586 intervals with default whole-genome size 1200000 as partitioning for GVCF import
2021-11-19 00:11:00 Hail: INFO: GVCF combiner plan:
Branch factor: 100
Batch size: 100
Combining 366 input files in 2 phases with 2 total jobs.
Phase 1: 1 job corresponding to 4 intermediate output files.
Phase 2: 1 job corresponding to 1 final output file.
2021-11-19 00:11:00 Hail: INFO: Starting phase 1/2, merging 366 input GVCFs in 1 job.
2021-11-19 00:11:00 Hail: INFO: Starting phase 1/2, job 1/1 to create 4 merged files, corresponding to ~50.0% of total I/O.
2021-11-19 04:54:57 Hail: INFO: Finished 1/2, job 1/1, 50.0% of total I/O finished.
2021-11-19 04:54:57 Hail: INFO: Finished phase 1/2.
2021-11-19 04:54:57 Hail: INFO: Starting phase 2/2, merging 4 intermediate sparse matrix tables in 1 job.
2021-11-19 04:55:47 Hail: INFO: Starting phase 2/2, job 1/1 to create 1 merged file, corresponding to ~50.0% of total I/O.
2021-11-19 06:12:58 Hail: INFO: wrote matrix table with 239680764 rows and 366 columns in 7975 partitions to hail_runCombiner_result/neuropathy_batch1_hail_runCombiner_chr2.mt
Total size: 138.13 GiB
* Rows/entries: 138.13 GiB
* Columns: 2.03 KiB
* Globals: 11.00 B
* Smallest partition: 0 rows (20.00 B)
* Largest partition: 35174 rows (34.36 MiB)
2021-11-19 06:12:58 Hail: INFO: Finished phase 2/2, job 1/1, 100% of total I/O finished.
2021-11-19 06:12:59 Hail: INFO: Finished phase 2/2.
2021-11-19 06:12:59 Hail: INFO: Finished!
--- 362.02374415397645 minute ---
Traceback (most recent call last):
File "hail_jointcalling_run_combiner.py", line 104, in <module>
hl.stop()
File "/opt/conda/lib/python3.7/site-packages/hail/context.py", line 361, in stop
Env.hc().stop()
File "/opt/conda/lib/python3.7/site-packages/hail/context.py", line 101, in stop
self._backend.stop()
File "/opt/conda/lib/python3.7/site-packages/hail/backend/spark_backend.py", line 219, in stop
self._jhc.stop()
File "/opt/conda/lib/python3.7/site-packages/py4j/java_gateway.py", line 1286, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/opt/conda/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 16, in deco
return f(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o42.stop.
: java.util.NoSuchElementException: key not found: __uid_9
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
at is.hail.expr.ir.functions.IRFunctionRegistry$.removeJVMFunction(Functions.scala:87)
at is.hail.expr.ir.functions.IRFunctionRegistry$$anonfun$clearUserFunctions$1.apply(Functions.scala:24)
at is.hail.expr.ir.functions.IRFunctionRegistry$$anonfun$clearUserFunctions$1.apply(Functions.scala:23)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at is.hail.expr.ir.functions.IRFunctionRegistry$.clearUserFunctions(Functions.scala:23)
at is.hail.HailContext$.stop(HailContext.scala:159)
at is.hail.HailContext.stop(HailContext.scala:407)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)