2022-03-02 22:08:06 Hail: INFO: ld_prune: running local pruning stage with max queue size of 176139 variants 2022-03-02 23:11:18 Hail: INFO: wrote table with 25605619 rows in 9147 partitions to /tmp/BIBcmFdxxBdewY53KJSZQJ Total size: 478.21 MiB * Rows: 478.21 MiB * Globals: 11.00 B * Smallest partition: 330 rows (7.92 KiB) * Largest partition: 53963 rows (1.52 MiB) 2022-03-03 00:12:34 Hail: INFO: Wrote all 6252 blocks of 25605619 x 1324 matrix with block size 4096. Traceback (most recent call last): (0 + 64) / 12559] File "", line 1, in File "", line 2, in ld_prune File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper return __original_func(*args_, **kwargs_) File "/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py", line 3564, in ld_prune entries.i, entries.j, keep=False, tie_breaker=tie_breaker, keyed=False) File "", line 2, in maximal_independent_set File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper return __original_func(*args_, **kwargs_) File "/usr/local/lib/python3.7/site-packages/hail/methods/misc.py", line 151, in maximal_independent_set edges.write(edges_path) File "", line 2, in write File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper return __original_func(*args_, **kwargs_) File "/usr/local/lib/python3.7/site-packages/hail/table.py", line 1271, in write Env.backend().execute(ir.TableWrite(self._tir, ir.TableNativeWriter(output, overwrite, stage_locally, _codec_spec))) File "/usr/local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 110, in execute raise e File "/usr/local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 86, in execute result_tuple = self._jhc.backend().executeEncode(jir, stream_codec) File "/usr/lib/spark/python/lib/py4j-src.zip/py4j/java_gateway.py", line 1305, in __call__ File "/usr/local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 31, in deco 'Error summary: %s' % (deepest, full, hail.__version__, deepest), error_id) from None hail.utils.java.FatalError: SparkException: Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-00-00-00-00.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-00-00-00-00.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . Driver stacktrace: Java stack trace: org.apache.spark.SparkException: Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-00-00-00-00.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-00-00-00-00.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2465) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2414) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2413) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2413) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1124) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1124) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1124) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2679) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2621) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2610) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:914) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2259) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2278) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) at org.apache.spark.rdd.RDD.collect(RDD.scala:1029) at is.hail.sparkextras.ContextRDD.collect(ContextRDD.scala:176) at is.hail.utils.richUtils.RichContextRDD.writePartitions(RichContextRDD.scala:106) at is.hail.io.RichContextRDDLong$.writeRows$extension(RichContextRDDRegionValue.scala:229) at is.hail.rvd.RVD.write(RVD.scala:765) at is.hail.expr.ir.TableNativeWriter.apply(TableWriter.scala:113) at is.hail.expr.ir.Interpret$.run(Interpret.scala:852) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:57) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:20) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:67) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:72) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:69) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14) at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:64) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:15) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:13) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:13) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:47) at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:381) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$2(SparkBackend.scala:417) at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$1(SparkBackend.scala:414) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeEncode(SparkBackend.scala:413) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) Hail version: 0.2.78-b17627756568 Error summary: SparkException: Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-00-00-00-00.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-00-00-00-00.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal