ArrayIndexOutOfBoundsException using _cdf_combine

Hail Query & hailctl
1

Hi,

I’m running the following:

ht_0 = hl.read_table(test_ht_path)
# Read in same ht for testing purposes
ht_1 = hl.read_table(test_ht_path)

# Join hts to get both MQRankSum annotations (created with lambda x: hl.agg.approx_cdf(x, k=10, _raw=True)
ht = ht_0.annotate(first_MQRankSum=ht_0.info.MQRankSum,
                   second_MQRankSum = ht_1[ht_0.key].info.MQRankSum)

# Combine MQRankSum cdfs
ht = ht.annotate(raw_cdf_combined = _cdf_combine(k=10, left=ht.first_MQRankSum, right=ht.second_MQRankSum))
ht.raw_cdf_combined.show()

Do you have any suggestions regarding this error message:
hail.utils.java.FatalError: ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2

Thank you!

2

Hi @klaricch,

Do you have a full stacktrace from that error? That would help us pin down the cause.

3 
Traceback (most recent call last):
  File "/tmp/715094ce17ce415c806d3cc28a2eaf4b/combine_cdfs.py", line 18, in <module>
    ht.raw_cdf_combined.show()
  File "<decorator-gen-540>", line 2, in show
  File "/opt/conda/default/lib/python3.10/site-packages/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/expr/expressions/base_expression.py", line 980, in show
    return ds.show(**{k: v for k, v in kwargs.items() if v is not None})
  File "<decorator-gen-1220>", line 2, in show
  File "/opt/conda/default/lib/python3.10/site-packages/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2373, in show
    return handler(self._show(n, width, truncate, types))
  File "/opt/conda/default/lib/python3.10/site-packages/IPython/core/display.py", line 285, in display
    print(*objs)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2154, in __str__
    return self._ascii_str()
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2180, in _ascii_str
    rows, has_more, dtype = self.data()
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2164, in data
    rows, has_more = t._take_n(self.n)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2310, in _take_n
    rows = self.take(n + 1)
  File "<decorator-gen-1232>", line 2, in take
  File "/opt/conda/default/lib/python3.10/site-packages/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 3027, in take
    return self.head(n).collect(_localize)
  File "<decorator-gen-1226>", line 2, in collect
  File "/opt/conda/default/lib/python3.10/site-packages/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/table.py", line 2814, in collect
    return Env.backend().execute(e._ir, timed=_timed)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/backend/spark_backend.py", line 226, in execute
    raise err
  File "/opt/conda/default/lib/python3.10/site-packages/hail/backend/spark_backend.py", line 218, in execute
    return super().execute(ir, timed)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/backend/backend.py", line 190, in execute
    raise e.maybe_user_error(ir) from None
  File "/opt/conda/default/lib/python3.10/site-packages/hail/backend/backend.py", line 188, in execute
    result, timings = self._rpc(ActionTag.EXECUTE, payload)
  File "/opt/conda/default/lib/python3.10/site-packages/hail/backend/py4j_backend.py", line 221, in _rpc
    raise fatal_error_from_java_error_triplet(
hail.utils.java.FatalError: ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2

Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 20 times, most recent failure: Lost task 1.19 in stage 0.0 (TID 37) (kmlt-sw-b9rj.c.broad-mpg-gnomad.internal executor 2): java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
	at is.hail.expr.ir.agg.ApproxCDFStateManager.findFullLevel(ApproxCDFStateManager.scala:674)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.compact(ApproxCDFStateManager.scala:687)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.seqOp(ApproxCDFStateManager.scala:609)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.combOp(ApproxCDFStateManager.scala:620)
	at __C49collect_distributed_array_table_head_recursive_count.__m111approxCDFCombine(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.__m60split_StreamLen(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.apply(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.apply(Unknown Source)
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$6(BackendUtils.scala:87)
	at is.hail.utils.package$.using(package.scala:664)
	at is.hail.annotations.RegionPool.scopedRegion(RegionPool.scala:166)
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$5(BackendUtils.scala:86)
	at is.hail.backend.spark.SparkBackendComputeRDD.compute(SparkBackend.scala:910)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2717)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2653)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2652)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2652)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1189)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1189)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1189)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2913)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2855)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2844)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:959)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2282)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2301)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2326)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1021)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1020)
	at is.hail.backend.spark.SparkBackend.parallelizeAndComputeWithIndex(SparkBackend.scala:429)
	at is.hail.backend.BackendUtils.collectDArray(BackendUtils.scala:82)
	at __C5Compiled.__m32split_TailLoop(Emit.scala)
	at __C5Compiled.__m7split_ToArray(Emit.scala)
	at __C5Compiled.apply(Emit.scala)
	at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$_apply$7(CompileAndEvaluate.scala:82)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:82)
	at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$apply$1(CompileAndEvaluate.scala:17)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:17)
	at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1486)
	at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1050)
	at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1654)
	at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:728)
	at is.hail.expr.ir.lowering.LowerTableIR$.apply(LowerTableIR.scala:823)
	at is.hail.expr.ir.lowering.LowerToCDA$.lower(LowerToCDA.scala:27)
	at is.hail.expr.ir.lowering.LowerToCDA$.apply(LowerToCDA.scala:11)
	at is.hail.expr.ir.lowering.LowerToDistributedArrayPass.transform(LoweringPass.scala:91)
	at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:27)
	at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:59)
	at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:64)
	at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:83)
	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:32)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:32)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:30)
	at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:29)
	at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:78)
	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:21)
	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:19)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:19)
	at is.hail.expr.ir.lowering.EvalRelationalLets$.execute$1(EvalRelationalLets.scala:13)
	at is.hail.expr.ir.lowering.EvalRelationalLets$.lower$1(EvalRelationalLets.scala:21)
	at is.hail.expr.ir.lowering.EvalRelationalLets$.apply(EvalRelationalLets.scala:35)
	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.transform(LoweringPass.scala:168)
	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:32)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:32)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:30)
	at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:29)
	at is.hail.expr.ir.lowering.EvalRelationalLetsPass.apply(LoweringPass.scala:162)
	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:21)
	at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:19)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:19)
	at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:45)
	at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:600)
	at is.hail.backend.spark.SparkBackend.$anonfun$execute$4(SparkBackend.scala:636)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:84)
	at is.hail.backend.spark.SparkBackend.$anonfun$execute$3(SparkBackend.scala:631)
	at is.hail.backend.spark.SparkBackend.$anonfun$execute$3$adapted(SparkBackend.scala:630)
	at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:78)
	at is.hail.utils.package$.using(package.scala:664)
	at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:78)
	at is.hail.utils.package$.using(package.scala:664)
	at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:13)
	at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:65)
	at is.hail.backend.spark.SparkBackend.$anonfun$withExecuteContext$2(SparkBackend.scala:407)
	at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:55)
	at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:62)
	at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:393)
	at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:630)
	at is.hail.backend.BackendHttpHandler.handle(BackendServer.scala:88)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:560)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:526)
	at java.base/java.lang.Thread.run(Thread.java:829)

java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
	at is.hail.expr.ir.agg.ApproxCDFStateManager.findFullLevel(ApproxCDFStateManager.scala:674)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.compact(ApproxCDFStateManager.scala:687)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.seqOp(ApproxCDFStateManager.scala:609)
	at is.hail.expr.ir.agg.ApproxCDFStateManager.combOp(ApproxCDFStateManager.scala:620)
	at __C49collect_distributed_array_table_head_recursive_count.__m111approxCDFCombine(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.__m60split_StreamLen(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.apply(Unknown Source)
	at __C49collect_distributed_array_table_head_recursive_count.apply(Unknown Source)
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$6(BackendUtils.scala:87)
	at is.hail.utils.package$.using(package.scala:664)
	at is.hail.annotations.RegionPool.scopedRegion(RegionPool.scala:166)
	at is.hail.backend.BackendUtils.$anonfun$collectDArray$5(BackendUtils.scala:86)
	at is.hail.backend.spark.SparkBackendComputeRDD.compute(SparkBackend.scala:910)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)




Hail version: 0.2.130-bea04d9c79b5
Error summary: ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
4

Thanks! This is definitely a Hail bug. Is the table at test_ht_path something you could share with us, by any chance? That would be a huge help in fixing the bug. If so, you can email us at hail@broadinstitute.org to coordinate sharing the data. If not, I might ask you some more questions to figure out what your data looks like, so we can try to reproduce the error.

5

Never mind, I’ve been able to reproduce the bug. I expect this to be fixed in our next release, but I’ll also try to find a workaround you can use in the meantime. I’ll keep you posted!

6

OK, thank you!

7

I think I have a working fix, but a workaround looks like it might be tricky. Are you able to wait for the next release (probably early January at the latest) for the fix? If not, I can build a custom wheel for you to install with the fix included.

8

Yeah waiting for the next release is fine