# ArrayIndexOutOfBoundsException with run\_combiner

**URL:** <https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008>\
**Category:** Hail Query & hailctl\
**Created:** [April 9, 2021, 8:41pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008 "2021-04-09T20:41:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![beryl\_maze](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@beryl\_maze](https://discuss.hail.is/u/beryl_maze)\
**Post date:** [April 9, 2021, 8:41pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/1 "2021-04-09T20:41:32Z")

</div>

Wanted to try merging a couple of gVCFs to try the experimental combiner and got the error below. Konrad suggested it may be a bug?  
[combiner.log](https://discuss.hail.is/uploads/short-url/6COte4Hh0C23qFwhA6bi6OWXtCo.log) (461.8 KB)  
Attaching the log file.

> import hail as hl
> 
> hl.init(log=‘/home/hail/combiner.log’)
> 
> gvcf\_list = [“gs://maze-data-external-private/answer-als/genomics/3\_vcf/CASE-NEUCF538BRM/CASE-NEUCF538BRM-00610/CASE-NEUCF538BRM-00610-G\_1.haplotypeCalls.er.raw.vcf.gz”,  
> “gs://maze-data-external-private/answer-als/genomics/3\_vcf/CASE-NEUAE228FF6/CASE-NEUAE228FF6-02221/CASE-NEUAE228FF6-02221-G\_1.haplotypeCalls.er.raw.vcf.gz”]
> 
> output\_file = ‘gs://human-genetics-berylc/ALS/answer\_als\_hail/tmp/output.mt’ # output destination  
> temp\_bucket = ‘gs://human-genetics-berylc/ALS/answer\_als\_hail/tmp/’ # bucket for storing intermediate files  
> hl.experimental.run\_combiner(gvcf\_list, out\_file=output\_file, tmp\_path=temp\_bucket, reference\_genome=‘GRCh38’,use\_genome\_default\_intervals = True)
> 
> Running on Apache Spark version 2.4.5  
> SparkUI available at [http://answer-m.us-central1-b.c.human-genetics-001.internal:4040](http://answer-m.us-central1-b.c.human-genetics-001.internal:4040)  
> Welcome to  
> \_\_ \_\_ \<\>\_\_  
> / /_/ /\_\_ \_\_/ /  
> / \_\_ / \_ `/ / /  
> /_/ /_/\_,_/_/_/ version 0.2.64-1ef70187dc78  
> LOGGING: writing to /home/hail/combiner.log  
> 2021-04-09 20:37:07 Hail: INFO: Using 2586 intervals with default whole-genome size 1200000 as partitioning for GVCF import  
> 2021-04-09 20:37:07 Hail: INFO: GVCF combiner plan:  
> Branch factor: 100  
> Batch size: 100  
> Combining 2 input files in 1 phases with 1 total jobs.  
> Phase 1: 1 job corresponding to 1 final output file.
> 
> ## 2021-04-09 20:37:07 Hail: INFO: Starting phase 1/1, merging 2 input GVCFs in 1 job. 2021-04-09 20:37:07 Hail: INFO: Starting phase 1/1, job 1/1 to create 1 merged file, corresponding to ~100.0% of total I/O.
> 
> FatalError Traceback (most recent call last)  
> in   
> 9 output\_file = ‘gs://human-genetics-berylc/ALS/answer\_als\_hail/tmp/output.mt’ # output destination  
> 10 temp\_bucket = ‘gs://human-genetics-berylc/ALS/answer\_als\_hail/tmp/’ # bucket for storing intermediate files  
> —\> 11 hl.experimental.run\_combiner(vcf\_list, out\_file=output\_file, tmp\_path=temp\_bucket, reference\_genome=‘GRCh38’,use\_genome\_default\_intervals = True)
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/experimental/vcf\_combiner/vcf\_combiner.py in run\_combiner(sample\_paths, out\_file, tmp\_path, intervals, import\_interval\_size, use\_genome\_default\_intervals, use\_exome\_default\_intervals, header, sample\_names, branch\_factor, batch\_size, target\_records, overwrite, reference\_genome, contig\_recoding, key\_by\_locus\_and\_alleles)  
> 679 if key\_by\_locus\_and\_alleles:  
> 680 final\_mt = MatrixTable(MatrixKeyRowsBy(final\_mt.\_mir, [‘locus’, ‘alleles’], is\_sorted=True))  
> → 681 final\_mt.write(out\_file, overwrite=overwrite)  
> 682 new\_files\_to\_merge = [out\_file]  
> 683 info(f"Finished phase {phase\_i}/{n\_phases}, job {job\_i}/{len(phase.jobs)}, 100% of total I/O finished.")
> 
> in write(self, output, overwrite, stage\_locally, \_codec\_spec, \_partitions)
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs)  
> 575 def wrapper(_original\_func, \*args, \*\*kwargs):  
> 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method)  
> → 577 return _original\_func(\*args, \*\*kwargs_)  
> 578  
> 579 return wrapper
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/matrixtable.py in write(self, output, overwrite, stage\_locally, \_codec\_spec, \_partitions)  
> 2526  
> 2527 writer = ir.MatrixNativeWriter(output, overwrite, stage\_locally, \_codec\_spec, \_partitions, \_partitions\_type)  
> → 2528 Env.backend().execute(ir.MatrixWrite(self.\_mir, writer))  
> 2529  
> 2530 class \_Show:
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j\_backend.py in execute(self, ir, timed)  
> 96 raise HailUserError(message\_and\_trace) from None  
> 97  
> —\> 98 raise e
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j\_backend.py in execute(self, ir, timed)  
> 72 # print(self.\_hail\_package.expr.ir.Pretty.apply(jir, True, -1))  
> 73 try:  
> —\> 74 result = json.loads(self.\_jhc.backend().executeJSON(jir))  
> 75 value = ir.typ.\_from\_json(result[‘value’])  
> 76 timings = result[‘timings’]
> 
> /usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py in **call** (self, \*args)  
> 1255 answer = self.gateway\_client.send\_command(command)  
> 1256 return\_value = get\_return\_value(  
> → 1257 answer, self.gateway\_client, self.target\_id, self.name)  
> 1258  
> 1259 for temp\_arg in temp\_args:
> 
> /opt/conda/miniconda3/lib/python3.6/site-packages/hail/backend/py4j\_backend.py in deco(\*args, \*\*kwargs)  
> 30 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’  
> 31 ‘Hail version: %s\n’  
> —\> 32 ‘Error summary: %s’ % (deepest, full, hail. **version** , deepest), error\_id) from None  
> 33 except pyspark.sql.utils.CapturedException as e:  
> 34 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’
> 
> FatalError: ArrayIndexOutOfBoundsException: null
> 
> Java stack trace:  
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 1.0 failed 20 times, most recent failure: Lost task 9.19 in stage 1.0 (TID 175, answer-w-1.us-central1-b.c.human-genetics-001.internal, executor 1): java.lang.ArrayIndexOutOfBoundsException
> 
> Driver stacktrace:  
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1892)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1880)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)  
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)  
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)  
> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1879)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)  
> at scala.Option.foreach(Option.scala:257)  
> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2113)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2062)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2051)  
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)  
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)  
> at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:990)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)  
> at org.apache.spark.rdd.RDD.collect(RDD.scala:989)  
> at is.hail.sparkextras.ContextRDD.collect(ContextRDD.scala:176)  
> at is.hail.rvd.RVD.writeRowsSplit(RVD.scala:953)  
> at is.hail.expr.ir.MatrixValue.write(MatrixValue.scala:246)  
> at is.hail.expr.ir.MatrixNativeWriter.apply(MatrixWriter.scala:62)  
> at is.hail.expr.ir.WrappedMatrixWriter.apply(MatrixWriter.scala:41)  
> at is.hail.expr.ir.Interpret$.run(Interpret.scala:819)  
> at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)  
> at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
> at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12)  
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)  
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)  
> at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12)  
> at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:28)  
> at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$\_execute(SparkBackend.scala:362)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:346)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:343)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:48)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:48)  
> at is.hail.utils.package$.using(package.scala:618)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:48)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:47)  
> at is.hail.utils.package$.using(package.scala:618)  
> at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:13)  
> at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:47)  
> at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:256)  
> at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:343)  
> at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:387)  
> at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:385)  
> at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52)  
> at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:385)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
> at py4j.Gateway.invoke(Gateway.java:282)  
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
> at py4j.commands.CallCommand.execute(CallCommand.java:79)  
> at py4j.GatewayConnection.run(GatewayConnection.java:238)  
> at java.lang.Thread.run(Thread.java:748)
> 
> java.lang.ArrayIndexOutOfBoundsException: null  
> at
> 
> Hail version: 0.2.64-1ef70187dc78  
> Error summary: ArrayIndexOutOfBoundsException: null

---

<div class="post-metadata">

**Author:** ![beryl\_maze](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@beryl\_maze](https://discuss.hail.is/u/beryl_maze)\
**Post date:** [April 12, 2021, 4:12pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/2 "2021-04-12T16:12:35Z")

</div>

Wanted to mention that just in the off chance that the failure was due to too few gVCFs, I tried this with the full dataset and got the same error. [ALS\_answer\_als\_hail\_tmp\_combiner.log](https://discuss.hail.is/uploads/short-url/bSlRslpXHF0amobPJzXGubcIbyI.log) (3.8 MB)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [April 12, 2021, 4:14pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/3 "2021-04-12T16:14:35Z")

</div>

@chrisvittal can you take this? Looks like the real error is in tabix:

```auto
2021-04-09 20:37:20 TaskSetManager: WARN: Lost task 4.0 in stage 1.0 (TID 6, answer-w-1.us-central1-b.c.human-genetics-001.internal, executor 1): java.lang.ArrayIndexOutOfBoundsException: 3366
	at is.hail.io.tabix.TabixReader$$anonfun$1.apply(TabixReader.scala:116)
	at is.hail.io.tabix.TabixReader$$anonfun$1.apply(TabixReader.scala:86)
	at is.hail.utils.package$.using(package.scala:618)
	at is.hail.io.tabix.TabixReader.<init>(TabixReader.scala:86)
	at is.hail.io.vcf.PartitionedVCFRDD.compute(LoadVCF.scala:1486)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)

```

---

<div class="post-metadata">

**Author:** ![beryl\_maze](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@beryl\_maze](https://discuss.hail.is/u/beryl_maze)\
**Post date:** [April 15, 2021, 3:50pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/4 "2021-04-15T15:50:07Z")

</div>

just wanted to bump this. cc @chrisvittal

---

<div class="post-metadata">

**Author:** ![beryl\_maze](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@beryl\_maze](https://discuss.hail.is/u/beryl_maze)\
**Post date:** [April 26, 2021, 5:08pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/5 "2021-04-26T17:08:18Z")

</div>

@tpoterba @chrisvittal any luck?

---

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [May 3, 2021, 4:33pm UTC](https://discuss.hail.is/t/arrayindexoutofboundsexception-with-run-combiner/2008/6 "2021-05-03T16:33:06Z")

</div>

From the look of it, the issue may be in the tabix files themselves. A well formed tabix file should not be able to hit the error you did. I don’t know of any tool to ensure tabix validity but this is essentially an assertion that we couldn’t parse your tabix indices.

The best solution I have is to re-tabix your gVCFs.