# VCF4.1, is.hail.utils.HailException: array index out of bounds

**URL:** <https://discuss.hail.is/t/vcf4-1-is-hail-utils-hailexception-array-index-out-of-bounds/1232>\
**Category:** Hail Query & hailctl\
**Created:** [December 30, 2019, 5:25pm UTC](https://discuss.hail.is/t/vcf4-1-is-hail-utils-hailexception-array-index-out-of-bounds/1232 "2019-12-30T17:25:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\
**Post date:** [December 30, 2019, 5:25pm UTC](https://discuss.hail.is/t/vcf4-1-is-hail-utils-hailexception-array-index-out-of-bounds/1232/1 "2019-12-30T17:25:12Z")

</div>

I am running

> mt.write(self.output().path, stage\_locally=True, overwrite=True)

line 63 here: [https://github.com/macarthur-lab/hail-elasticsearch-pipelines/blob/master/luigi\_pipeline/seqr\_loading.py](https://github.com/macarthur-lab/hail-elasticsearch-pipelines/blob/master/luigi_pipeline/seqr_loading.py)

And getting the error:

> hail.utils.java.FatalError: HailException: array index out of bounds: 1 / 1. IR: (ArrayRef  
> (Ref \_\_iruid\_5324)  
> (If  
> (ApplyComparisonOp LT  
> (Ref \_\_iruid\_5325)  
> (I32 0 …
> 
> Java stack trace:  
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 16 in stage 7.0 failed 4 times, most recent failure: Lost task 16.3 in stage 7.0 (TID 3028, 137.187.60.61, executor 2): is.hail.utils.HailException: array index out of bounds: 1 / 1. IR: (ArrayRef  
> (Ref \_\_iruid\_5324)  
> (If  
> (ApplyComparisonOp LT  
> (Ref \_\_iruid\_5325)  
> (I32 0 …  
> at is.hail.codegen.generated.C51.method\_4(Unknown Source)  
> at is.hail.codegen.generated.C51.method\_3(Unknown Source)  
> at is.hail.codegen.generated.C51.method\_2(Unknown Source)  
> at is.hail.codegen.generated.C51.apply(Unknown Source)  
> at is.hail.codegen.generated.C51.apply(Unknown Source)  
> at is.hail.expr.ir.TableMapRows$$anonfun$55$$anonfun$apply$19.apply(TableIR.scala:1123)  
> at is.hail.expr.ir.TableMapRows$$anonfun$55$$anonfun$apply$19.apply(TableIR.scala:1122)  
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)  
> at scala.collection.Iterator$$anon$12.next(Iterator.scala:445)  
> at is.hail.io.RichContextRDDRegionValue$$anonfun$boundary$extension$1$$anon$1.next(RichContextRDDRegionValue.scala:193)

It seems similar to the error:

> <https://github.com/hail-is/hail/issues/5832>
>
> replicate:
> 
> \`\`\`
> hl.utils.range\_matrix\_table(1, 1).annotate\_entries(x=5).show(…)
> \`\`\`
> 
> \`\`\`
> E is.hail.utils.HailException: array index out of bounds: 1 / 1. IR: (ArrayRef
> E (GetField entries
> E (In Struct{row\_idx:Int32,entries:Array\[Struct{x:Int32}\]} 1))
> E (I3 ...
> \`\`\`
> 
> The fix should also fix this problem:
> 
> \`\`\`
> In \[2\]: hl.utils.range\_matrix\_table(1, 1).show()
> +---------+
> | row\_idx |
> +---------+
> | int32 |
> +---------+
> | 0 |
> +---------+
> showing first 43 columns
> \`\`\`

It happens with some of our old **.vcf** that we ran using Hail 0.1 successfully a year ago, but works perfectly with new ones. Old one is **VCF4.1** , but new one is **VCF4.2**.

Do you know what could be done to be able to run it still?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 2, 2020, 11:43am UTC](https://discuss.hail.is/t/vcf4-1-is-hail-utils-hailexception-array-index-out-of-bounds/1232/2 "2020-01-02T11:43:47Z")

</div>

One of the most confusing pieces of Hail for newer users is the fact that most operations in Hail are lazy, and only executed at the end of a pipeline.

For instance, if you run the following pipeline:

```python
mt = hl.read_matrix_table(...) # 1
mt = hl.variant_qc(mt) # 2
mt.write(...) # 3

```

You’ll notice that #1 and #2 are both very fast – they’re not actually doing anything, just recording the a sequence of transformations that comprises the `mt` variable that we’ll write in #3. #3 (the write) triggers a computation that executes the read, qc computations, and write.

This array index out of bounds error is coming from something earlier in the pipeline – `mt.write` can’t cause this error. My guess is you have somewhere a `thing[1]` in your script, and `thing` has only one element. Maybe the problem is with things like AC/AF in INFO fields, where we often see length confusion?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 2, 2020, 2:43pm UTC](https://discuss.hail.is/t/vcf4-1-is-hail-utils-hailexception-array-index-out-of-bounds/1232/3 "2020-01-02T14:43:12Z")

</div>

Note that I expect 0.2.31 to include the Python stack trace for the invalid index:

> <https://github.com/hail-is/hail/pull/7792>