Error index out of bounds

Hail Query & hailctl
1

I tried to run this pipeline and get the error of index out of bounds
data=hl.import_vcf(myFolder+"/"+myFile,force_bgz=True,reference_genome=‘GRCh38’,array_elements_required=False)
data2=hl.split_multi_hts(data, permit_shuffle=True)
data2=data2.cache()
mycount=data2.count()[0]

File “/usr/local/lib/python3.7/site-packages/hail/matrixtable.py”, line 2426, in count
return Env.backend().execute(count_ir)
File “/usr/local/lib/python3.7/site-packages/hail/backend/spark_backend.py”, line 297, in execute
result = json.loads(self._jhc.backend().executeJSON(jir))
File “/usr/local/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File “/usr/local/lib/python3.7/site-packages/hail/backend/spark_backend.py”, line 42, in deco
‘Error summary: %s’ % (deepest, full, hail.version, deepest)) from None
hail.utils.java.FatalError: HailException: array index out of bounds: index=2, length=2

Python traceback:
File “indexingES.py”, line 43, in
main(sys.argv[1:])

File “indexingES.py”, line 19, in main
data2=hl.split_multi_hts(data, permit_shuffle=True)

File “”, line 2, in split_multi_hts

File “/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py”, line 1786, in split_multi_hts
(hl.range(0, 3).map(lambda i:

File “”, line 2, in map

File “/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py”, line 1790, in
).map(lambda j: split.PL[j]))))))

File “”, line 2, in map

File “/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py”, line 1790, in
).map(lambda j: split.PL[j]))))))

Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 50 in stage 4.0 failed 1 times, most recent failure: Lost task 50.0 in stage 4.0 (TID 4142, localhost, executor driver): is.hail.utils.HailException: array index out of bounds: index=2, length=2

2

You’ve got input that isn’t what Hail expects – in particular, a PL field with only 2 elements.

Do you have haploid calls? I think that would break hl.split_multi_hts.

3

In other file I get this error related with PL:
Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 65 in stage 4.0 failed 1 times, most recent failure: Lost task 65.0 in stage 4.0 (TID 4405, localhost, executor driver): is.hail.utils.HailException: PL cannot have missing elements.

4

Ok. So, what can I do?. Really I am only interested in the GT field of the entries. If I select the GT, and then drop the others, this are going to work?.

5

I think that should work, yes.

6

Thank you!!. Yes, selecting only GT works. Maybe would be better if the exception outputs also the entry data of the error.

7

@tpoterba I am having a similar issue to this. However, I would like to keep all fields if possible? Do you have a possible workaround for this issue? Thank you!

Error as follows:

HailUserError: Error summary: HailException: array index out of bounds: index=3, length=3

Hail stack trace:
File “”, line 5, in
mt = hl.split_multi_hts(mt)

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/methods/statgen.py”, line 2322, in split_multi_hts
(hl.range(0, 3).map(lambda i:

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/methods/statgen.py”, line 2326, in
).map(lambda j: split.PL[j]))))))

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/methods/statgen.py”, line 2326, in
).map(lambda j: split.PL[j]))))))

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/expr/expressions/typed_expressions.py”, line 481, in getitem
return self._method(“indexArray”, self.dtype.element_type, item)

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/expr/expressions/base_expression.py”, line 596, in _method
x = ir.Apply(name, ret_type, self._ir, *(a._ir for a in args))

File “/apps/genomics/hail/0.2/el7/AVX512/gnu-7.3/hail/ir/ir.py”, line 2138, in init
self.save_error_info()