# VCFParseError on write MatrixTable

**URL:** <https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161>\
**Category:** Hail Query & hailctl\
**Created:** [November 13, 2019, 4:12pm UTC](https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161 "2019-11-13T16:12:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![carleshf](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/carleshf/32/305_2.png) [@carleshf](https://discuss.hail.is/u/carleshf)\
**Post date:** [November 13, 2019, 4:12pm UTC](https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161/1 "2019-11-13T16:12:12Z")

</div>

Hi,

I am laoding a multisaple VCF as:

```
dta = hl.import_vcf(filename, force_bgz = True, array_elements_required = True, min_partitions = 2)
mt = dta.key_rows_by('locus') \
    .distinct_by_row() \
    .key_rows_by('locus', 'alleles')

```

I have multiple variants annotated at the same loci, so I am spliting the file with:

```
x1 = hl.split_multi_hts(mt)

```

Then I am creating some annotations with transmute\_entries and annotate\_rows and I want to save the file.

But:

```
x1.write('~/annotated', overwrite = True)
[...]
FatalError: VCFParseError: missing value in FORMAT array. Import with argument 'array_elements_required=False'
Java stack trace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 53.0 failed 1 times, most recent failure: Lost task 3.0 in stage 53.0 (TID 559, localhost, executor driver): is.hail.utils.HailException: medpercan_20191011.1.red.vcf:column 12174: missing value in FORMAT array. Import with argument 'array_elements_required=False'
... 3:.:.:0,73,892,73,892,892 0/1:12,14,.:26:99:0|1:3789194_G_GCGGCCC:638,0, ...
                                        ^
offending line: 1	3789195	.	T	TGTCCCTGCCGC,C	289664	.	BaseQRankSum=0.374;Cli...
[...]
Hail version: 0.2.22-597b3bd86135
Error summary: VCFParseError: missing value in FORMAT array. Import with argument 'array_elements_required=False'

```

Any help understanding what is happening or solving the issue is welcome.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 14, 2019, 3:50am UTC](https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161/2 "2019-11-14T03:50:18Z")

</div>

By default, Hail expects that there are no missing values inside of FORMAT array fields like AD and PL. This is to enable optimizations that save on space and compute.

You do have missingness here, so import the vcf with the flag `array_elements_required=False` as the error suggests.

---

<div class="post-metadata">

**Author:** ![alanwilter](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/alanwilter/32/390_2.png) [@alanwilter](https://discuss.hail.is/u/alanwilter)\
**Post date:** [May 9, 2020, 8:36pm UTC](https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161/3 "2020-05-09T20:36:13Z")

</div>

Just because I’d like to understand better about `array_elements_required = False`. When set, what would happened specifically to the row that had failed above?  
Would it still be parsed? What would happen to the specific offending `,.:` field?

Thanks, Alan

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [May 9, 2020, 10:04pm UTC](https://discuss.hail.is/t/vcfparseerror-on-write-matrixtable/1161/4 "2020-05-09T22:04:12Z")

</div>

using the default `array_elements_required=True`, we don’t need to keep track of missingness of array elements, which leads to smaller files and faster processing. The only parse error that should be thrown is the one above, that explicitly calls out a missing value in an array that is assumed to be always-present.