# Export\_vcf with VEP **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [May 12, 2020, 7:53pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409 "2020-05-12T19:53:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![alanwilter](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/alanwilter/32/390_2.png) [@alanwilter](https://discuss.hail.is/u/alanwilter)\ **Post date:** [May 12, 2020, 7:53pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/1 "2020-05-12T19:53:53Z")
Hi, so I managed to run `vep` on my original vcf.bgz file, save a mt\_vep for analysis but I’d like to save a vep.vcf.bgz file as well. So I tried this first: ```auto mt_vep = hl.read_matrix_table('gs://...vep.mt') meta_orig=hl.get_vcf_metadata('gs://...orig.vcf.bgz') hl.export_vcf(mt_vep, 'gs://...vep.vcf.bgz', parallel='header_per_shard', metadata=meta_orig) ``` Then I saw this warning: ```auto 2020-05-12 19:45:06 Hail: WARN: export_vcf: ignored the following fields: 'vep' (row) ``` I obviously missed something in [https://hail.is/docs/0.2/methods/impex.html#hail.methods.export\_vcf](https://hail.is/docs/0.2/methods/impex.html#hail.methods.export_vcf), but what exactly please? Thanks, Alan ---
**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\ **Post date:** [May 12, 2020, 8:05pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/2 "2020-05-12T20:05:23Z")
So I think the relevant bit from the docs is: > Hail exports the fields of struct info as INFO fields, the elements of `set` filters as FILTERS, the value of str rsid as ID, and the value of float64 qual as QUAL. No other row fields are exported. I’m more on the engineering side than genetics side, so I don’t really use VEP, but I think the answer here is that anything you want to export from VEP has to go into a row annotation called `info`. ---
**Author:** ![alanwilter](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/alanwilter/32/390_2.png) [@alanwilter](https://discuss.hail.is/u/alanwilter)\ **Post date:** [May 12, 2020, 10:09pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/3 "2020-05-12T22:09:18Z")
Thanks @johnc1231. I found this as well: [https://discuss.hail.is/t/vcf-ignoring-vep-and-non-reference-samples-while-exporting](https://discuss.hail.is/t/vcf-ignoring-vep-and-non-reference-samples-while-exporting) I can live with that. My main target is to get the data in a way I can query, like in Amazon Athena. ---
**Author:** ![TomK](https://avatars.discourse-cdn.com/v4/letter/t/ee59a6/32.png) [@TomK](https://discuss.hail.is/u/TomK)\ **Post date:** [May 12, 2021, 7:47pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/4 "2021-05-12T19:47:22Z")
Hi @alanwilter, did you find a solution to that? Thanks, Tom ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 12, 2021, 7:49pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/5 "2021-05-12T19:49:30Z")
If you’re trying to export the VEP CSQ field in VCF info, you can use `csq=True` on `hl.vep` ---
**Author:** ![TomK](https://avatars.discourse-cdn.com/v4/letter/t/ee59a6/32.png) [@TomK](https://discuss.hail.is/u/TomK)\ **Post date:** [May 12, 2021, 8:38pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/6 "2021-05-12T20:38:04Z")
Hi @tpoterba, Thank you for your answer. So my steps are: ``` >>> import hail as hl >>> hl.init(default_reference='GRCh38') ... >>> ds = hl.import_vcf(file, force_bgz=True, contig_recoding=contig_recoding, skip_invalid_loci=True, min_partitions=500) ... >>> ds = hl.vep(ds, vep_config, csq=True) >>> ds = ds.annotate_rows(info=ds.info.annotate(vep=ds.vep)) >>> hl.export_vcf(ds, 'clinvar_20210424_GRCh38_vep.vcf.bgz') ``` I’m getting: is.hail.utils.HailException: VEP returned invalid variant Any idea on how to debug that? Thanks, Tom ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 12, 2021, 8:52pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/7 "2021-05-12T20:52:01Z")
What’s the full error message? ---
**Author:** ![TomK](https://avatars.discourse-cdn.com/v4/letter/t/ee59a6/32.png) [@TomK](https://discuss.hail.is/u/TomK)\ **Post date:** [May 12, 2021, 9:05pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/8 "2021-05-12T21:05:33Z")
It’s pretty long I’m not sure I should paste it on here. Basically I’m getting a java.lang.ArrayIndexOutOfBoundsException: 1 My understanding is that with csq=True the vep field should be of type tstr, but I have this: ``` >>> ds.head(1).describe() Row fields: 'locus': locus 'alleles': array 'rsid': str 'qual': float64 'filters': set 'info': struct { ... } **'vep': array** 'vep_proc_id': struct { part_idx: int32, block_idx: int32 } ``` vep is of type array\ instead. Let me know if you want me to paste the full output on here. Thanks, Tom – EDIT – I believe this is due to my vep\_config, using --json instead of --vcf when csq=True. Thanks, Tom ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 12, 2021, 11:45pm UTC](https://discuss.hail.is/t/export-vcf-with-vep/1409/9 "2021-05-12T23:45:13Z")
> [@TomK](#): > > It’s pretty long I’m not sure I should paste it on here. We need to see this, I think. Pasting the java stack trace (don’t care about the python one) here or in a github gist are good options. The `array` field exports just fine to VCF, I believe.