# Import existing VEP annotations from vcf or CSQ **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [October 9, 2019, 12:31pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117 "2019-10-09T12:31:33Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [October 9, 2019, 12:31pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/1 "2019-10-09T12:31:33Z")
Hello, I am pretty new to hail, so please excuse if I might ask some basic questions. I would like to use existing CSQ annotations from a vcf which have been created before importing it to hail. Is there some script to smoothly extract the CSQ information in a row field, as hl.vep would do? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [October 9, 2019, 12:32pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/2 "2019-10-09T12:32:47Z")
> [@johnnyr](#): > > Is there some script to smoothly extract the CSQ information in a row field, as hl.vep would do? I don’t know of one. Dealing with the VCF CSQ field is a nightmare. It’s probably worth it to re-vep in Hail, if you’re running on the cloud. ---
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [October 9, 2019, 12:44pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/3 "2019-10-09T12:44:43Z")
Thanks for the instant reply. Okay, will keep trying. So far I had some issues running it. It starts to create the respective row fields, but then it stops before adding any annotation. The last line of the log is: > 2019-10-09 14:23:26 root: INFO: is/hail/codegen/generated/C\_etypeEncode\_78.ENCODE\_o\_tuple\_of\_END\_TO\_o\_struct\_of\_END\_1 instruction count: 7 any take on this ? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [October 9, 2019, 12:46pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/4 "2019-10-09T12:46:06Z")
what is your pipeline? ---
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [October 9, 2019, 3:35pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/5 "2019-10-09T15:35:53Z")
Just figured out that customized VEP runs (local) are probably an issue. Probably due to configuration of the vep\_json\_schema as I suppose. Maybe a solution is to import annotation (–tab output from VEP) as extra table and use the variant identifier like chrom\_pos\_ref\_alt for merging. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 10, 2019, 5:30pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/6 "2019-10-10T17:30:37Z")
Are you using Hail on GCP Dataproc? The `hailctl` utility handles all VEP configuration for you. ---
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [October 18, 2019, 2:43pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/7 "2019-10-18T14:43:29Z")
No, I want to (have to) use it locally. I’ve now imported output from local vep as table along with an import of the vcf. The tab output from vep only provides variant annotation but no genotypes, so this is why I have to also import the vcf. hl.import\_vcf(‘VEP.vcf’,reference\_genome=‘hg38’).write(‘data/test.mt’, overwrite=True) mt = hl.read\_matrix\_table(‘data/test.mt’) mtanno=hl.import\_table(‘VEP.txt’) mtanno = mtanno.key\_by(‘Uploaded\_variation’) Now I need to create a shared identifier from “mt” to merge with “mtanno” key, which looks like this: “1\_100\_C/G” “1\_1000\_AA/-” Any suggestions on how to do this ? Thanx ! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 18, 2019, 2:48pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/8 "2019-10-18T14:48:43Z")
That doesn’t look like a min-repped, left aligned variant, which will create problems for joining against your dataset, if your dataset is min-repped and left aligned. What does `mt.filter_rows(mt.locus == hl.locus("1", 1000)).alleles.show()` print? Are you sure the VEP output doesn’t have a standard variant format like `1:100:A:T`? We have parsers for that. If not, you’ll need to hack something yourself with [`split`](https://hail.is/docs/0.2/hail.expr.StringExpression.html?highlight=split#hail.expr.StringExpression.split). ---
**Author:** ![toandd](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/toandd/32/337_2.png) [@toandd](https://discuss.hail.is/u/toandd)\ **Post date:** [November 19, 2019, 2:53am UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/9 "2019-11-19T02:53:51Z")
Same my problem. Did you find out some solutions @johnnyr? ---
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [November 20, 2019, 2:14pm UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/10 "2019-11-20T14:14:52Z")
Unfortunately I had no time to work on it recently, but will look at it again. Will update you once I have found a solution. If you find a good one meanwhile, let me know. ---
**Author:** ![johnnyr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnnyr/32/572_2.png) [@johnnyr](https://discuss.hail.is/u/johnnyr)\ **Post date:** [November 27, 2019, 8:35am UTC](https://discuss.hail.is/t/import-existing-vep-annotations-from-vcf-or-csq/1117/11 "2019-11-27T08:35:13Z")
Okay, here is what I can come up with so far. When performing GATK “LeftAlignAndTrimVariants” I include the option “–split-multi-allelics” to get one line each per alternate allele of a variant. Next, I have annotated a vcf using ensemble-VEP including “–pick” flag to restrict annotations to only the major consequence. Of course this is a critical point and one might consider alternative prefiltering. I run VEP twice to get a vcf as output “–vcf” and with “–tab” (.txt) option again, to also get VEP annotations in a tabular format, inluding a column named “Uploaded\_variation”. This column serves as key for annotation in hail. To get the very same key in hail for the vcf, I used a workaround in R with VariantAnnotation tool, to add this key as a seperate entry in the INFO field of the vcf. > #!/usr/bin/env Rscript > library(VariantAnnotation) > samples\<-c(“ID1”,“ID2”) > param\<-ScanVcfParam(geno=c(“GT”), info=c(“CSQ”), samples = samples) > vcf ← readVcf(“path to VEP annotated vcf”, “GRCh38”, param) > VARID ← DataFrame(Number=1, Type=“String”, > Description=“VARID for annotation in hail”, > row.names=“VARID”) > info(header(vcf)) ← rbind(info(header(vcf)), VARID) > info(vcf)$VARID\<-gsub(“\:”,“\_”,as.matrix(rownames(vcf))) > writeVcf(vcf,“path to VEP annotated vcf with VARID annotation in info field”,index=F) Next, I import both into hail: > hl.import\_vcf(‘path to VEP annotated vcf with VARID annotation in info field’,reference\_genome=‘hg38’).write(‘data/test.mt’, overwrite=True) > mt = hl.read\_matrix\_table(‘data/test.mt’) > mt = mt.key\_rows\_by(mt.info.VARID) > mtanno=hl.import\_table(‘VEP.txt’) > mtanno = mtanno.key\_by(‘Uploaded\_variation’) and finally, I am able to annotate the matrix table in hail with my annotation table: > mt = mt.annotate\_rows(VEP = mtanno[mt.info.VARID]) When performing this for the full “tab” output from VEP, I had some problems, probably due to incorrect field types. So I have now tested only with one annotation column (String), which works nicely. I haven’t performed in depth quality controls if all annotations got right, so treat with care. Probably there are more elegant solutions in hail, which I would love to test, also. So any comments are welcome.