VEP for deletions(SV) is not working

Hello,
I’m trying to annotate a matrix table with VEP. The table was from Manta VCF file. When I annotated deletions(SV) of which alleles was like ['ATGCATGC...ATGC', 'A'], VEP was working well. The problem was the deletions that has alleles like ['A', '<DEL>'] was not annotated with VEP, therefore VEP results was NA. But when I ran VEP without hail, strangely, there was no problem annotating that deletions.

I use hail 0.2.74, python 3.7.10, spark 3.1.1, VEP ver104.

Here is my vep-config.json. I use csq=True option because of StructuralVariantOverlap Plugin parsing error.

{
    "command": [
        "/home/a7420174/tools/ensembl-vep/vep",
        "--format", "vcf",
        "__OUTPUT_FORMAT_FLAG__",
        "--everything",
        "--no_stats", "--overlaps",
        "--cache", "--offline",
        "--assembly", "GRCh38",
        "--per_gene", "--pick", "--pick_order", "canonical,appris,tsl,biotype,ccds,rank,length",
        "--plugin", "StructuralVariantOverlap,file=/home/a7420174/WGS/ASD_WGS_SV/resources/nstd166.GRCh38.variant_call.vcf.gz,reciprocal=1,same_type=1",
        "--nearest", "gene",
        "-o", "STDOUT"
    ],
    "env": {
        "PERL5LIB": "/home/a7420174/perl5/lib/perl5:$PERL5LIB"
    },
    "vep_json_schema": "Struct{assembly_name:String,allele_string:String,ancestral:String,custom_annotations:Struct{gnomADg:Array[Struct{fields:Struct{AF:Float64,FILTER:String},allele:String,name:String}]},colocated_variants:Array[Struct{AA:String,AFR:String,AMR:String,EA:String,EAS:String,EUR:String,SAS:String,aa_allele:String,aa_maf:Float64,afr_allele:String,afr_maf:Float64,allele_string:String,amr_allele:String,amr_maf:Float64,clin_sig:Array[String],clin_sig_allele:String,end:Int32,eas_allele:String,eas_maf:Float64,ea_allele:String,ea_maf:Float64,eur_allele:String,eur_maf:Float64,exac_adj_allele:String,exac_adj_maf:Float64,exac_allele:String,exac_afr_allele:String,exac_afr_maf:Float64,exac_amr_allele:String,exac_amr_maf:Float64,exac_eas_allele:String,exac_eas_maf:Float64,exac_fin_allele:String,exac_fin_maf:Float64,exac_maf:Float64,exac_nfe_allele:String,exac_nfe_maf:Float64,exac_oth_allele:String,exac_oth_maf:Float64,exac_sas_allele:String,exac_sas_maf:Float64,gnomAD:String,gnomAD_AFR:String,gnomAD_AMR:String,gnomAD_ASJ:String,gnomAD_EAS:String,gnomAD_FIN:String,gnomAD_NFE:String,gnomAD_OTH:String,gnomAD_SAS:String,id:String,frequencies:Struct{G:Struct{eur:Float64,eas:Float64,amr:Float64,afr:Float64,sas:Float64}},minor_allele:String,minor_allele_freq:Float64,phenotype_or_disease:Int32,pubmed:Array[Int32],sas_allele:String,sas_maf:Float64,seq_region_name:String,somatic:Int32,start:Int32,strand:Int32,var_synonyms:String}],context:String,end:Int32,id:String,input:String,intergenic_consequences:Array[Struct{allele_num:Int32,cadd_raw:Float64,cadd_phred:Float64,consequence_terms:Array[String],impact:String,minimised:Int32,variant_allele:String,sv_overlap_name:String,bp_overlap:Int32,percentage_overlap:Float64,sv_overlap_pc:Int32,sv_overlap_af:Float64}],most_severe_consequence:String,motif_feature_consequences:Array[Struct{allele_num:Int32,cadd_phred:Float64,cadd_raw:Float64,consequence_terms:Array[String],high_inf_pos:String,impact:String,minimised:Int32,motif_feature_id:String,motif_name:String,motif_pos:Int32,motif_score_change:Float64,strand:Int32,transcription_factors:Array[String],variant_allele:String,sv_overlap_name:String,bp_overlap:Int32,percentage_overlap:Float64,sv_overlap_pc:Int32,sv_overlap_af:Float64}],regulatory_feature_consequences:Array[Struct{allele_num:Int32,biotype:String,cadd_raw:Float64,cadd_phred:Float64,consequence_terms:Array[String],impact:String,minimised:Int32,regulatory_feature_id:String,variant_allele:String,sv_overlap_name:String,bp_overlap:Int32,percentage_overlap:Float64,sv_overlap_pc:Int32,sv_overlap_af:Float64}],seq_region_name:String,nearest:Array[String],start:Int32,strand:Int32,transcript_consequences:Array[Struct{allele_num:Int32,amino_acids:String,appris:String,biotype:String,cadd_phred:Float64,cadd_raw:Float64,canonical:Int32,ccds:String,cdna_start:Int32,cdna_end:Int32,cds_end:Int32,cds_start:Int32,codons:String,consequence_terms:Array[String],distance:Int32,domains:Array[Struct{db:String,name:String}],exon:String,flags:Array[String],cds_end_NF:Array[String],gene_id:String,gene_pheno:Int32,gene_symbol:String,gene_symbol_source:String,hgnc_id:String,hgvsg:String,hgvsc:String,hgvsp:String,hgvs_offset:Int32,impact:String,intron:String,lof:String,lof_flags:String,lof_filter:String,lof_info:String,mane:String,minimised:Int32,polyphen_prediction:String,polyphen_score:Float64,protein_end:Int32,protein_start:Int32,protein_id:String,sift_prediction:String,sift_score:Float64,strand:Int32,swissprot:Array[String],transcript_id:String,trembl:Array[String],tsl:Int32,uniparc:Array[String],uniprot_isoform:Array[String],mane_select:String,variant_allele:String,sv_overlap_name:String,bp_overlap:Int32,percentage_overlap:Float64,sv_overlap_pc:Int32,sv_overlap_af:Float64}],variant_class:String}"
}

Is there any suggestion you can offer?

Thanks

Hi @a7420174 , unfortunately it is difficult for us to diagnose custom VEP configurations, and we don’t use the StructuralVariantOverlap plugin ourselves by default. If you can reproduce this problem on hail dataproc vep with a minimal example (1 line VCF), then we can more likely work through it. You might find this other debugging VEP post useful, and otherwise check the cache directories for any abnormalities. If none of the above is possible, we would suggest running VEP separately. Sorry we could not be more helpful in this instance, we know how frustrating it can be to debug issues when it comes to VEP!

Umm… Unfortunately, I have never used hail dataproc vep before. But I can show you hail log file in local-mode. Also, even if I remove StructuralVariantOverlap plugin and run VEP, the deletions are not annotated. Can you please check this log file lastly?
hail_20210820.log (218.2 KB)

Thanks in advance!