# Redirect (or find) VEP (or other) error output from a Hail pipeline **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [February 24, 2020, 4:38pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308 "2020-02-24T16:38:02Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 24, 2020, 4:38pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/1 "2020-02-24T16:38:02Z")
I run a **Hail** pipeline and get a **VEP** error but there is no error output from the **VEP** tool itself, **Hail** just says that **VEP** failed and the **code is 2**. Where is the real **VEP** error output written, which file? And if there is none, is there a way to figure this out somehow or redirect **VEP** output to a new file? The error happens when **mt.write()** triggers **VEP** execution in the pipeline: > The question is related to another one that I asked here: > [@No VEP debug output](https://discuss.hail.is/t/no-vep-debug-output/1302/6): > > I verified and VEP is working perfectly when called in a standalone mode. However, running it from Hail gives VEP error code 2 somehow. No debug output pretty much and I am not sure where to look further. All permissions on all of the relevant folders and files I set to 777: hadoop files permissions, VEP, loftee. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 24, 2020, 4:50pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/2 "2020-02-24T16:50:31Z")
Is the error in the hail log file? The location of the hail log file is printed when Hail starts. ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 24, 2020, 5:00pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/3 "2020-02-24T17:00:29Z")
No, it’s not, and that’s a thing. I need to see **VEP** error output while running **Hail** pipeline and it is not there. I can run **VEP** separately and it has some very informative output in case of an error, but when run by **Hail** pipeline nothing is seen except for _error code 2_ and that **VEP** failed. Its either the problem of **Hail** itself or the **luigi** pipeline that I am running and I do not know which one but people who developed the pipeline are silent: > > > I run successfully VEP locally, the very same command that fails from within my current Hail 0.2 pipeline: > /vep/ensembl-tools-release-99/vep -i batch1.vcf --format... It all comes down to finding a way of debugging. The actual issue specified in the title of the **github** is not that relevant. The **VEP** output gets completely lost and that is the main issue here. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 24, 2020, 5:16pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/4 "2020-02-24T17:16:10Z")
What version of Hail are you using? The Hail error message should look something like this: ```auto VEP command '/path/to/vep arg1 arg2 arg3' failed with non-zero exit status 0 VEP Error output: The complete error message from VEP should appear here. ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 24, 2020, 5:20pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/5 "2020-02-24T17:20:03Z")
Just read the other issue. That means that VEP isn’t printing any output when it errors. The next thing I would try is starting from a VEP command that works and iteratively changing options until you find the one that causes VEP to fail. Does the VEP configuration in the docs for `hl.vep` work for you? ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 24, 2020, 5:32pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/6 "2020-02-24T17:32:02Z")
Because I am converting the pipeline now to **grch38** , there is no pipeline that actually works. Yeah, so under **VEP Error output:** there is an empty line and nothing shown as if the output was somehow lost. Yeah, the very same configuration of **VEP** does work when I run it locally in a standalone mode. So, I just copy-paste the command, add **-i** parameter to specify the input file, and then it produces correct annotated file. **The version of Hail is 0.2** Here I am also attaching the screenshot for how it looks in my case: ![Screen Shot 2020-02-24 at 1.30.46 PM](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/b361a2b57c8c10a55d842dfc01709901038564ce.png) So, as you can see its empty. Java stack trace is pretty useless. ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 24, 2020, 6:57pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/7 "2020-02-24T18:57:42Z")
I made a sample test file to figure out whether it is **Hail** itself or a **luigi** pipeline and it is the former. Here is the file that I created: ``` import hail as hl mt = hl.import_vcf(['/datasets-grch38/batch1_subset.vcf'], reference_genome='GRCh38', force_bgz=True, min_partitions=500) config = '/vep/GRCh38/vep95-loftee-grch38.json' mt = hl.vep(mt, config=config) mt.write('/seqr-loading-test/batch1_subset.mt', stage_locally=True, overwrite=True) ``` It fails without any output from **VEP** shown in the **Hail** log. Same as presented in the previous comment. The output file is getting created, so it’s not the permissions of the output folder otherwise I would expect the file not be created at all. All **vep** folders have permission **777** set. So, now I am stuck and the main reason is that Hail swallows up **VEP** error output. ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 24, 2020, 10:30pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/8 "2020-02-24T22:30:20Z")
Ok, I figured it out: the output does not go to the **Hail** log, but to _Spark work folder_ of the worker that executed the process. I just looked into the respective **spark work** folders **stderr** files and was able to find there the specific **VEP** error. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 24, 2020, 11:24pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/9 "2020-02-24T23:24:46Z")
Hmm. Something seems wrong then. So VEP is somehow writing to the stderr of the worker? We’ll check to make sure we’re capturing all the output. ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [February 25, 2020, 3:00pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/10 "2020-02-25T15:00:19Z")
Just in case, I used pre-compiled binaries of **Hail** found here: > **[macarthur-lab/hail-elasticsearch-pipelines](https://github.com/macarthur-lab/hail-elasticsearch-pipelines/tree/master/hail_builds/v02)** > > hail-based pipelines for annotating variant callsets and exporting them to elasticsearch - macarthur-lab/hail-elasticsearch-pipelines I have not tried it with the newest **Hail** on the master branch. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 25, 2020, 4:31pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/11 "2020-02-25T16:31:01Z")
Please do not use those JARs, they’re four months old. Hail development moves very quickly and we’ve made numerous improvements to Hail since then. We provide [detailed instructions](https://hail.is/docs/0.2/getting_started.html) for the installation of Hail in a number of environments. If anything there is unclear, please ask questions here and we will improve the documentation. Also, we recommend using [`hailctl dataproc`](https://hail.is/docs/0.2/hail_on_the_cloud.html) to start and stop clusters. The cluster control scripts in that repository are not recommended. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 25, 2020, 6:10pm UTC](https://discuss.hail.is/t/redirect-or-find-vep-or-other-error-output-from-a-hail-pipeline/1308/12 "2020-02-25T18:10:02Z")
We’re investigating the VEP output issue here: [https://github.com/hail-is/hail/issues/8146](https://github.com/hail-is/hail/issues/8146).