# Help \[0.1\] **URL:** https://discuss.hail.is/c/help-01/6.md?page=2 [Latest](https://discuss.hail.is/latest.md) · [Categories](https://discuss.hail.is/categories.md) **Page:** 3 --- ## [Sample\_qc error](https://discuss.hail.is/t/sample-qc-error/556)
**Author:** [@nnekaede](https://discuss.hail.is/u/nnekaede)\ **Replies:** 9\ **Last updated:** [July 10, 2018, 10:15pm UTC](https://discuss.hail.is/t/sample-qc-error/556 "2018-07-10T22:15:36Z")
I used the following line of code: vds = hl.sample\_qc(vds) vds.describe() And I got the following error: --------------------------------------------------------------------------- FatalError … --- ## [java.util.NoSuchElementException: key not found: \_PYSPARK\_DRIVER\_CALLBACK\_HOST](https://discuss.hail.is/t/java-util-nosuchelementexception-key-not-found-pyspark-driver-callback-host/533)
**Author:** [@hhx037](https://discuss.hail.is/u/hhx037)\ **Replies:** 7\ **Last updated:** [July 9, 2018, 2:13pm UTC](https://discuss.hail.is/t/java-util-nosuchelementexception-key-not-found-pyspark-driver-callback-host/533 "2018-07-09T14:13:10Z")
Hi, I’m trying to get Spark 0.2 to run, but it’s failing and I don’t know what to do. Below are the commands and error message. Python 3.6.5 (default, May 3 2018, 10:08:28) \[GCC 5.4.0 20160609\] on linux Type “help”… --- ## [Extract a subset of SNP towards per individuals](https://discuss.hail.is/t/extract-a-subset-of-snp-towards-per-individuals/558)
**Author:** [@Win](https://discuss.hail.is/u/Win)\ **Replies:** 4\ **Last updated:** [July 9, 2018, 12:52pm UTC](https://discuss.hail.is/t/extract-a-subset-of-snp-towards-per-individuals/558 "2018-07-09T12:52:26Z")
Anyone could give me a hand? --- ## [Import\_table error](https://discuss.hail.is/t/import-table-error/555)
**Author:** [@nnekaede](https://discuss.hail.is/u/nnekaede)\ **Replies:** 1\ **Last updated:** [July 5, 2018, 6:02pm UTC](https://discuss.hail.is/t/import-table-error/555 "2018-07-05T18:02:18Z")
I am trying to import a .csv file with annotations for the 1k genome project. However, when I run this code: table = (hl.import\_table('gs://1k-genome/1000-genomes/other/sample\_info/sample\_info.csv', impute = True, delim… --- ## [Run a pipeline using Hail 01. for Spark and elastic](https://discuss.hail.is/t/run-a-pipeline-using-hail-01-for-spark-and-elastic/552)
**Author:** [@ojuarez](https://discuss.hail.is/u/ojuarez)\ **Replies:** 7\ **Last updated:** [July 2, 2018, 6:48pm UTC](https://discuss.hail.is/t/run-a-pipeline-using-hail-01-for-spark-and-elastic/552 "2018-07-02T18:48:28Z")
hail.java.FatalError: HailException: property \`hail.vep.location’ required I am not sure how to configure vep. vds = vds.vep(config="/vep/vep-gcloud.properties", root=‘va.vep’, block\_size=1000) --- ## [Difficulty annotating matrix table with phenotypes on Hail 0.2](https://discuss.hail.is/t/difficulty-annotating-matrix-table-with-phenotypes-on-hail-0-2/530)
**Author:** [@David\_Ficenec](https://discuss.hail.is/u/David_Ficenec)\ **Replies:** 4\ **Last updated:** [July 2, 2018, 2:07pm UTC](https://discuss.hail.is/t/difficulty-annotating-matrix-table-with-phenotypes-on-hail-0-2/530 "2018-07-02T14:07:49Z")
We are having difficulty annotating our UKBB dataset on Hail 0.2 (5/30 commit). Our dataset consists of roughly 800k genotypes x 500k samples x 16k phenotypes. We’re running on AWS EMR 5.12.1, Spark 2.2.0 with 4 x m4.10… --- ## [SparkContext shutting down](https://discuss.hail.is/t/sparkcontext-shutting-down/526)
**Author:** [@Ruth\_Mitchell](https://discuss.hail.is/u/Ruth_Mitchell)\ **Replies:** 1\ **Last updated:** [July 2, 2018, 2:03pm UTC](https://discuss.hail.is/t/sparkcontext-shutting-down/526 "2018-07-02T14:03:52Z")
Hi there, I’m trying to export a vcf but the SparkContext is shutting down. What should I do bout it? Here is the trace back: hl.export\_vcf(dataset\_result, 'chr22.vcf.bgz') 2018-06-14 16:42:24 Hail: WARN: export\_vcf fo… --- ## [VEP fails to generate annotations](https://discuss.hail.is/t/vep-fails-to-generate-annotations/506)
**Author:** [@mike](https://discuss.hail.is/u/mike)\ **Replies:** 17\ **Last updated:** [June 19, 2018, 5:24pm UTC](https://discuss.hail.is/t/vep-fails-to-generate-annotations/506 "2018-06-19T17:24:43Z")
I’m using Spark 2.2.1 with Hail 0.2, and trying to use VEP to annotate some sample records from ClinVar. These are for GRCh38, and I am extending the VEP 92 Docker image (ensemblorg/ensembl-vep:release\_92.1). This means … --- ## [GP to DS conversion](https://discuss.hail.is/t/gp-to-ds-conversion/525)
**Author:** [@Ruth\_Mitchell](https://discuss.hail.is/u/Ruth_Mitchell)\ **Replies:** 4\ **Last updated:** [June 19, 2018, 4:16pm UTC](https://discuss.hail.is/t/gp-to-ds-conversion/525 "2018-06-19T16:16:36Z")
Is there a way to go from bgen files with genotype probabilites to vcf files with genotype dosages? I’m wondering if the gp\_dosage can be extended to a full bgen file once imported? Or if Hail provides another way of an… --- ## [Hail 0.2 on EMR?](https://discuss.hail.is/t/hail-0-2-on-emr/486)
**Author:** [@Oron\_Navon](https://discuss.hail.is/u/Oron_Navon)\ **Replies:** 17\ **Last updated:** [June 18, 2018, 2:15pm UTC](https://discuss.hail.is/t/hail-0-2-on-emr/486 "2018-06-18T14:15:35Z")
Hi, has anyone managed to get Hail 0.2 working on EMR? I’ve found bits and pieces but no straightforward guide… Thanks, Oron --- ## [Issue building Hail 0.2 on EMR](https://discuss.hail.is/t/issue-building-hail-0-2-on-emr/528)
**Author:** [@David\_Ficenec](https://discuss.hail.is/u/David_Ficenec)\ **Replies:** 5\ **Last updated:** [June 18, 2018, 1:48pm UTC](https://discuss.hail.is/t/issue-building-hail-0-2-on-emr/528 "2018-06-18T13:48:37Z")
We are using the AWS Hail build script s3://aws-bigdata-blog/artifacts/hail-on-emr/build\_hail.sh (parameters EMR 5.12.1, Spark 2.2.0) to build Hail 0.2. This is implemented as one of several boot strap actions in our Clo… --- ## [How to get Hail's Java and Python Libraries](https://discuss.hail.is/t/how-to-get-hails-java-and-python-libraries/515)
**Author:** [@nnekaede](https://discuss.hail.is/u/nnekaede)\ **Replies:** 1\ **Last updated:** [June 18, 2018, 1:26pm UTC](https://discuss.hail.is/t/how-to-get-hails-java-and-python-libraries/515 "2018-06-18T13:26:17Z")
I have been trying to use the hail-deployment notebook from the Hail tutorial on databricks, and it is asking me to Download Hail’s Java and Python libraries built with the latest stable version of Hail for Spark 2.1.1 a… --- ## [Hail on Databricks with Spark Cluster](https://discuss.hail.is/t/hail-on-databricks-with-spark-cluster/511)
**Author:** [@nnekaede](https://discuss.hail.is/u/nnekaede)\ **Replies:** 11\ **Last updated:** [June 18, 2018, 1:25pm UTC](https://discuss.hail.is/t/hail-on-databricks-with-spark-cluster/511 "2018-06-18T13:25:58Z")
I am new to Hail. I have been trying to use the hail tutorials on databricks using the spark cluster. However, when I try to import hail with import hail as \* but I get the error ImportError: No module named hail. What a… --- ## [Read VDS in Hail 0.2](https://discuss.hail.is/t/read-vds-in-hail-0-2/509)
**Author:** [@ines](https://discuss.hail.is/u/ines)\ **Replies:** 3\ **Last updated:** [May 29, 2018, 6:46pm UTC](https://discuss.hail.is/t/read-vds-in-hail-0-2/509 "2018-05-29T18:46:43Z")
Hello, How can I read the vds file that I have written with Hail 0.1 in the new version of Hail 0.2 ? Thank you ! --- ## [Hail on Goocle Cloud with Windows OS](https://discuss.hail.is/t/hail-on-goocle-cloud-with-windows-os/494)
**Author:** [@ines](https://discuss.hail.is/u/ines)\ **Replies:** 23\ **Last updated:** [May 25, 2018, 1:17am UTC](https://discuss.hail.is/t/hail-on-goocle-cloud-with-windows-os/494 "2018-05-25T01:17:13Z")
Hello, I have several installation issues on Google Cloud. I follow this tutorial: And I ask myself the question: can it be an OS problem? I work on windows. Let me explain. On the github of NealeLab (GitHub - Nealel… --- ## [Hail 0.2 - method group\_rows\_by requieres lot of space on disk](https://discuss.hail.is/t/hail-0-2-method-group-rows-by-requieres-lot-of-space-on-disk/508)
**Author:** [@enriquea](https://discuss.hail.is/u/enriquea)\ **Replies:** 3\ **Last updated:** [May 24, 2018, 9:00am UTC](https://discuss.hail.is/t/hail-0-2-method-group-rows-by-requieres-lot-of-space-on-disk/508 "2018-05-24T09:00:01Z")
Hi everyone, Trying to aggregate a MatrixTable (~100 GB on disk) by rows (e.g. gene and variant consequences), I’ve observed that Hail (Spark behind the scenes) requires until 5X disk space to do the job. Running some … --- ## [Missing class when reading .vds from S3](https://discuss.hail.is/t/missing-class-when-reading-vds-from-s3/507)
**Author:** [@gaborkorodi-hms](https://discuss.hail.is/u/gaborkorodi-hms)\ **Replies:** 1\ **Last updated:** [May 22, 2018, 7:47pm UTC](https://discuss.hail.is/t/missing-class-when-reading-vds-from-s3/507 "2018-05-22T19:47:48Z")
The following error is popping up the new cluster, which was created with the old CloudFormation script (command line works, but this is from the JupyterNotebook) FatalErrorTraceback (most recent call last) \ **Author:** [@Ruth\_Mitchell](https://discuss.hail.is/u/Ruth_Mitchell)\ **Replies:** 6\ **Last updated:** [May 16, 2018, 4:05pm UTC](https://discuss.hail.is/t/error-creating-hail-context/503 "2018-05-16T16:05:20Z") I’m having trouble creating a HailContext. We’ve installed hail and its dependencies (Java, spark and python - can I double check which version of python is needed as the website say 2 yet the installation docs specify p… --- ## [is.hail.kryo.HailKryoRegistrator ClassNotFoundException](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484)
**Author:** [@atebbe](https://discuss.hail.is/u/atebbe)\ **Replies:** 9\ **Last updated:** [May 4, 2018, 7:10pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484 "2018-05-04T19:10:38Z")
We are getting an exception when calling the hl.import\_vcf() function. The hl.init() function completes without issue, which makes me think the hail jar is loading properly. The import\_vcf function is the first one we… --- ## [Using Hail 0.1 import\_vcf error](https://discuss.hail.is/t/using-hail-0-1-import-vcf-error/479)
**Author:** [@ojuarez412](https://discuss.hail.is/u/ojuarez412)\ **Replies:** 2\ **Last updated:** [April 27, 2018, 5:49pm UTC](https://discuss.hail.is/t/using-hail-0-1-import-vcf-error/479 "2018-04-27T17:49:29Z")
Note: I was using Hail0.2 and the following script works: import hail as hl hl.import\_vcf(’/tmp/resources/sample.vcf’).write(’/tmp/output/sample.vds’) But I switch to 0.1 because we are using SEQR and using some pipe… --- ## [Running hail on AWS EMR](https://discuss.hail.is/t/running-hail-on-aws-emr/467)
**Author:** [@chirag\_lakhani](https://discuss.hail.is/u/chirag_lakhani)\ **Replies:** 6\ **Last updated:** [April 27, 2018, 4:54pm UTC](https://discuss.hail.is/t/running-hail-on-aws-emr/467 "2018-04-27T16:54:23Z")
Hi, I apologize if this is a dumb question but I’m a Hail newbie. I am trying to simply run Hail on Amazon EMR. I have setup a small EMR cluster using emr-5.13.0 which has Spark 2.3.0. Following some scripts built by… --- ## [Split multi ERROR](https://discuss.hail.is/t/split-multi-error/476)
**Author:** [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\ **Replies:** 0\ **Last updated:** [April 24, 2018, 2:37pm UTC](https://discuss.hail.is/t/split-multi-error/476 "2018-04-24T14:37:50Z")
Hi, I’m trying to annotate a vcf with another vcf (gnomAD annotations), and the steps I follow are: annotation\_vds = hc.import\_vcf(“annotations\_vcf.vcf”).split\_multi() sample\_vds = hc.import\_vcf(“sample\_vcf.vcf”).spli… --- ## [Using Spark ML to create and apply Random Forests](https://discuss.hail.is/t/using-spark-ml-to-create-and-apply-random-forests/204)
**Author:** [@lfrancioli](https://discuss.hail.is/u/lfrancioli)\ **Replies:** 2\ **Last updated:** [April 26, 2018, 4:21pm UTC](https://discuss.hail.is/t/using-spark-ml-to-create-and-apply-random-forests/204 "2018-04-26T16:21:27Z")
The code below is used to compute the Random Forests model to filter gnomAD variants. A dummy example of the intended usage flow is implemented in run\_rf\_test(vds, output) import logging from hail.expr import \* from p… --- ## [Cluster size for using Hail on AWS?](https://discuss.hail.is/t/cluster-size-for-using-hail-on-aws/475)
**Author:** [@ines](https://discuss.hail.is/u/ines)\ **Replies:** 3\ **Last updated:** [April 24, 2018, 1:53pm UTC](https://discuss.hail.is/t/cluster-size-for-using-hail-on-aws/475 "2018-04-24T13:53:58Z")
Hi everyone, I am working on data from the Simon Simplex collection which is a permanent repository of genetic samples from 2,600 simplex families. I want to import and analyze his data on Hail using Amazon Cloud (AWS)… --- ## [Hail with Spark 2.1.1?](https://discuss.hail.is/t/hail-with-spark-2-1-1/470)
**Author:** [@pescobar](https://discuss.hail.is/u/pescobar)\ **Replies:** 1\ **Last updated:** [April 18, 2018, 2:38pm UTC](https://discuss.hail.is/t/hail-with-spark-2-1-1/470 "2018-04-18T14:38:44Z")
I have booted a spark cluster in the cloud using Elasticluster. Elasticluster uses the Apache Bigtop distribution, which currently builds with Spark 2.1.1. I have tried to build a hail jar in this cluster following the o… --- ## [In hail 0.1, is it possible to export variants table with selected columns into a pandas dataframe?](https://discuss.hail.is/t/in-hail-0-1-is-it-possible-to-export-variants-table-with-selected-columns-into-a-pandas-dataframe/464)
**Author:** [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Replies:** 2\ **Last updated:** [April 13, 2018, 6:25pm UTC](https://discuss.hail.is/t/in-hail-0-1-is-it-possible-to-export-variants-table-with-selected-columns-into-a-pandas-dataframe/464 "2018-04-13T18:25:02Z")
Hi everyone, I’m new to hail. I’m looking for some function like the following one: vds.export\_variants(‘output/file.tsv’, ‘variant = v, QC = va.qc.\*’) However, instead of saving it into a file, I’d like to receive … --- ## [A problem with KeyTable.from\_pandas in hail v0.1](https://discuss.hail.is/t/a-problem-with-keytable-from-pandas-in-hail-v0-1/461)
**Author:** [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Replies:** 4\ **Last updated:** [April 12, 2018, 4:20pm UTC](https://discuss.hail.is/t/a-problem-with-keytable-from-pandas-in-hail-v0-1/461 "2018-04-12T16:20:41Z")
Hi everyone, I’m new to hail. I have a problem annotating vds using the KeyTable generated from a pandas dataframe. The code looks like this: load gene expression data residuals = pd.read\_table(‘eQTL/data/residuals\_5.… --- ## [Using Hail on Spark 2.1.1 Azure HDInsight causes error](https://discuss.hail.is/t/using-hail-on-spark-2-1-1-azure-hdinsight-causes-error/453)
**Author:** [@yugagarin](https://discuss.hail.is/u/yugagarin)\ **Replies:** 16\ **Last updated:** [April 6, 2018, 1:42pm UTC](https://discuss.hail.is/t/using-hail-on-spark-2-1-1-azure-hdinsight-causes-error/453 "2018-04-06T13:42:00Z")
Good day! I am using Spark 2.1.1 on HDinsight in Azure. I have built Hail as listed here https://hail.is/docs/stable/getting\_started.html for Spark version 2.1.1 . However, when I execute in ipython from hail import… --- ## [VDS annotation triggers shuffle](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450)
**Author:** [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\ **Replies:** 6\ **Last updated:** [April 4, 2018, 1:04pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450 "2018-04-04T13:04:04Z")
Hi, I’m importing and annotating a vcf with just one chromosome, and I get the following message when annotating (with annotate\_variants\_table): Hail: INFO: Ordering unsorted dataset with network shuffle The vcf file … --- ## [Annotate with bed files in hail 0.2beta](https://discuss.hail.is/t/annotate-with-bed-files-in-hail-0-2beta/446)
**Author:** [@timstrom](https://discuss.hail.is/u/timstrom)\ **Replies:** 5\ **Last updated:** [March 29, 2018, 7:41pm UTC](https://discuss.hail.is/t/annotate-with-bed-files-in-hail-0-2beta/446 "2018-03-29T19:41:00Z")
Hi Hail team, we try to migrate to hail 0.2beta. In Hail 0.1 one could use bed files to annotate variants: bed = KeyTable.import\_bed(‘genomicSuperDups.bed’) vds = vds.annotate\_variants\_table(bed, root=‘va.genomicSupe… [Previous page](https://discuss.hail.is/c/help-01/6.md?page=1) [Next page](https://discuss.hail.is/c/help-01/6.md?page=3)