# Development

**URL:** https://discuss.hail.is/c/development/12.md

[Latest](https://discuss.hail.is/latest.md) · [Categories](https://discuss.hail.is/categories.md)

---

## [About the Development category](https://discuss.hail.is/t/about-the-development-category/21)

<div class="topic-metadata">

**Author:** [@hail-team](https://discuss.hail.is/u/hail-team)\
**Replies:** 0

</div>

Discussion of Hail software development. Are you hacking on Hail at the source? Post here to engage with the rest of the Hail developer community. The more clearly and completely you explain the issue, the more likely …

---

## [Reducing serialization size on high-partition jobs](https://discuss.hail.is/t/reducing-serialization-size-on-high-partition-jobs/4254)

<div class="topic-metadata">

**Author:** [@Thouis](https://discuss.hail.is/u/Thouis)\
**Replies:** 2\
**Last updated:** [September 11, 2026, 11:28am UTC](https://discuss.hail.is/t/reducing-serialization-size-on-high-partition-jobs/4254 "2026-09-11T11:28:46Z")

</div>

Hi all, I’ve been working on some code for computing PRSes in AllOfUs, and was hitting warnings like: WARN TaskSetManager: Stage 2 contains a task of very large size (1528 KiB). The maximum recommended task size is 100…

---

## [Reproduce hl.pc\_project output using Plink](https://discuss.hail.is/t/reproduce-hl-pc-project-output-using-plink/4190)

<div class="topic-metadata">

**Author:** [@Hatoon](https://discuss.hail.is/u/Hatoon)\
**Replies:** 2\
**Last updated:** [October 31, 2025, 7:49pm UTC](https://discuss.hail.is/t/reproduce-hl-pc-project-output-using-plink/4190 "2025-10-31T19:49:05Z")

</div>

Hi! I am trying to reproduce Hail’s pc\_project output using plink –score. I am adding the variance-standardize option and keeping mean imputation. Plink notes that “PCs will be scaled a bit differently from ref\_data.eig…

---

## [Hail index file compression type](https://discuss.hail.is/t/hail-index-file-compression-type/4196)

<div class="topic-metadata">

**Author:** [@PHILIPP111007](https://discuss.hail.is/u/PHILIPP111007)\
**Replies:** 4\
**Last updated:** [October 29, 2025, 7:27pm UTC](https://discuss.hail.is/t/hail-index-file-compression-type/4196 "2025-10-29T19:27:58Z")

</div>

Hello, what type of compression does the index file have? (matrix\_table/index/part-0/index) Sincerely, Roshchin Philipp

---

## [Hail having difficulty scaling to 400K](https://discuss.hail.is/t/hail-having-difficulty-scaling-to-400k/4158)

<div class="topic-metadata">

**Author:** [@Elizabeth\_Kiernan](https://discuss.hail.is/u/Elizabeth_Kiernan)\
**Replies:** 2\
**Last updated:** [June 9, 2025, 1:09pm UTC](https://discuss.hail.is/t/hail-having-difficulty-scaling-to-400k/4158 "2025-06-09T13:09:05Z")

</div>

Hi Hail team, We recently tried to filter a 400K sample VDS to 10K samples and export it to a hail MT. We hit problems at this scale in a Terra Jupyter notebook with a spark cluster. We ran 400 primary workers, 100 pree…

---

## [How is MatrixTable Entry data partitioned?](https://discuss.hail.is/t/how-is-matrixtable-entry-data-partitioned/3622)

<div class="topic-metadata">

**Author:** [@roise3000](https://discuss.hail.is/u/roise3000)\
**Replies:** 2\
**Last updated:** [October 20, 2023, 2:03am UTC](https://discuss.hail.is/t/how-is-matrixtable-entry-data-partitioned/3622 "2023-10-20T02:03:35Z")

</div>

Hi THere! I’m wondering how does HAIL partition data exactly? Does each spark partition contain the data for a portion of the entries (markers/variants and a portion of the samples)? let’s say we have a MatrixTable wi…

---

## [Tempo of infrastructure upgrades/containerization](https://discuss.hail.is/t/tempo-of-infrastructure-upgrades-containerization/3146)

<div class="topic-metadata">

**Author:** [@VinceCarey](https://discuss.hail.is/u/VinceCarey)\
**Replies:** 5\
**Last updated:** [June 15, 2023, 6:16am UTC](https://discuss.hail.is/t/tempo-of-infrastructure-upgrades-containerization/3146 "2023-06-15T06:16:26Z")

</div>

My understanding is that use of hail requires JDK 8 … to avoid changing my machine’s JDK I would prefer to use hail in a container. I have seen a dockerfile in the discussions and in a github repo but it doesn’t seem …

---

## [Hail Docker container?](https://discuss.hail.is/t/hail-docker-container/3392)

<div class="topic-metadata">

**Author:** [@okbishop](https://discuss.hail.is/u/okbishop)\
**Replies:** 1\
**Last updated:** [May 18, 2023, 3:18pm UTC](https://discuss.hail.is/t/hail-docker-container/3392 "2023-05-18T15:18:34Z")

</div>

Is there a Docker that’s published that’s got Hail in it? If so can you point me to it? Thanks

---

## [Hail with spark 3.2.0](https://discuss.hail.is/t/hail-with-spark-3-2-0/2386)

<div class="topic-metadata">

**Author:** [@skytzi](https://discuss.hail.is/u/skytzi)\
**Replies:** 5\
**Last updated:** [January 12, 2023, 8:07pm UTC](https://discuss.hail.is/t/hail-with-spark-3-2-0/2386 "2023-01-12T20:07:19Z")

</div>

Hi, according to hail documentation currently supported spark version is 3.1.2 (or 3.1.x). Spark released 3.2.0 version in October 2021. Is there a plan to support this new Spark version (Spark Release 3.2.0 | Apache Sp…

---

## [Combining VDSs with same samples](https://discuss.hail.is/t/combining-vdss-with-same-samples/2840)

<div class="topic-metadata">

**Author:** [@jbgaither](https://discuss.hail.is/u/jbgaither)\
**Replies:** 2\
**Last updated:** [October 5, 2022, 3:11am UTC](https://discuss.hail.is/t/combining-vdss-with-same-samples/2840 "2022-10-05T03:11:08Z")

</div>

I’ve built a collection of 22 VDSs, each containing variants from the same 100 samples, but covering a different chromosome: path/vds\_chr1 … path/vds\_chr22. I want to merge these into a single VDS. However, when I tr…

---

## [Dimension reduction techniques (t-SNE, UMAP) in Haill](https://discuss.hail.is/t/dimension-reduction-techniques-t-sne-umap-in-haill/2783)

<div class="topic-metadata">

**Author:** [@Alexey](https://discuss.hail.is/u/Alexey)\
**Replies:** 1\
**Last updated:** [August 13, 2022, 8:59pm UTC](https://discuss.hail.is/t/dimension-reduction-techniques-t-sne-umap-in-haill/2783 "2022-08-13T20:59:17Z")

</div>

Hello everyone. i would like to introduce a new topic on dimension reduction techniques. Particularly i would like to introduce new statistical methods to Hill. As far as i understand currently only one dimension reducti…

---

## [Nirvana from hail in jupyter](https://discuss.hail.is/t/nirvana-from-hail-in-jupyter/2633)

<div class="topic-metadata">

**Author:** [@mhebrard](https://discuss.hail.is/u/mhebrard)\
**Replies:** 0\
**Last updated:** [April 26, 2022, 7:07am UTC](https://discuss.hail.is/t/nirvana-from-hail-in-jupyter/2633 "2022-04-26T07:07:54Z")

</div>

Hi, I successfully installed VEP and run it through HAIL using ht\_vep = hl.vep(ht,'s3://bucket/path/vep95\_GRCh38\_config.json') I am now looking for Nirvana. I am aware it is a historical experimental function in hail …

---

## [Scala version incompatible between hail and spark?](https://discuss.hail.is/t/scala-version-incompatible-between-hail-and-spark/2566)

<div class="topic-metadata">

**Author:** [@SimonLi5601](https://discuss.hail.is/u/SimonLi5601)\
**Replies:** 6\
**Last updated:** [April 1, 2022, 1:39pm UTC](https://discuss.hail.is/t/scala-version-incompatible-between-hail-and-spark/2566 "2022-04-01T13:39:18Z")

</div>

One thing I don’t understand why hail (0.2.83) only allows Scala 2.12.x with the Spark version is 3.x, and when Spark version is 2.4.x, it forces to use Scala 2.11.x? But does Spark 2.4.x requires Scala 2.12.x? e.g. 2.4.…

---

## [Can't locate DBI.pm in @INC (you may need to install the DBI module)](https://discuss.hail.is/t/cant-locate-dbi-pm-in-inc-you-may-need-to-install-the-dbi-module/2563)

<div class="topic-metadata">

**Author:** [@Abhishek](https://discuss.hail.is/u/Abhishek)\
**Replies:** 3\
**Last updated:** [March 23, 2022, 1:09pm UTC](https://discuss.hail.is/t/cant-locate-dbi-pm-in-inc-you-may-need-to-install-the-dbi-module/2563 "2022-03-23T13:09:54Z")

</div>

Hi, I am unable to use VEP using hail built on top of spark cluster on DNAnexus platform. The error stack trace: Can't locate DBI.pm in @INC (you may need to install the DBI module) (@INC contains: /cluster/vep/module…

---

## [DNANexus: hail installation on custom spark cluster](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527)

<div class="topic-metadata">

**Author:** [@Abhishek](https://discuss.hail.is/u/Abhishek)\
**Replies:** 3\
**Last updated:** [March 3, 2022, 7:14am UTC](https://discuss.hail.is/t/dnanexus-hail-installation-on-custom-spark-cluster/2527 "2022-03-03T07:14:07Z")

</div>

Hi all, Context: I am working on DNANexus platform. I am trying to install hail version 0.2.83 with spark 3.2.0 version. WARNING: Hail primarily tested with Spark 3.1.2, use other versions at your own risk. \* FAILURE: …

---

## [Thoughts on making hail compilation more offline friendly?](https://discuss.hail.is/t/thoughts-on-making-hail-compilation-more-offline-friendly/2433)

<div class="topic-metadata">

**Author:** [@CreRecombinase](https://discuss.hail.is/u/CreRecombinase)\
**Replies:** 3\
**Last updated:** [February 24, 2022, 6:27pm UTC](https://discuss.hail.is/t/thoughts-on-making-hail-compilation-more-offline-friendly/2433 "2022-02-24T18:27:03Z")

</div>

I’m trying to build hail in an environment where I don’t have access to the public internet, and have run into some issues. There are a few places in the makefile where hail pulls specific versions of header-only librar…

---

## [What should be the format of the Nirvana annotation structure?](https://discuss.hail.is/t/what-should-be-the-format-of-the-nirvana-annotation-structure/2356)

<div class="topic-metadata">

**Author:** [@thondeboer](https://discuss.hail.is/u/thondeboer)\
**Replies:** 1\
**Last updated:** [November 17, 2021, 1:57pm UTC](https://discuss.hail.is/t/what-should-be-the-format-of-the-nirvana-annotation-structure/2356 "2021-11-17T13:57:00Z")

</div>

In the docs on the VEP structure schema it states: vep\_json\_schema (string): The type of the VEP JSON schema (as produced by the VEP when invoked with the –json option). Note: This is the old-style ‘parseable’ Hail typ…

---

## [Hail cluster mode intall error](https://discuss.hail.is/t/hail-cluster-mode-intall-error/2292)

<div class="topic-metadata">

**Author:** [@ksmpooh](https://discuss.hail.is/u/ksmpooh)\
**Replies:** 0\
**Last updated:** [October 13, 2021, 1:08am UTC](https://discuss.hail.is/t/hail-cluster-mode-intall-error/2292 "2021-10-13T01:08:25Z")

</div>

Dear Hail members~ I tried to install hail cluster mode on ubuntu 16.04 python3 version is 3.7 Error massage like this command : ------------------- make install-on-cluster HAIL\_COMPILE\_NATIVES=1 SCALA\_VERSION=2.12.…

---

## [Load vcf from SMB share](https://discuss.hail.is/t/load-vcf-from-smb-share/2224)

<div class="topic-metadata">

**Author:** [@quantumbarista](https://discuss.hail.is/u/quantumbarista)\
**Replies:** 5\
**Last updated:** [September 15, 2021, 8:50pm UTC](https://discuss.hail.is/t/load-vcf-from-smb-share/2224 "2021-09-15T20:50:25Z")

</div>

I’m trying to load VCF files from an Isilon server (since I don’t have TB of storage on my laptop). The problem seems to be that Hadoop doesn’t like the : character in my SMB relative file path. I can open the file at …

---

## [Unable to deploy/compile latest Hail on aws](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216)

<div class="topic-metadata">

**Author:** [@yc000000](https://discuss.hail.is/u/yc000000)\
**Replies:** 14\
**Last updated:** [September 10, 2021, 8:41pm UTC](https://discuss.hail.is/t/unable-to-deploy-compile-latest-hail-on-aws/2216 "2021-09-10T20:41:06Z")

</div>

Hi, I am trying to update Hail from 0.1 to the latest version but kept failing. Here is what I did (from this site: Hail | For Software Developers) echo 'Installing pre-reqs' yum install -y g++ cmake git …

---

## [Building hail in Docker](https://discuss.hail.is/t/building-hail-in-docker/2041)

<div class="topic-metadata">

**Author:** [@MariusDanner](https://discuss.hail.is/u/MariusDanner)\
**Replies:** 5\
**Last updated:** [May 13, 2021, 12:53pm UTC](https://discuss.hail.is/t/building-hail-in-docker/2041 "2021-05-13T12:53:27Z")

</div>

Hi, I am currently trying to create a Docker image with a custom version of hail but I am running into problems. The Dockerfile looks somewhat like this: FROM ubuntu:20.10 AS hail RUN apt-get update && apt-get…

---

## [P-value differ from R for linear regression](https://discuss.hail.is/t/p-value-differ-from-r-for-linear-regression/1964)

<div class="topic-metadata">

**Author:** [@mocksu](https://discuss.hail.is/u/mocksu)\
**Replies:** 6\
**Last updated:** [March 31, 2021, 12:49am UTC](https://discuss.hail.is/t/p-value-differ-from-r-for-linear-regression/1964 "2021-03-31T00:49:26Z")

</div>

It’s a big surprise that I got different p-values from hail and from R. The input data as I manually checked is identical. The hail code is as follows: mt = mt.annotate\_rows(gwas = hl.agg.linreg(hl.float(mt.phenos.heigh…

---

## [How do I fix "java.net.BindException: Can't assign requested address"?](https://discuss.hail.is/t/how-do-i-fix-java-net-bindexception-cant-assign-requested-address/115)

<div class="topic-metadata">

**Author:** [@danking](https://discuss.hail.is/u/danking)\
**Replies:** 4\
**Last updated:** [December 4, 2020, 7:52pm UTC](https://discuss.hail.is/t/how-do-i-fix-java-net-bindexception-cant-assign-requested-address/115 "2020-12-04T19:52:28Z")

</div>

The full message looks like: Exception in thread "main" java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries! Consider explicitly setting the appropriate port for the s…

---

## [Hail in a docker](https://discuss.hail.is/t/hail-in-a-docker/1132)

<div class="topic-metadata">

**Author:** [@xinming](https://discuss.hail.is/u/xinming)\
**Replies:** 9\
**Last updated:** [November 15, 2020, 9:27pm UTC](https://discuss.hail.is/t/hail-in-a-docker/1132 "2020-11-15T21:27:21Z")

</div>

Looking forward to a docker version of Hail.

---

## [Availability of files in hail-data?](https://discuss.hail.is/t/availability-of-files-in-hail-data/1240)

<div class="topic-metadata">

**Author:** [@hammer](https://discuss.hail.is/u/hammer)\
**Replies:** 1\
**Last updated:** [January 13, 2020, 9:38pm UTC](https://discuss.hail.is/t/availability-of-files-in-hail-data/1240 "2020-01-13T21:38:20Z")

</div>

We’re trying to identify places where Hail (and Glow) really lag PLINK in performance, and in the process, have been seeking example data sets to see how performance scales up. It looks like y’all have a nice collection…

---

## [0.3 Hail release](https://discuss.hail.is/t/0-3-hail-release/1229)

<div class="topic-metadata">

**Author:** [@hammer](https://discuss.hail.is/u/hammer)\
**Replies:** 4\
**Last updated:** [December 30, 2019, 12:51pm UTC](https://discuss.hail.is/t/0-3-hail-release/1229 "2019-12-30T12:51:04Z")

</div>

I was excited to see the 0.3 milestone and 0.3 issue label while looking over the Hail repo this evening. Do y’all have a target date and/or set of features for the 0.3 release?

---

## [Getting rg update up and running](https://discuss.hail.is/t/getting-rg-update-up-and-running/1108)

<div class="topic-metadata">

**Author:** [@astheeggeggs](https://discuss.hail.is/u/astheeggeggs)\
**Replies:** 2\
**Last updated:** [October 4, 2019, 10:43pm UTC](https://discuss.hail.is/t/getting-rg-update-up-and-running/1108 "2019-10-04T22:43:19Z")

</div>

Hi guys, who would be best to talk to about getting the updated rg website up and running? I’m keen to get it running before gwasbot links to the outdated site. Thanks!

---

## [Haploid support](https://discuss.hail.is/t/haploid-support/355)

<div class="topic-metadata">

**Author:** [@cseed](https://discuss.hail.is/u/cseed)\
**Replies:** 0\
**Last updated:** [November 12, 2017, 1:57am UTC](https://discuss.hail.is/t/haploid-support/355 "2017-11-12T01:57:57Z")

</div>

We’re considering adding true haploid support in the 0.2 development version. We’re discussing it on the development forum, but the interface might be of general interest: http://dev.hail.is/t/haploids-and-generic-genot…

---

## [Online Median Calculation](https://discuss.hail.is/t/online-median-calculation/275)

<div class="topic-metadata">

**Author:** [@danking](https://discuss.hail.is/u/danking)\
**Replies:** 0\
**Last updated:** [September 7, 2017, 7:26pm UTC](https://discuss.hail.is/t/online-median-calculation/275 "2017-09-07T19:26:27Z")

</div>

cc @tpoterba We need online median calculation in Hail. Currently, the closest thing you can get to a median is to calculate a histogram of your data using hist and then determine the bin in which the median resides (by…

---

## [In the \`testHail\` gradle step, I get "ImportError: No module named pyspark.sql"](https://discuss.hail.is/t/in-the-testhail-gradle-step-i-get-importerror-no-module-named-pyspark-sql/120)

<div class="topic-metadata">

**Author:** [@danking](https://discuss.hail.is/u/danking)\
**Replies:** 1\
**Last updated:** [February 9, 2017, 3:53pm UTC](https://discuss.hail.is/t/in-the-testhail-gradle-step-i-get-importerror-no-module-named-pyspark-sql/120 "2017-02-09T15:53:29Z")

</div>

How do I fix this? The full log looks like this: \[ec2-user@ip-172-31-54-96 hail\]$ PYTHONPATH=$SPARK\_HOME/python:$SPARK\_HOME/python/lib/py4j-0.10.3-src.zip:$HAIL\_HOME/python SPARK\_CLASSPATH=$HAIL\_HOME/build/libs/hail-a…

[Next page](https://discuss.hail.is/c/development/12.md?page=1)
