Quality Control, filtering false positive variants

[timstrom]

The German Center for Heart Research (DZHK) is analyzing 1205 genomes
from 5 general population cohorts (KORA, SHIP, Hamburg City, Mainz City,
Uni Heidelberg) plus 548 genomes from patients with coronary heart
disease or cardiomyopathy. Sequencing was performed at the DKFZ on
HiSeq X machines,

We would like to analyze the data in a way that is compatible with
an integration of these data into gnomAD. We would be very grateful if
you could support us on technical details.

Sequencing depth is at least 30 fold (mean 40).

We are currently analyzing the data according to the GATK best
practices guidelines and perform statistics for quality control
in hail.

For QC we are using 24 trios (sequenced on HiSeq4000). We
used GSA arrays and overlap with gnomAD variants as measures for sensitivity
and Ti/TV ratio and singletons in offspring of trios as measures for
specificity.

We evaluated two settings for base quality score recalibration
(SNV_context 2 or 4, indel_context 3 or 8). Variant quality
score recalibration was done with 1kg or gnomAD (see link).

We conclude from the number of singletons that the use of
SNV_context 4 and indel_context 8 is superior to 2 and 4, respectively.

But unfortunately, we observed that we had too many singletons
after VQSR (many thousands). Our attempt to use hard thresholds
on GQ, alternative read fraction, and/or MQ reduced the number
of singletons, but also the overlap with gnomAD SNVs. We concluded
that this approach is not feasible.

When we inspect the BAM files around the singletons,
we noticed that a large fraction is located in or near of
homopolymers or tandem repeats with many soft-clipped reads.
Visual inspection would relatively easily identify this false
positives.

We have the impression that the random forest approach used
in gnomAD filters out these ‘dirty’ regions much better than
VQSR does.

We would like to ask if you could recommend how to proceed further.

Tim

More details about QC can be found at

2018-06-14-dzhk.pdf

[tpoterba]

@lfrancioli @konradjk any ideas? Is the random forest code up somewhere and compatible with 0.2?

[konradjk]

We have an RF module with some helper functions here: https://github.com/macarthur-lab/gnomad_hail/blob/master/utils/rf.py

The code to use those functions is not yet public but will be (we’re currently actively tweaking it). @lfrancioli can comment further as to the exact usage.

[timstrom]

I got part of the gnomad_qc scripts running with older versions of hail, gnomad_qc and gnomad_hail (hail from October 2108 Hail-devel-b0342b11e1af).

I tried to update to the newest precompiled version of hail (0.2.4-82d3af919183) and the 2nd December version of gnomad_qc and gnomad_hail.

I am receiving several errors in generate_qc_annotations.py older versions of which run without problems.

For example:

1. When running ‘–generate_qc_annotations’, I receive the following error:

Traceback …
in generate_qc_annotations mt.PL[0] > 3000, -300, hl.log10(gp0)
… aggregators.py", line 26, in check assert(len(x._ir.search(lambda node:
isinstance(node, BaseApplyAggOp))) != 0) AssertionError

2. When running ‘–annotate_truth_data’, truth_data can not be repartitioned:

Traceback …
in main annotate_truth_data(mt).write(annotations_ht_path(data_type, ‘truth_data’), args.overwrite)
…AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles]…

I downloaded the truth_data from gs://gnomad-public.

Is it more likely that these are version problems or problems of our vcf data?

[tpoterba]

Can we have the full stack trace for these errors? Hard to say just based on the first bit.

[timstrom]

This is code from ‘generate_qc_annotations.py’ and how I modified it. Truth data is downloaded from gs://gnomad-public.

This is the command :

python  /home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py --overwrite --genomes --annotate_truth_data

This is the code (modification is indicated). I think, the error occurs before the modification.

def annotate_truth_data(mt: hl.MatrixTable) -> hl.Table:
"""
Writes bi-allelic sites MT with the following annotations:
 - truth_data (presence in Omni, HapMap, 1KG high conf SNVs, Mills)

:param MatrixTable mt: Full MT
:return: Table with qc annotations
:rtype: Table
"""
mt = mt.select_rows()

truth_mtes = {
    'hapmap': hapmap_mt_path(),
    'omni': omni_mt_path(),
    'mills': mills_mt_path(),
    'kgp_high_conf_snvs': kgp_high_conf_snvs_mt_path()
}
truth_mtes = {key: hl.split_multi_hts(hl.read_matrix_table(path).repartition(1000), left_aligned=False)
              for key, path in truth_mtes.items()}
#tim
#return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt[mt.row_key, :])
#                                                for root, truth_mt in truth_mtes.items()})).rows()
return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt.rows()[mt.row_key])
                                                for root, truth_mt in truth_mtes.items()})).rows()

This modified code is working with a October version of hail.

Currently I think, the error is in ‘hl.read_matrix_table(path).repartition(1000)’ (see below)
I am receiving the following traceback:

Traceback (most recent call last):
  File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 304, in <module>
    main(args)
  File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 276, in main
    annotate_truth_data(mt).write(annotations_ht_path(data_type, 'truth_data'), args.overwrite)
  File "<decorator-gen-690>", line 2, in write
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 560, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py", line 1153, in write
    Env.hc()._backend.interpret(TableWrite(self._tir, output, overwrite, stage_locally, _codec_spec))
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/backend/backend.py", line 25, in interpret
    result = Env.hail().expr.ir.Interpret.interpretPyIR(code, {}, ir_map)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/utils/java.py", line 210, in deco
    'Error summary: %s' % (deepest, full, hail.__version__, deepest)) from None
hail.utils.java.FatalError: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}

Java stack trace:
java.lang.AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
at scala.Predef$.assert(Predef.scala:170)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:42)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1$$anonfun$applyOrElse$1.apply(LowerMatrixIR.scala:164)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1$$anonfun$applyOrElse$1.apply(LowerMatrixIR.scala:164)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:164)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:150)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:196)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:196)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$.lower(LowerMatrixIR.scala:28)
at is.hail.expr.ir.LowerMatrixIR$.apply(LowerMatrixIR.scala:13)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:104)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:77)
at is.hail.expr.ir.Interpret$.interpretPyIR(Interpret.scala:31)
at is.hail.expr.ir.Interpret$.interpretPyIR(Interpret.scala:25)
at is.hail.expr.ir.Interpret.interpretPyIR(Interpret.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)



Hail version: 0.2.4-82d3af919183
Error summary: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}

When I execute part of the code within ipython, I receive the following error:
I loaded the matrix table ’ truth-sets/hail-0.2/hybrid.m37m.mt’ and performed a repartition.

In [2]: mt = hl.read_matrix_table('/home/tim/readme/hail_scripts/basics-public/truth-sets/hail-0.2/hybrid.m37m.mt').repartition(1000) 
   ...:                                                                                                                                                     

In [4]: mt.rows().show()                                                                                                                                    
---------------------------------------------------------------------------
FatalError                                Traceback (most recent call last)
<ipython-input-4-ea8938ea6669> in <module>
----> 1 mt.rows().show()

<decorator-gen-807> in show(self, n, width, truncate, types, handler)

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
	558	def wrapper(__original_func, *args, **kwargs):
	559	    args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
--> 560	    return __original_func(*args_, **kwargs_)
	561 
	562	return wrapper

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py in show(self, n, width, truncate, types, handler)
   1187		Handler function for data string.
   1188	    """
-> 1189	    handler(self._show(n,width, truncate, types))
   1190 
   1191	def _show(self, n=10, width=90, truncate=None, types=True):

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py in _show(self, n, width, truncate, types)
   1190 
   1191     def _show(self, n=10, width=90, truncate=None, types=True):
-> 1192         return self._jt.showString(n, joption(truncate), types, width)
   1193 
   1194     def index(self, *exprs):

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/utils/java.py in deco(*args, **kwargs)
    208             raise FatalError('%s\n\nJava stack trace:\n%s\n'
    209                              'Hail version: %s\n'
--> 210                              'Error summary: %s' % (deepest, full, hail.__version__, deepest)) from None
    211         except pyspark.sql.utils.CapturedException as e:
    212             raise FatalError('%s\n\nJava stack trace:\n%s\n'

FatalError: AssertionError: assertion failed

Java stack trace:
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:156)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:35)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$1.apply(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$1.apply(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.apply(LowerMatrixIR.scala:14)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:50)
at is.hail.table.Table.x$3$lzycompute(Table.scala:216)
at is.hail.table.Table.x$3(Table.scala:216)
at is.hail.table.Table.value$lzycompute(Table.scala:216)
at is.hail.table.Table.value(Table.scala:216)
at is.hail.table.Table.rdd$lzycompute(Table.scala:220)
at is.hail.table.Table.rdd(Table.scala:220)
at is.hail.table.Table.collect(Table.scala:538)
at is.hail.table.Table.showString(Table.scala:582)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)



Hail version: 0.2.4-82d3af919183
Error summary: AssertionError: assertion failed

[tpoterba]

I think I see the problem. We broke compatibility with files produced by an ancient version.

Will have a fix soon.

[tpoterba]

will solve your problem:

https://github.com/hail-is/hail/pull/4889

We had fixed this in the past and then broke it recently once more.

We have a test now so that will not happen again

[timstrom]

Thank you. Can I already download or should I wait? Was my modification
of the code correct?

The second error is also in generate_qc_annotations.py. This part has been
modified several times in the last months.

The command is:

python  /home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py --overwrite --genomes --generate_qc_annotations

The code is (line with error indicated by !):

for group in sample_group_filters.keys():
    criteria = mt[group]
    hets = criteria & mt.GT.is_het()
    non_refs = criteria & mt.GT.is_non_ref()
    nrq = -hl.log10(gp0)

    het_length = hl.agg.count_where(hets)

  ! qual_agg = -10 * hl.agg.sum(hl.agg.filter(non_refs, hl.cond(
  !     mt.PL[0] > 3000, -300, hl.log10(gp0)
  ! )))

The traceback is:

 Traceback (most recent call last):
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 304, in <module>
 	 main(args)
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 249, in main
 	 mt = generate_qc_annotations(mt, all_annotations=args.calculate_all_annotations, medians=args.calculate_medians)
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 87, in generate_qc_annotations
 	 mt.PL[0] > 3000, -300, hl.log10(gp0)
   File "<decorator-gen-884>", line 2, in filter
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 559, in wrapper
 	 args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 487, in check_all
 	 args_.append(checker.check(arg, name, arg_name))
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/expr/aggregators/aggregators.py", line 26, in check
 	 assert(len(x._ir.search(lambda node: isinstance(node, BaseApplyAggOp))) != 0)
 AssertionError

[tpoterba]

The fix isn’t in yet - that pull request hasn’t been merged.

re: the other error, see this post: [BREAKING CHANGE] redesign of aggregator interface

You need to put hl.agg.filter on the outside now.

[timstrom]

I made the following change in generate_qc_annotations.py.
Do you know if and how this and other changes should be incorporated into
https://github.com/macarthur-lab/gnomad_qc?

    #qual_agg = -10 * hl.agg.sum(hl.agg.filter(non_refs, hl.cond(
    #    mt.PL[0] > 3000, -300, hl.log10(gp0)
    #)))
    qual_agg = -10 * hl.agg.filter(non_refs, hl.agg.sum(hl.cond(
        mt.PL[0] > 3000, -300, hl.log10(gp0)
    )))

[tpoterba]

this conversation on zulip discusses the same thing:

hail

I think the gnomAD team will update the library, yes.

[timstrom]

I also propose the following change in generate_qc_annotations.py

#return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt[mt.row_key, :])
#                                                for root, truth_mt in truth_mtes.items()})).rows()
return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt.rows()[mt.row_key])
                                                for root, truth_mt in truth_mtes.items()})).rows()

[timstrom]

Is the update of gnomad-public finished?

                             gs://gnomad-public/intervals/
                             gs://gnomad-public/legacy/
                             gs://gnomad-public/release/
                             gs://gnomad-public/resources/
                             gs://gnomad-public/tools/
                             gs://gnomad-public/truth-sets/

[tpoterba]

which update? The code update?

I don’t think so.

[timstrom]

I understood that you are going to update the vds-files in gnomad-public.

You wrote
will solve your problem:

Re-fix compatibility with files with required global type

[timstrom]

Sorry, I should update hail. Is that correct?

[tpoterba]

Yes, sorry I wasn’t clear – I meant updating the code, not the files.

The fix is now in the git mainline, and deployed to google cloud. For pip, we’ll need to wait until 0.2.5 is released today or tomorrow.