# Quality Control, filtering false positive variants

**URL:** https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600
**Category:** Science
**Created:** [August 8, 2018, 1:37pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600 "2018-08-08T13:37:00Z")
**Posts on this page:** 18
**Page:** 1

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [August 8, 2018, 1:37pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/1 "2018-08-08T13:37:00Z")

</div>

The German Center for Heart Research (DZHK) is analyzing 1205 genomes  
from 5 general population cohorts (KORA, SHIP, Hamburg City, Mainz City,  
Uni Heidelberg) plus 548 genomes from patients with coronary heart  
disease or cardiomyopathy. Sequencing was performed at the DKFZ on  
HiSeq X machines,

We would like to analyze the data in a way that is compatible with  
an integration of these data into gnomAD. We would be very grateful if  
you could support us on technical details.

Sequencing depth is at least 30 fold (mean 40).

We are currently analyzing the data according to the GATK best  
practices guidelines and perform statistics for quality control  
in hail.

For QC we are using 24 trios (sequenced on HiSeq4000). We  
used GSA arrays and overlap with gnomAD variants as measures for sensitivity  
and Ti/TV ratio and singletons in offspring of trios as measures for  
specificity.

We evaluated two settings for base quality score recalibration  
(SNV\_context 2 or 4, indel\_context 3 or 8). Variant quality  
score recalibration was done with 1kg or gnomAD (see link).

We conclude from the number of singletons that the use of  
SNV\_context 4 and indel\_context 8 is superior to 2 and 4, respectively.

But unfortunately, we observed that we had too many singletons  
after VQSR (many thousands). Our attempt to use hard thresholds  
on GQ, alternative read fraction, and/or MQ reduced the number  
of singletons, but also the overlap with gnomAD SNVs. We concluded  
that this approach is not feasible.

When we inspect the BAM files around the singletons,  
we noticed that a large fraction is located in or near of  
homopolymers or tandem repeats with many soft-clipped reads.  
Visual inspection would relatively easily identify this false  
positives.

We have the impression that the random forest approach used  
in gnomAD filters out these ‘dirty’ regions much better than  
VQSR does.

We would like to ask if you could recommend how to proceed further.

Tim

More details about QC can be found at

> **[2018-06-14-dzhk.pdf](https://ihg.helmholtz-muenchen.de/dzhk/2018-06-14-dzhk.pdf)**
>
> 609.83 KB

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [August 8, 2018, 1:55pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/2 "2018-08-08T13:55:15Z")

</div>

@lfrancioli @konradjk any ideas? Is the random forest code up somewhere and compatible with 0.2?

---

<div class="post-metadata">

### Author: ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)
#### Post date: [August 8, 2018, 3:32pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/3 "2018-08-08T15:32:48Z")

</div>

We have an RF module with some helper functions here: [https://github.com/macarthur-lab/gnomad\_hail/blob/master/utils/rf.py](https://github.com/macarthur-lab/gnomad_hail/blob/master/utils/rf.py)

The code to use those functions is not yet public but will be (we’re currently actively tweaking it). @lfrancioli can comment further as to the exact usage.

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 4, 2018, 3:31pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/4 "2018-12-04T15:31:50Z")

</div>

I got part of the gnomad\_qc scripts running with older versions of hail, gnomad\_qc and gnomad\_hail (hail from October 2108 Hail-devel-b0342b11e1af).

I tried to update to the newest precompiled version of hail (0.2.4-82d3af919183) and the 2nd December version of gnomad\_qc and gnomad\_hail.

I am receiving several errors in generate\_qc\_annotations.py older versions of which run without problems.

For example:

1. When running ‘–generate\_qc\_annotations’, I receive the following error:

Traceback …  
in generate\_qc\_annotations mt.PL[0] \> 3000, -300, hl.log10(gp0)  
… aggregators.py", line 26, in check assert(len(x.\_ir.search(lambda node:  
isinstance(node, BaseApplyAggOp))) != 0) AssertionError

1. When running ‘–annotate\_truth\_data’, truth\_data can not be repartitioned:

Traceback …  
in main annotate\_truth\_data(mt).write(annotations\_ht\_path(data\_type, ‘truth\_data’), args.overwrite)  
…AssertionError: assertion failed: Table{global:Struct{\_\_cols:Array[Struct{s:String}]},key:[locus,alleles]…

I downloaded the truth\_data from gs://gnomad-public.

Is it more likely that these are version problems or problems of our vcf data?

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 4, 2018, 5:37pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/5 "2018-12-04T17:37:22Z")

</div>

Can we have the full stack trace for these errors? Hard to say just based on the first bit.

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 4, 2018, 7:17pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/6 "2018-12-04T19:17:07Z")

</div>

This is code from ‘generate\_qc\_annotations.py’ and how I modified it. Truth data is downloaded from gs://gnomad-public.

This is the command :

```
python /home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py --overwrite --genomes --annotate_truth_data

```

This is the code (modification is indicated). I think, the error occurs before the modification.

```
def annotate_truth_data(mt: hl.MatrixTable) -> hl.Table:
"""
Writes bi-allelic sites MT with the following annotations:
 - truth_data (presence in Omni, HapMap, 1KG high conf SNVs, Mills)

:param MatrixTable mt: Full MT
:return: Table with qc annotations
:rtype: Table
"""
mt = mt.select_rows()

truth_mtes = {
    'hapmap': hapmap_mt_path(),
    'omni': omni_mt_path(),
    'mills': mills_mt_path(),
    'kgp_high_conf_snvs': kgp_high_conf_snvs_mt_path()
}
truth_mtes = {key: hl.split_multi_hts(hl.read_matrix_table(path).repartition(1000), left_aligned=False)
              for key, path in truth_mtes.items()}
#tim
#return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt[mt.row_key, :])
# for root, truth_mt in truth_mtes.items()})).rows()
return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt.rows()[mt.row_key])
                                                for root, truth_mt in truth_mtes.items()})).rows()

```

This modified code is working with a October version of hail.

Currently I think, the error is in ‘hl.read\_matrix\_table(path).repartition(1000)’ (see below)  
I am receiving the following traceback:

```
Traceback (most recent call last):
  File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 304, in <module>
    main(args)
  File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 276, in main
    annotate_truth_data(mt).write(annotations_ht_path(data_type, 'truth_data'), args.overwrite)
  File "<decorator-gen-690>", line 2, in write
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 560, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py", line 1153, in write
    Env.hc()._backend.interpret(TableWrite(self._tir, output, overwrite, stage_locally, _codec_spec))
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/backend/backend.py", line 25, in interpret
    result = Env.hail().expr.ir.Interpret.interpretPyIR(code, {}, ir_map)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/utils/java.py", line 210, in deco
    'Error summary: %s' % (deepest, full, hail. __version__ , deepest)) from None
hail.utils.java.FatalError: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}

Java stack trace:
java.lang.AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
at scala.Predef$.assert(Predef.scala:170)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:42)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:96)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1$$anonfun$applyOrElse$1.apply(LowerMatrixIR.scala:164)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1$$anonfun$applyOrElse$1.apply(LowerMatrixIR.scala:164)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:164)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:150)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:196)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:24)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:41)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$2.apply(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:196)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$.lower(LowerMatrixIR.scala:28)
at is.hail.expr.ir.LowerMatrixIR$.apply(LowerMatrixIR.scala:13)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:104)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:77)
at is.hail.expr.ir.Interpret$.interpretPyIR(Interpret.scala:31)
at is.hail.expr.ir.Interpret$.interpretPyIR(Interpret.scala:25)
at is.hail.expr.ir.Interpret.interpretPyIR(Interpret.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.4-82d3af919183
Error summary: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{}],__uid_1:Array[Struct{alleles:Array[String],locus:Locus(GRCh37),a_index:Int32,was_split:Boolean}]}}

```

When I execute part of the code within ipython, I receive the following error:  
I loaded the matrix table ’ truth-sets/hail-0.2/hybrid.m37m.mt’ and performed a repartition.

```
In [2]: mt = hl.read_matrix_table('/home/tim/readme/hail_scripts/basics-public/truth-sets/hail-0.2/hybrid.m37m.mt').repartition(1000) 
   ...:                                                                                                                                                     

In [4]: mt.rows().show()                                                                                                                                    
---------------------------------------------------------------------------
FatalError Traceback (most recent call last)
<ipython-input-4-ea8938ea6669> in <module>
----> 1 mt.rows().show()

<decorator-gen-807> in show(self, n, width, truncate, types, handler)

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
	558	def wrapper(__original_func, *args, **kwargs):
	559 args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
--> 560 return __original_func(*args_, **kwargs_)
	561 
	562	return wrapper

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py in show(self, n, width, truncate, types, handler)
   1187 Handler function for data string.
   1188 """
-> 1189 handler(self._show(n,width, truncate, types))
   1190 
   1191	def _show(self, n=10, width=90, truncate=None, types=True):

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/table.py in _show(self, n, width, truncate, types)
   1190 
   1191 def _show(self, n=10, width=90, truncate=None, types=True):
-> 1192 return self._jt.showString(n, joption(truncate), types, width)
   1193 
   1194 def index(self, *exprs):

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/py4j/java_gateway.py in __call__ (self, *args)
   1255 answer = self.gateway_client.send_command(command)
   1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259 for temp_arg in temp_args:

/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/utils/java.py in deco(*args, **kwargs)
    208 raise FatalError('%s\n\nJava stack trace:\n%s\n'
    209 'Hail version: %s\n'
--> 210 'Error summary: %s' % (deepest, full, hail. __version__ , deepest)) from None
    211 except pyspark.sql.utils.CapturedException as e:
    212 raise FatalError('%s\n\nJava stack trace:\n%s\n'

FatalError: AssertionError: assertion failed

Java stack trace:
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:156)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:35)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:23)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$3.apply(LowerMatrixIR.scala:47)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lowerChildren(LowerMatrixIR.scala:47)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$1.apply(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$1.apply(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$tableRules$1.applyOrElse(LowerMatrixIR.scala:191)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:34)
at is.hail.expr.ir.LowerMatrixIR$.apply(LowerMatrixIR.scala:14)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:50)
at is.hail.table.Table.x$3$lzycompute(Table.scala:216)
at is.hail.table.Table.x$3(Table.scala:216)
at is.hail.table.Table.value$lzycompute(Table.scala:216)
at is.hail.table.Table.value(Table.scala:216)
at is.hail.table.Table.rdd$lzycompute(Table.scala:220)
at is.hail.table.Table.rdd(Table.scala:220)
at is.hail.table.Table.collect(Table.scala:538)
at is.hail.table.Table.showString(Table.scala:582)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.4-82d3af919183
Error summary: AssertionError: assertion failed
```

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 4, 2018, 7:37pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/7 "2018-12-04T19:37:07Z")

</div>

I think I see the problem. We broke compatibility with files produced by an ancient version.

Will have a fix soon.

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 4, 2018, 7:52pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/8 "2018-12-04T19:52:17Z")

</div>

will solve your problem:

[https://github.com/hail-is/hail/pull/4889](https://github.com/hail-is/hail/pull/4889)

We had fixed this in the past and then broke it recently once more.

We have a test now so that will not happen again 🙂

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 4, 2018, 8:11pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/9 "2018-12-04T20:11:45Z")

</div>

Thank you. Can I already download or should I wait? Was my modification  
of the code correct?

The second error is also in generate\_qc\_annotations.py. This part has been  
modified several times in the last months.

The command is:

```
python /home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py --overwrite --genomes --generate_qc_annotations

```

The code is (line with error indicated by !):

```
for group in sample_group_filters.keys():
    criteria = mt[group]
    hets = criteria & mt.GT.is_het()
    non_refs = criteria & mt.GT.is_non_ref()
    nrq = -hl.log10(gp0)

    het_length = hl.agg.count_where(hets)

  ! qual_agg = -10 * hl.agg.sum(hl.agg.filter(non_refs, hl.cond(
  ! mt.PL[0] > 3000, -300, hl.log10(gp0)
  ! )))

```

The traceback is:

```
 Traceback (most recent call last):
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 304, in <module>
 	 main(args)
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 249, in main
 	 mt = generate_qc_annotations(mt, all_annotations=args.calculate_all_annotations, medians=args.calculate_medians)
   File "/home/tim/readme/hail_scripts/gnomad_qc/annotations/generate_qc_annotations.py", line 87, in generate_qc_annotations
 	 mt.PL[0] > 3000, -300, hl.log10(gp0)
   File "<decorator-gen-884>", line 2, in filter
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 559, in wrapper
 	 args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/typecheck/check.py", line 487, in check_all
 	 args_.append(checker.check(arg, name, arg_name))
   File "/usr/local/packages/anaconda3/envs/hail/lib/python3.7/site-packages/hail/expr/aggregators/aggregators.py", line 26, in check
 	 assert(len(x._ir.search(lambda node: isinstance(node, BaseApplyAggOp))) != 0)
 AssertionError
```

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 4, 2018, 9:51pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/10 "2018-12-04T21:51:02Z")

</div>

The fix isn’t in yet - that pull request hasn’t been merged.

re: the other error, see this post: [[BREAKING CHANGE] redesign of aggregator interface](http://discuss.hail.is/t/breaking-change-redesign-of-aggregator-interface/701)

You need to put `hl.agg.filter` on the outside now.

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 5, 2018, 12:25pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/11 "2018-12-05T12:25:02Z")

</div>

I made the following change in generate\_qc\_annotations.py.  
Do you know if and how this and other changes should be incorporated into  
[https://github.com/macarthur-lab/gnomad\_qc](https://github.com/macarthur-lab/gnomad_qc)?

```
    #qual_agg = -10 * hl.agg.sum(hl.agg.filter(non_refs, hl.cond(
    # mt.PL[0] > 3000, -300, hl.log10(gp0)
    #)))
    qual_agg = -10 * hl.agg.filter(non_refs, hl.agg.sum(hl.cond(
        mt.PL[0] > 3000, -300, hl.log10(gp0)
    )))
```

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 5, 2018, 12:47pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/12 "2018-12-05T12:47:15Z")

</div>

this conversation on zulip discusses the same thing:

> **[hail](https://hail.zulipchat.com/login/)**
>
> Hail is an open-source, scalable framework for exploring and analyzing biological data, originally developed to process some of the largest genetic datasets ever created. | Be sure to check out the docs and the forum as well!

I think the gnomAD team will update the library, yes.

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 5, 2018, 1:15pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/13 "2018-12-05T13:15:20Z")

</div>

I also propose the following change in generate\_qc\_annotations.py

```
#return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt[mt.row_key, :])
# for root, truth_mt in truth_mtes.items()})).rows()
return mt.annotate_rows(truth_data=hl.struct(**{root: hl.is_defined(truth_mt.rows()[mt.row_key])
                                                for root, truth_mt in truth_mtes.items()})).rows()
```

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 6, 2018, 11:13am UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/14 "2018-12-06T11:13:49Z")

</div>

Is the update of gnomad-public finished?

```
                             gs://gnomad-public/intervals/
                             gs://gnomad-public/legacy/
                             gs://gnomad-public/release/
                             gs://gnomad-public/resources/
                             gs://gnomad-public/tools/
                             gs://gnomad-public/truth-sets/
```

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 6, 2018, 12:50pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/15 "2018-12-06T12:50:53Z")

</div>

which update? The code update?

I don’t think so.

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 6, 2018, 1:22pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/16 "2018-12-06T13:22:03Z")

</div>

I understood that you are going to update the vds-files in gnomad-public.

You wrote  
will solve your problem:

#### [Re-fix compatibility with files with required global type](https://github.com/hail-is/hail/pull/4889)

---

<div class="post-metadata">

### Author: ![timstrom](https://avatars.discourse-cdn.com/v4/letter/t/71c47a/32.png) [@timstrom](https://discuss.hail.is/u/timstrom)
#### Post date: [December 6, 2018, 1:27pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/17 "2018-12-06T13:27:02Z")

</div>

Sorry, I should update hail. Is that correct?

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [December 6, 2018, 1:34pm UTC](https://discuss.hail.is/t/quality-control-filtering-false-positive-variants/600/18 "2018-12-06T13:34:20Z")

</div>

Yes, sorry I wasn’t clear – I meant updating the code, not the files.

The fix is now in the git mainline, and deployed to google cloud. For pip, we’ll need to wait until 0.2.5 is released today or tomorrow.