# \`filter\_entries\` introduces NA instead of removing entries from a matrixtable

**URL:** <https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382>\
**Category:** Hail Query & hailctl\
**Created:** [May 10, 2023, 9:04pm UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382 "2023-05-10T21:04:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![faizal\_eeman](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@faizal\_eeman](https://discuss.hail.is/u/faizal_eeman)\
**Post date:** [May 10, 2023, 9:04pm UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382/1 "2023-05-10T21:04:46Z")

</div>

I’m trying to filter out variants with AF \< 0.8 from a matrixtable. I can of course use `hl.variant_qc` with `mt.filter_rows` to get this done but my VCF has AF fields with values 0, 0.5 and 1.0 which is not helpful for a fine filtering.

So I created a filter condition using AD and DP values in the `entries` field to filter out variants.

```python
mt = mt.filter_entries(mt.AD[1]/mt.DP > 0.8)

```

However, when I see the outputs I see `NA` has been assigned where the filter condition evaluated to `False`. See below,

```python
mt.AD.show()

+---------------+------------+--------------+
| locus | alleles | 'test'.AD |
+---------------+------------+--------------+
| locus<GRCh38> | array<str> | array<int32> |
+---------------+------------+--------------+
| chrX:22849 | ["A","G"] | [0,22] |
| chrX:26601 | ["G","T"] | NA |
| chrX:26883 | ["C","T"] | [0,89] |
| chrX:26987 | ["A","C"] | NA |
| chrX:27266 | ["C","G"] | NA |
+---------------+------------+--------------+

```

I’m not sure why I see NAs when I expect the variants to be removed from `mt`.

I also tried to export `mt` as a VCF and I still see variants with AF\<0.8 but the sample field is replaced with `./.`

I basically want to get rid of variants which fail for my filtering condition (mt.AD[1]/mt.DP \> 0.8) but I’m not sure what I’m missing in my implementation.

Any thoughts would be appreciated!

Faizal

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [May 11, 2023, 1:57am UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382/2 "2023-05-11T01:57:57Z")

</div>

You’ll need to use `filter_rows` instead of `filter_entries` to remove rows (variants) instead of FORMAT field groups (entries).

If you only have a single column/sample, I understand why this is confusing, and matrixtable is probably not the right interface for you to use. If you do:

```python
ht = mt.make_table()

```

You can filter your table with `ht.filter(ht['test.AD'] / ht['test.DP'] > 0.8)`

---

<div class="post-metadata">

**Author:** ![faizal\_eeman](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@faizal\_eeman](https://discuss.hail.is/u/faizal_eeman)\
**Post date:** [May 12, 2023, 2:34am UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382/3 "2023-05-12T02:34:20Z")

</div>

Cool, this worked. Appreciate the clarification!

---

<div class="post-metadata">

**Author:** ![faizal\_eeman](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@faizal\_eeman](https://discuss.hail.is/u/faizal_eeman)\
**Post date:** [May 16, 2023, 3:13am UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382/4 "2023-05-16T03:13:05Z")

</div>

Follow up issue:  
After filtering the table and exporting it as a VCF file, I see the output VCF doesn’t have the FORMAT field. I understand hail doesn’t add FORMAT field while exporting from a hail table. But is there a work-around to add the FORMAT field to the output VCF?

Appreciate your help!

Faizal

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [May 16, 2023, 3:54am UTC](https://discuss.hail.is/t/filter-entries-introduces-na-instead-of-removing-entries-from-a-matrixtable/3382/5 "2023-05-16T03:54:35Z")

</div>

The interface is pretty rough here but I’d construct a single field, sample1, that is an hl.struct with all the genotype fields then use [Hail | Table](https://hail.is/docs/0.2/hail.Table.html#hail.Table.to_matrix_table_row_major) to convert back to MT and then use standard MT export.