# Filtering variants from a hail matrix present in another hail matrix **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [December 16, 2020, 2:44am UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852 "2020-12-16T02:44:24Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![emauryg](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/emauryg/32/502_2.png) [@emauryg](https://discuss.hail.is/u/emauryg)\ **Post date:** [December 16, 2020, 2:44am UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/1 "2020-12-16T02:44:24Z")
I’m trying to filter out the variants present in the controls from variants present in cases for a case control study. right now I have the following: mt\_denovo\_control = mt\_denovo.filter\_cols(mt\_denovo.is\_control == “control”) mt\_denovo\_cases = mt\_denovo.filter\_cols(mt\_denovo.is\_control == “case”) I have tried doing this: mt\_denovo\_cases = mt\_denovo\_cases.filter\_rows(hl.is\_missing(mt\_denovo\_control.rows()[mt\_denovo\_cases.row\_key])) but that just removes all the variants from the cases which I don’t think is appropriate behavior. Is there a way to just filter out the variants present in control samples from the cases hail matrix? Thank you! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 16, 2020, 3:22am UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/2 "2020-12-16T03:22:20Z")
Hey @emauryg! I generally recommend working with a single matrix table rather than two distinct datasets. I realize that sometimes data isn’t sequenced or genotyped together though! As to your question. You want to remove all variants in your “cases” dataset that are also present in the “control” dataset. The second code block you pasted does exactly that: ```auto mt_denovo_cases = mt_denovo_cases.filter_rows( hl.is_missing(mt_denovo_control.rows()[mt_denovo_cases.row_key])) ``` If, after executing that, `mt_denovo_cases.count()` indicates no remaining rows, then the rows in your “cases” dataset are a subset of those in the “controls” dataset. I’d be curious of the output of these operations: ```python print(mt_denovo_cases.count()) print(mt_denovo_control.count()) # x.anti_join_rows(y) removes rows from x whose keys appear in y print(mt_denovo_cases.anti_join_rows(mt_denovo_control.rows()).count()) print(mt_denovo_control.anti_join_rows(mt_denovo_cases.rows()).count()) ``` If the last two statements both indicate there are zero rows, then it sounds like your datasets have the exact same set of rows. In that case, I think you’ll have an easier time working with your data if you combined these Matrix Tables into one Matrix Table: ```python mt_denovo_cases = mt_denovo_cases.annotate_cols(is_case = True) mt_denovo_control = mt_denovo_control.annotate_cols(is_case = False) mt = mt_denovo_cases.union_cols(mt_denovo_control) ``` If you want to only keep rows in which at least one case has at least one non-reference allele, try this: ```python mt = mt.filter_rows( hl.agg.any( mt.is_case & (mt.GT.n_alt_alleles() > 0))) ``` ---
**Author:** ![emauryg](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/emauryg/32/502_2.png) [@emauryg](https://discuss.hail.is/u/emauryg)\ **Post date:** [December 16, 2020, 3:55am UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/3 "2020-12-16T03:55:55Z")
With the command you showed I get the following ``` (550161, 11058) (550161, 10518) (0, 11058) (0, 10518) ``` However, when I run the following, there are differences: ``` mt_denovo_cases.aggregate_entries(hl.agg.count_where(mt_denovo_cases.GT.is_non_ref())) ``` `1793` `mt_denovo_control.aggregate_entries(hl.agg.count_where(my_denovo_control.GT.is_non_ref()))` `250` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 16, 2020, 3:12pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/4 "2020-12-16T15:12:22Z")
In Hail, the presence or absence of a row in a dataset is distinct from the presence or absence of genotypes with non-reference alleles in a row. However, you can make those notions equivalent by keeping only those rows that have at least one non-reference allele: ```python mt_denovo_cases = mt_denovo_cases.filter_rows(hl.agg.any(mt.GT.is_non_ref()), keep=True) mt_denovo_controls = mt_denovo_controls.filter_rows(hl.agg.any(mt.GT.is_non_ref()), keep=True) ``` After this operation, I expect the latter two `count` operations will indicate a non-zero number of rows. ---
**Author:** ![emauryg](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/emauryg/32/502_2.png) [@emauryg](https://discuss.hail.is/u/emauryg)\ **Post date:** [December 16, 2020, 8:22pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/5 "2020-12-16T20:22:04Z")
That seems more correct thank you! ``` 2020-12-16 19:28:15 Hail: INFO: Ordering unsorted dataset with network shuffle (328, 11058) 2020-12-16 19:38:40 Hail: INFO: Ordering unsorted dataset with network shuffle (77, 10518) 2020-12-16 19:55:11 Hail: INFO: Ordering unsorted dataset with network shuffle 2020-12-16 19:58:34 Hail: INFO: Ordering unsorted dataset with network shuffle (262, 11058) 2020-12-16 20:15:23 Hail: INFO: Ordering unsorted dataset with network shuffle 2020-12-16 20:18:49 Hail: INFO: Ordering unsorted dataset with network shuffle (11, 10518) ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 16, 2020, 8:33pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/6 "2020-12-16T20:33:35Z")
Just a word of caution: “Ordering unsorted dataset with network shuffle” means Hail is doing possibly the slowest thing it can do to your data. Did your data not come from a file sorted by chromosome and position? You’ll experience _dramatically_ better performance if you save your data as a Hail Table immediately after import from this unsorted source: ```python mt_denovo_cases.write('...') # do slow re-ordering of variants once, re-use this file constantly mt_denovo_cases = hl.read_matrix_table('...') mt_denovo_cases = mt_denovo_cases.filter_rows(...) mt_denovo_cases.count() # much faster! ``` ---
**Author:** ![emauryg](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/emauryg/32/502_2.png) [@emauryg](https://discuss.hail.is/u/emauryg)\ **Post date:** [December 17, 2020, 7:29pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/7 "2020-12-17T19:29:30Z")
Thanks that does speed things up! So to merge the hail matrices after the filtering above, is it ok to just use the mt.union\_cols() function as you indicated above? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 17, 2020, 7:55pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/8 "2020-12-17T19:55:38Z")
After the filtering, you cannot `union_cols` the matrix tables because they don’t have the same variants anymore. Note that in the output you shared the number of rows is different for cases and controls. If you want a single MT, filtered to just those variants that have at least one non-reference allele in the cases, I recommend starting from the two original files and doing the following. ```python mt_denovo_cases = # import the case data mt_denovo_cases.write('...') # do slow re-ordering of variants once, re-use this file constantly mt_denovo_cases = hl.read_matrix_table('...') mt_denovo_control = # import the control data mt_denovo_control.write('...') # do slow re-ordering of variants once, re-use this file constantly mt_denovo_control = hl.read_matrix_table('...') # all data now converted to the efficient Hail MT format. mt_denovo_cases = mt_denovo_cases.annotate_cols(is_case = True) mt_denovo_control = mt_denovo_control.annotate_cols(is_case = False) mt = mt_denovo_cases.union_cols(mt_denovo_control) mt = mt.filter_rows( hl.agg.any( mt.is_case & (mt.GT.n_alt_alleles() > 0))) print(mt.count()) mt.show() ``` ---
**Author:** ![emauryg](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/emauryg/32/502_2.png) [@emauryg](https://discuss.hail.is/u/emauryg)\ **Post date:** [December 17, 2020, 7:59pm UTC](https://discuss.hail.is/t/filtering-variants-from-a-hail-matrix-present-in-another-hail-matrix/1852/9 "2020-12-17T19:59:44Z")
Great thank you!