# Avoiding duplicate rows after MatrixTable.union\_rows

**URL:** <https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887>\
**Category:** Hail Query & hailctl\
**Created:** [March 1, 2019, 1:46pm UTC](https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887 "2019-03-01T13:46:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hhx037](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/hhx037/32/160_2.png) [@hhx037](https://discuss.hail.is/u/hhx037)\
**Post date:** [March 1, 2019, 1:46pm UTC](https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887/1 "2019-03-01T13:46:17Z")

</div>

I want to merge two genotype sets for the same cohort using MatrixTable.union\_rows, but want to avoid duplicate rows. I previously tried to make a list of variants in the first set, and exclude them from the second set before merging, but there are 13 million variants in common, and the massive shuffle during the exclusion process crashed.

Is there a cleaner way to only merge from the second set the variants which are not already in the first set?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [March 1, 2019, 1:53pm UTC](https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887/2 "2019-03-01T13:53:38Z")

</div>

something like this:

```python
mt3 = mt1.union_rows(mt2.filter_rows(hl.is_missing(mt1.rows()[mt2.row_key])))

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [March 1, 2019, 1:53pm UTC](https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887/3 "2019-03-01T13:53:45Z")

</div>

union\_rows shouldn’t require a shuffle

---

<div class="post-metadata">

**Author:** ![hhx037](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/hhx037/32/160_2.png) [@hhx037](https://discuss.hail.is/u/hhx037)\
**Post date:** [March 1, 2019, 3:15pm UTC](https://discuss.hail.is/t/avoiding-duplicate-rows-after-matrixtable-union-rows/887/4 "2019-03-01T15:15:46Z")

</div>

Brilliant, thank you very much, I’ll try this 🙂