# \[Hail 0.2\] Merge two MatrixTable **URL:** \ **Category:** Help \[0.1\]\ **Created:** [June 22, 2018, 6:48pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551 "2018-06-22T18:48:25Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![In-Hee\_Lee](https://avatars.discourse-cdn.com/v4/letter/i/5f8ce5/32.png) [@In-Hee\_Lee](https://discuss.hail.is/u/In-Hee_Lee)\ **Post date:** [June 22, 2018, 6:48pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/1 "2018-06-22T18:48:25Z")
Hello. I’m new to Hail and trying to find a way to merge two MatrixTables. I have two MatrixTables (each created from import\_vcf function). They don’t have overlapping samples, but do have overlapping variants. Is there a way to merge two MatrixTables in Hail 0.2? Hail 0.1 seems to have a way to join two VariantDataset, but I’d rather stay with Hail 0.2… Awaiting any suggestions… Thank you. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 22, 2018, 7:06pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/2 "2018-06-22T19:06:35Z")
is [https://hail.is/docs/devel/hail.MatrixTable.html#hail.MatrixTable.union\_cols](https://hail.is/docs/devel/hail.MatrixTable.html#hail.MatrixTable.union_cols) what you’re looking for? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 22, 2018, 7:06pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/3 "2018-06-22T19:06:57Z")
VariantDataset.join (0.1) and MatrixTable.union\_cols (0.2) are identical. ---
**Author:** ![In-Hee\_Lee](https://avatars.discourse-cdn.com/v4/letter/i/5f8ce5/32.png) [@In-Hee\_Lee](https://discuss.hail.is/u/In-Hee_Lee)\ **Post date:** [June 22, 2018, 7:38pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/4 "2018-06-22T19:38:57Z")
From the union\_cols() document, only rows common to both dataset will be kept… What I want is to have a MatrixTable with union of rows & union of cols… ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 22, 2018, 7:55pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/5 "2018-06-22T19:55:44Z")
There is no way to do this in either 0.1 or 0.2 currently. It’s coming soon. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 22, 2018, 7:56pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/6 "2018-06-22T19:56:10Z")
could you describe what the result should look like? For genotypes that appear in both? in one? ---
**Author:** ![In-Hee\_Lee](https://avatars.discourse-cdn.com/v4/letter/i/5f8ce5/32.png) [@In-Hee\_Lee](https://discuss.hail.is/u/In-Hee_Lee)\ **Post date:** [June 22, 2018, 9:27pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/7 "2018-06-22T21:27:14Z")
The result I’m looking for would be: - The resulting MatrixTable should have all genotypes appeared in any of input MatrixTables. (ideally without duplications, but it’s okay not deduplicated.) - The resulting MatrixTable have all samples appeared in any of input MatrixTables. (again, hopefully without duplications, but can deduplicate samples later) Does this make sense? Thank you. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 23, 2018, 1:31pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/8 "2018-06-23T13:31:12Z")
This sounds like an outer join on rows and columns, which we’re working on and should appear in the next 4-8 weeks I think. ---
**Author:** ![jjfarrell](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/jjfarrell/32/108_2.png) [@jjfarrell](https://discuss.hail.is/u/jjfarrell)\ **Post date:** [November 13, 2019, 1:23pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/9 "2019-11-13T13:23:32Z")
Has there been any progress on a Hail merge feature for VCFs or matrix tables? I am looking for similar functionality as the bcftools merge. SV programs run on single samples. So we have 5000 vcfs for each sample and SV pipeline that we would like to merge together. We would like every variant site and sample in the resulting vcf/matrix table. There are options on how to set the filter field andmissing genotypes in bcftools merge that would also be useful. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [November 14, 2019, 3:48am UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/10 "2019-11-14T03:48:14Z")
We do have the aforementioned outer join function `hl.experimental.outer_join_mt` (which I’ve just now realized doesn’t appear in the docs; I’ll fix that). We have been working on building a scalable joint calling / genotype gVCFs algorithm in Hail, which is more in line with what you’re doing, I’m guessing. What do the SV VCFs look like? ---
**Author:** ![jjfarrell](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/jjfarrell/32/108_2.png) [@jjfarrell](https://discuss.hail.is/u/jjfarrell)\ **Post date:** [November 16, 2019, 7:51pm UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/11 "2019-11-16T19:51:35Z")
Thanks! I found this in the documentation: hail.experimental.full\_outer\_join\_mt. Is that the function that I should be using? The SV programs generate a standard vcf file for each sample so there is no gVCF-like file like in the gatk pipelines. There are SV specs in VCF 4.2 ([https://samtools.github.io/hts-specs/VCFv4.2.pdf](https://samtools.github.io/hts-specs/VCFv4.2.pdf)). The sample SV sites are then compiled into one comprehensive list and then that site list genotyped in each sample from each samples cram individually. This step adds the homozygous ref 0/0 calls with some quality scores. The individual VCFs are then merged. How the alt is specified varies between programs. Some will use ~~and other abbreviations specified in the vcf specs to minimize the space required to include the sequeunce.~~ ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [November 19, 2019, 10:28am UTC](https://discuss.hail.is/t/hail-0-2-merge-two-matrixtable/551/12 "2019-11-19T10:28:06Z")
You could certainly implement something with that function, but I think there’s probably no easy way to join 5000 VCFs in this manner right now – it’s essentially a data transpose. It’s possible that doing a hierarchical `full_outer_join_mt` will perform adequately, as long as it’s broken up into ~2 pipelines (13 merges will be required to get to 5k, by log2(5000)), so doing 6-7 levels in each should work.