# Store multiple vcfs into single MatrixTable

**URL:** <https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632>\
**Category:** Hail Query & hailctl\
**Created:** [September 8, 2020, 7:58pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632 "2020-09-08T19:58:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 8, 2020, 7:58pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/1 "2020-09-08T19:58:39Z")

</div>

Hello,

I have multiple VCFs files and I want to store them into a single MatrixTable locally. I want to try to use HAIL as a database, where I can anytime add new vcfs to my current dataset or load the dataset and do my statistics.  
Is this possible for multiple vcfs and how?

Thank you in advance!

Cheers,

---

<div class="post-metadata">

**Author:** ![Aks](https://avatars.discourse-cdn.com/v4/letter/a/3e96dc/32.png) [@Aks](https://discuss.hail.is/u/Aks)\
**Post date:** [September 9, 2020, 6:28am UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/2 "2020-09-09T06:28:28Z")

</div>

Hi @nonchev,

Am also currently dealing with a problem like you where I have to combine multiple VCF files amd store them in one matrix table… please do let me know as well if you find a way to do this… I guess if its one or two files, I can try the joins but the problem is I have around 200 VCF files. I can segregate them based on groups where each will have 70 odd vcf files…

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 9, 2020, 7:12pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/3 "2020-09-09T19:12:53Z")

</div>

I am still stuck on this problem. It seems impossible with the current version of HAIL.  
I have also hundreds of VCFs, which I want to store in a single MatrixTable.  
I am struggling even with the join on two vcfs with the message:  
`mt1.union_rows(mt2)`  
`ValueError: 'MatrixTable.union_rows' expects all datasets to have the same columns. Datasets 0 and 1 have different columns (or possibly different order).`  
Did you menage to join two distinct vcfs? And how?

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [September 9, 2020, 7:17pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/4 "2020-09-09T19:17:21Z")

</div>

Are these VCFs all for the same individuals, but just contain different segments of the genome? Or are they all different individuals?

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 9, 2020, 7:18pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/5 "2020-09-09T19:18:52Z")

</div>

different individuals

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [September 9, 2020, 7:31pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/6 "2020-09-09T19:31:50Z")

</div>

If all of those VCFs had the exact same variants, you could combine them with `union_cols` (columns are samples, so you want `union_cols`, not `union_rows` like you have above).

Otherwise, we have the VCF Combiner, though that requires you to actually have a bunch of single sample GVCFs: [https://hail.is/docs/0.2/experimental/vcf\_combiner.html#vcf-combiner](https://hail.is/docs/0.2/experimental/vcf_combiner.html#vcf-combiner)

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 9, 2020, 7:44pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/7 "2020-09-09T19:44:44Z")

</div>

No, the variants are not necessary the same in every vcf file…

I saw the documentation of run\_combiner and also tried it myself, but I am getting this exception:

> Hail version: 0.2.57-582b2e31b8bd  
> Error summary: HailException: invalid genotype signature: expected signatures to be identical for all inputs.  
> /path/104250.vcf.gz: +PCStruct{AD:PCArray[PInt32],DP:PInt32,GQ:PInt32,GT:PCCall,PGT:PCCall,PID:PCString,PL:PCArray[PInt32],RGQ:PInt32,SB:PCArray[PInt32]}  
> /path/118008.vcf.gz: +PCStruct{AD:PCArray[PInt32],DP:PInt32,F1R2:PCArray[PInt32],F2R1:PCArray[PInt32],GQ:PInt32,GT:PCCall,PGT:PCCall,PID:PCString,PL:PCArray[PInt32],PS:PInt32,RGQ:PInt32,SB:PCArray[PInt32]}

I am not sure how I can fix this… when I use import\_vcf for each vcf individually everything is fine

---

<div class="post-metadata">

**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\
**Post date:** [September 9, 2020, 8:10pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/8 "2020-09-09T20:10:08Z")

</div>

If you look at your VCFs, they have different genotype signatures. Like it looks like one has a F1R2 field and the other doesn.t

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 9, 2020, 8:13pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/9 "2020-09-09T20:13:43Z")

</div>

Ok, but the vcfs files are not standardised and I cannot expect that all of them have the exact same structure.  
Is there a way just do add a dummy value like ‘null’ and just collect fields of interest like DP,GQ, GT?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [September 9, 2020, 8:14pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/10 "2020-09-09T20:14:27Z")

</div>

The VCF combiner requires the same signature. I think if you make one VCF header file that has the union of all fields in all input GVCFs you can use that header for each file with `header_file=...`

---

<div class="post-metadata">

**Author:** ![nonchev](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nonchev](https://discuss.hail.is/u/nonchev)\
**Post date:** [September 9, 2020, 8:19pm UTC](https://discuss.hail.is/t/store-multiple-vcfs-into-single-matrixtable/1632/11 "2020-09-09T20:19:42Z")

</div>

Ok, I got the idea, but is there a way to select in the header, which fields to be collected and just to ignore the rest? I am concerned that maybe in the future I would get a vcf file with a new unknown field to my previous files and than I would have to rerun everything, just because this one field, which I even do not need…

I have a second question about run\_combiner():  
Is it just for one run to combine the given vcfs and the next time it will just overwrite the saved matrix in **out\_file** or it can combine also the given vcfs with the old matrix in the **out\_file**?  
EDIT: I tried run\_combiner, but It can only combine single sample vcfs files **once** and in the future I cannot add more vcfs to the saved matrixtable. correct?

Are you planning in the future to add this feature, so we can just store together different vcfs in a single matrix table, which can be updated with new information(vcfs)?
