# Appending to an existing Matrixtable
**URL:** https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956
**Category:** Hail Query & hailctl
**Created:** [March 15, 2021, 2:40pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956 "2021-03-15T14:40:21Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@hailstorm](https://discuss.hail.is/u/hailstorm)
#### Post date: [March 15, 2021, 2:40pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/1 "2021-03-15T14:40:21Z")
Hi,
I had a problem statement and wanted to check with the community as to what will be an ideal solution for this.
1. I have an existing matrix table with say 5000 vcf files imported in it.
2. I would like to append another vcf file to the above Mt
3. Perform analysis say regression on the merged dataset and save it to disk.
4. Repeat #2 and #3 many times.
The dataset will grow over time as each vcf file added subsequently is appended to the Mt.
Would union be an option or is there any other efficient solution? Thanks in advance!
---
### Author:  [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [March 15, 2021, 2:43pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/2 "2021-03-15T14:43:57Z")
To answer this, we need more information about these input VCFs. Are these project VCFs (GT, GQ, etc FORMAT fields) for a group of samples from sequencing data? Those cannot be losslessly combined (a site might appear in one VCF but not another).
If your VCFs are genotype data, it’s probably possible to combine since those have the same set of variants, and in that case I think `union_cols` is probably what you want.
---
### Author:  [@hailstorm](https://discuss.hail.is/u/hailstorm)
#### Post date: [March 15, 2021, 2:50pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/3 "2021-03-15T14:50:51Z")
Thanks @johnc1231. The VCFs are from genotyping. I tried implementing union\_cols but hail does not permit overwriting to existing Mt (get an error that input and output query are the same). I am now creating a temporary Mt which is a union of the old Mt and the new VCF Mt every time a new VCF needs to be imported.
Was wondering if there is a more efficient solution out there.
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [March 15, 2021, 2:57pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/4 "2021-03-15T14:57:48Z")
> [@hailstorm](#):
>
> Was wondering if there is a more efficient solution out there.
It’s possible that once the MT is large (~100s of thousands of samples), that writing for each new VCF is going to be the dominating expense. In that case, you might consider not writing each time, but instead building a sort of [log-structured merge tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) where you only write the merged dataset when you hit some threshold, and do the regressions by joining with union\_cols when the new samples come in.
I think the simple solution you’ve proposed will probably scale to half a million samples reasonably well, though.
---
### Author:  [@hailstorm](https://discuss.hail.is/u/hailstorm)
#### Post date: [March 15, 2021, 3:12pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/5 "2021-03-15T15:12:18Z")
Thanks @tpoterba. I am on a 4 core CPU with 8GB RAM and it took me ~4 hours to do a union of 100 VCFs so far. Is this normal?
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [March 15, 2021, 3:20pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/6 "2021-03-15T15:20:39Z")
Is each VCF a single sample, or a batch of samples?
---
### Author:  [@hailstorm](https://discuss.hail.is/u/hailstorm)
#### Post date: [March 15, 2021, 3:26pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/7 "2021-03-15T15:26:15Z")
Each VCF is a single sample.
---
### Author:  [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [March 15, 2021, 4:02pm UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/8 "2021-03-15T16:02:28Z")
Are they gVCFs?
---
### Author:  [@hailstorm](https://discuss.hail.is/u/hailstorm)
#### Post date: [March 16, 2021, 1:30am UTC](https://discuss.hail.is/t/appending-to-an-existing-matrixtable/1956/9 "2021-03-16T01:30:28Z")
They are not gVCFs.