# Errors after iterative calls to union\_cols

**URL:** https://discuss.hail.is/t/errors-after-iterative-calls-to-union-cols/790
**Category:** Hail Query & hailctl
**Created:** [November 29, 2018, 12:51am UTC](https://discuss.hail.is/t/errors-after-iterative-calls-to-union-cols/790 "2018-11-29T00:51:17Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [November 29, 2018, 1:43am UTC](https://discuss.hail.is/t/errors-after-iterative-calls-to-union-cols/790/2 "2018-11-29T01:43:56Z")

</div>

1. why is this having trouble writing to the file?

It’s important to note that almost every operation in Hail is lazy – that is, everything upstream is executed when you are executing the `write`.

1. is there a better way to do the 1:many calculation for the na12878 comparisons.

I think the iterative union\_cols is quadratic in the number of samples – see this thread involving the same issue:

> [@Performance of writing matrixtable on 0.2](http://discuss.hail.is/t/performance-of-writing-matrixtable-on-0-2/663/9):
>
> for now you can also get down to O(N \* log2 N) doing a tree union: mts\_ = mts[:] iteration = 0 while (len(mts\_) \> 1): iteration += 1 print(f'iteration {iteration}') tmp = [] for i in range(0, len(mts\_), 2): tmp.append(hl.MatrixTable.union\_cols(mts\_[i:i+2])) mts\_ = tmp[:] [final\_mt] = mts\_

The issue is that Hail doesn’t have great ways right now to load a batch of single-sample VCFs into a single joint MatrixTable. We’re working on a gVCF merging algorithm, which we expect to come online in the next six months.

For the time being, things may work if you use my N \* log(N) solution in the linked comment.

---

_[View the full topic](https://discuss.hail.is/t/errors-after-iterative-calls-to-union-cols/790)._