# Exception(s) when writing dense matrix after multiple merges of densify matrices

**URL:** https://discuss.hail.is/t/exception-s-when-writing-dense-matrix-after-multiple-merges-of-densify-matrices/1343
**Category:** Hail Query & hailctl
**Created:** [March 19, 2020, 9:28am UTC](https://discuss.hail.is/t/exception-s-when-writing-dense-matrix-after-multiple-merges-of-densify-matrices/1343 "2020-03-19T09:28:02Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [March 19, 2020, 11:57am UTC](https://discuss.hail.is/t/exception-s-when-writing-dense-matrix-after-multiple-merges-of-densify-matrices/1343/2 "2020-03-19T11:57:40Z")

</div>

These two errors are symptoms of the same problem – a computational graph that is too large/deep. This isn’t your fault, and we should handle these kind of massive queries gracefully by using data structures on the heap instead of using the function stack in Python and the scala backend.

How many families do you have? This part is likely triggering the depth error:

```python
denseMatrix = denseByFamily[0] 
for ii in range(1 , len( denseByFamily ) ):
    denseMatrix = full_outer_join_mt( denseMatrix, denseByFamily[ii] )
denseMatrix.write( outputDenseMatrix, overwrite = True )

```

It’s possible that redesigning this to do a tree merge could help, something like this:

> [@Performance of writing matrixtable on 0.2](https://discuss.hail.is/t/performance-of-writing-matrixtable-on-0-2/663/9):
>
> for now you can also get down to O(N \* log2 N) doing a tree union: mts\_ = mts[:] iteration = 0 while (len(mts\_) \> 1): iteration += 1 print(f'iteration {iteration}') tmp = [] for i in range(0, len(mts\_), 2): tmp.append(hl.MatrixTable.union\_cols(mts\_[i:i+2])) mts\_ = tmp[:] [final\_mt] = mts\_

but with full\_outer\_join\_mt instead of union\_cols.

---

_[View the full topic](https://discuss.hail.is/t/exception-s-when-writing-dense-matrix-after-multiple-merges-of-densify-matrices/1343)._