# Densify sparse mt

**URL:** <https://discuss.hail.is/t/densify-sparse-mt/1177>\
**Category:** Hail Query & hailctl\
**Created:** [November 19, 2019, 2:18pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177 "2019-11-19T14:18:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ch-kr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ch-kr/32/317_2.png) [@ch-kr](https://discuss.hail.is/u/ch-kr)\
**Post date:** [November 19, 2019, 2:18pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/1 "2019-11-19T14:18:28Z")

</div>

I’m trying to densify a sparse mt of over 200,000 samples and 2787 partitions. This is the code I’m trying to run:

```auto
    hl.init(default_reference='GRCh38', log='/densify_sparse_mt.log')
    freeze = args.freeze

    logger.info('Reading in input mt (raw sparse mt)')
    mt = hl.read_matrix_table(args.input)
    logger.info(f'Sparse mt count: {mt.count()}')

    logger.info('Densifying mt')
    mt = hl.experimental.densify(mt)

    logger.info('Filtering out lines that are only reference')
    mt = mt.filter_rows(hl.len(mt.alleles) > 1)
    logger.info(f'Count after filtration: {mt.count()}')

    logger.info('Writing out mt')
    mt = mt.repartition(30000)
    mt.write(raw_mt_path('broad', freeze), args.overwrite)

```

This has been running for over 17 hours, and it hasn’t downscaled (the cluster has autoscaling set). About how long does it take to downscale when using `densify`? Also, would it be better to write out the mt and then repartition?

---

<div class="post-metadata">

**Author:** ![ch-kr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ch-kr/32/317_2.png) [@ch-kr](https://discuss.hail.is/u/ch-kr)\
**Post date:** [November 19, 2019, 2:22pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/2 "2019-11-19T14:22:20Z")

</div>

[densify\_sparse\_mt.log](https://discuss.hail.is/uploads/short-url/xf3b6jDwMYEhl3VRW2fPpOC5ciU.log) (3.7 MB)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 21, 2019, 1:36pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/3 "2019-11-21T13:36:36Z")

</div>

Oops, thought I had responded!

I definitely wouldn’t repartition this using `mt.repartition`. It’s possible to repartition on `read` by reading with a set of intervals.

---

<div class="post-metadata">

**Author:** ![ch-kr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ch-kr/32/317_2.png) [@ch-kr](https://discuss.hail.is/u/ch-kr)\
**Post date:** [November 21, 2019, 1:41pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/4 "2019-11-21T13:41:06Z")

</div>

thanks! how do I repartition while reading? I’ve never used the `_intervals` parameter.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 21, 2019, 1:53pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/5 "2019-11-21T13:53:47Z")

</div>

It’s not super easy, involves some of the combiner functions. Give me until the end of today to try to clean it up? I also found a possible bug…

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 21, 2019, 1:57pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/6 "2019-11-21T13:57:40Z")

</div>

er nevermind was just confusing myself. This should work for now:

```python
def rep_on_read(path, n_desired):
     mt = hl.read_matrix_table(path)
     intervals = mt._calculate_new_partitions(n_desired)
     return hl.read_matrix_table(path, _intervals=intervals)

```

---

<div class="post-metadata">

**Author:** ![ch-kr](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ch-kr/32/317_2.png) [@ch-kr](https://discuss.hail.is/u/ch-kr)\
**Post date:** [November 21, 2019, 1:58pm UTC](https://discuss.hail.is/t/densify-sparse-mt/1177/7 "2019-11-21T13:58:09Z")

</div>

awesome, thank you!!