# Densify running out of memory

**URL:** <https://discuss.hail.is/t/densify-running-out-of-memory/1360>\
**Category:** Hail Query & hailctl\
**Created:** [April 2, 2020, 3:31pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360 "2020-04-02T15:31:34Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [June 12, 2020, 6:10pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/21 "2020-06-12T18:10:35Z")

</div>

We scraped everything using `gcloud dataproc clusters diagnose` and it seemed that results were being put in HDFS.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [June 12, 2020, 6:17pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/22 "2020-06-12T18:17:04Z")

</div>

oh, totally, it’s the aggs per partition: [https://github.com/hail-is/hail/blob/10e525a56787dd00b6aaed62d856c1eabd3c0f3a/hail/src/main/scala/is/hail/expr/ir/TableIR.scala#L1666-L1674](https://github.com/hail-is/hail/blob/10e525a56787dd00b6aaed62d856c1eabd3c0f3a/hail/src/main/scala/is/hail/expr/ir/TableIR.scala#L1666-L1674)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [June 12, 2020, 6:17pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/23 "2020-06-12T18:17:24Z")

</div>

What was the cluster config? non\_preempts, preempts?

---

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [June 12, 2020, 6:19pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/24 "2020-06-12T18:19:45Z")

</div>

It’s 300 workers all with lots of disk, we’re not running out. I suggested lower workers with more cores and that failed too.

---

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [June 26, 2020, 3:26pm UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/25 "2020-06-26T15:26:25Z")

</div>

Update here to not forget. The resource we are running out of in the latest bug is definitely memory 😬. I’ve tracked the change that introduced the regression to [https://github.com/hail-is/hail/pull/8794](https://github.com/hail-is/hail/pull/8794) and we are continuing to investigate.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [June 30, 2020, 1:01am UTC](https://discuss.hail.is/t/densify-running-out-of-memory/1360/26 "2020-06-30T01:01:40Z")

</div>

This PR fixes the problem:

> <https://github.com/hail-is/hail/pull/9028>

[Previous page](https://discuss.hail.is/t/densify-running-out-of-memory/1360.md?page=1)