# Aggregate\_intermediates in tmp\_dir filling up

**URL:** <https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768>\
**Category:** Hail Query & hailctl\
**Created:** [July 27, 2022, 7:35am UTC](https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768 "2022-07-27T07:35:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ruth\_Eberhardt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ruth_eberhardt/32/742_2.png) [@Ruth\_Eberhardt](https://discuss.hail.is/u/Ruth_Eberhardt)\
**Post date:** [July 27, 2022, 7:35am UTC](https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768/1 "2022-07-27T07:35:44Z")

</div>

Hi  
I’ve had a couple of issues when I’ve been performing a lot of aggregations, the aggregate\_intermediates directory in the location specified by tmp\_dir when Hail is initialised fills up and I get the following error:

`2022-07-26 17:28:14 TaskSetManager: WARN: Lost task 874.0 in stage 336.0 (TID 529110) (192.168.252.56 executor 40): org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /aggregate_intermediates is exceeded: limit=1048576 items=1048576`

If I go into my file system and manually clear this then I can run aggregations again.  
Is there a way to ensure that this subdirectory if tmp\_dir is cleared at intervals?  
Thank you!

---

<div class="post-metadata">

**Author:** ![AB.Hail](https://avatars.discourse-cdn.com/v4/letter/a/d2c977/32.png) [@AB.Hail](https://discuss.hail.is/u/AB.Hail)\
**Post date:** [August 8, 2022, 1:26am UTC](https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768/2 "2022-08-08T01:26:58Z")

</div>

I have experienced the same issue from time to time and I manually delete the directory to fix the issue. Is there a spark option to increase the size of this directory? Also is this possible to use a different path for each user so that deleting aggregate\_intermediates for one user does not affect other users’ jobs?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [August 8, 2022, 12:08pm UTC](https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768/3 "2022-08-08T12:08:28Z")

</div>

This is really a Hail issue – we need to be eagerly cleaning up files when they’re no longer necessary.

@AB.Hail - there’s no Spark option here, since it’s a Hail parameter. You can set the Hail temp dir on init with `hl.init(...other args..., tmp_dir='...')` You can set this to a blob store path (google/s3 bucket, etc) and that will work fine if you’re running on the cloud.

---

<div class="post-metadata">

**Author:** ![Ruth\_Eberhardt](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ruth_eberhardt/32/742_2.png) [@Ruth\_Eberhardt](https://discuss.hail.is/u/Ruth_Eberhardt)\
**Post date:** [August 10, 2022, 8:23am UTC](https://discuss.hail.is/t/aggregate-intermediates-in-tmp-dir-filling-up/2768/4 "2022-08-10T08:23:13Z")

</div>

Thank you @AB.Hail and @tpoterba  
I resorted to manually removing the contents of aggregate\_intermediates as I was running sample\_qc so many times in one script that I was running out of space in my tmp\_dir.