# Running hail locally - number of cores

**URL:** <https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273>\
**Category:** Hail Query & hailctl\
**Created:** [March 27, 2023, 4:23pm UTC](https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273 "2023-03-27T16:23:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ag14774](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ag14774/32/933_2.png) [@ag14774](https://discuss.hail.is/u/ag14774)\
**Post date:** [March 27, 2023, 4:23pm UTC](https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273/1 "2023-03-27T16:23:53Z")

</div>

It is unclear from the docs currently how to correctly initialize hail to maximize performance. In my case I am running Hail locally on a large machine with 1TB of memory and 96 cores. 1) Does hail automatically use all the available memory? 2) Does hail automatically use all available cores? What do I need to adjust to make hail use all resources? Any particular configurations to pass to hail.init()?

Thank you

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [March 27, 2023, 4:27pm UTC](https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273/2 "2023-03-27T16:27:35Z")

</div>

Hail running on Apache Spark in local mode will use all available cores by default, but you can load the Spark WebUI (the URL is printed in initialization) and look at the Executors pane to ensure everything is getting used. Using `hl.init(master='local[96]')` can explicitly request 96 local cores.

You should also make sure that you set memory with PYSPARK\_SUBMIT\_ARGS:

> [@How do I increase the memory or RAM available to the JVM when I start Hail through Python?](https://discuss.hail.is/t/how-do-i-increase-the-memory-or-ram-available-to-the-jvm-when-i-start-hail-through-python/133/2):
>
> You can set the memory using an environment variable: PYSPARK\_SUBMIT\_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK\_SUBMIT\_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"

---

<div class="post-metadata">

**Author:** ![ag14774](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ag14774/32/933_2.png) [@ag14774](https://discuss.hail.is/u/ag14774)\
**Post date:** [March 27, 2023, 4:49pm UTC](https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273/3 "2023-03-27T16:49:08Z")

</div>

Thanks for the reply!

In the “Environment” tab it also says this under “Resource Profiles”:

```python
Executor Reqs:
	cores: [amount: 1]
Task Reqs:
	cpus: [amount: 1.0]

```

In the Executors pane though it shows all cores. Does that mean it is all ok and it is using all cores?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [March 28, 2023, 10:14am UTC](https://discuss.hail.is/t/running-hail-locally-number-of-cores/3273/4 "2023-03-28T10:14:31Z")

</div>

Should be, yes!

Spark sometimes has issues with resource contention when running with such a high degree of multithreading in a single java process, so post here if anything else looks funky as you start running computations.