How to overcome memory issue?

[annalisasnow]

Well, as far as I can see there’re several questions regarding that issue. However, I guess mine is gonna be a little bit different. So, the main point is, while importing files to matrix or matrices there are memory issues. Sometimes those could be fixed with:

hl.init(min_block_size=128, master=local[16])

Unfortunately that still isn’t enough:

Hail version: 0.2.74-0c3a74d12093
Error summary: OutOfMemoryError: GC overhead limit exceeded

Hail version: 0.2.74-0c3a74d12093
Error summary: OutOfMemoryError: Java heap space

What else can I do, given I am running it on a computer via Snakemake? Is it possible to send some Java stuff, like -Xms5g -Xmx10g or somewhat else to prevent the error while running it on local machine.

[danking]

You’re right that the issue is Java / Spark. You’ll need to set PYSPARK_SUBMIT_ARGS as described here.

How do I increase the memory or RAM available to the JVM when I start Hail through Python?

You can set the memory using an environment variable: PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"

[annalisasnow]

what if I don’t use neither iPython nor notebook?

[danking]

PYSPARK_SUBMIT_ARGS works for any Python program that uses Spark and/or Hail.

[annalisasnow]

Thx for your remark, made it!