# Heap out of memory

**URL:** https://discuss.hail.is/t/heap-out-of-memory/1536
**Category:** Hail Query & hailctl
**Created:** [July 16, 2020, 8:09pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536 "2020-07-16T20:09:51Z")
**Posts on this page:** 15
**Page:** 1

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 16, 2020, 8:09pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/1 "2020-07-16T20:09:51Z")

</div>

Hi there,

I’m using the hl.agg.linreg() option to run a weighted GWAS. Everything went well when I only included 10 covariates, but when I raised the number of covariates to 20, the program crashed and returned an OutOfMemoryError. Here’s the error log:

> Java stack trace:  
> java.lang.OutOfMemoryError: Java heap space  
> at java.util.Arrays.copyOf(Arrays.java:3236)  
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)  
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)  
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)  
> at is.hail.io.StreamBlockOutputBuffer.writeBlock(OutputBuffers.scala:293)  
> at is.hail.io.LZ4OutputBlockBuffer.writeBlock(OutputBuffers.scala:313)  
> at is.hail.io.BlockingOutputBuffer.writeBlock(OutputBuffers.scala:190)  
> at is.hail.io.BlockingOutputBuffer.writeDouble(OutputBuffers.scala:234)  
> at is.hail.io.LEB128OutputBuffer.writeDouble(OutputBuffers.scala:170)  
> at \_\_C402etypeEncode.\_\_m408ENCODE\_r\_float64\_TO\_r\_float64(Unknown Source)  
> at \_\_C402etypeEncode.\_\_m407write\_fields\_group\_0(Unknown Source)  
> at \_\_C402etypeEncode.\_\_m406ENCODE\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDEND\_TO\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDEND(Unknown Source)  
> at \_\_C402etypeEncode.\_\_m405ENCODE\_r\_array\_of\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDEND\_TO\_r\_array\_of\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDEND(Unknown Source)  
> at \_\_C402etypeEncode.\_\_m404write\_fields\_group\_0(Unknown Source)  
> at \_\_C402etypeEncode.\_\_m403ENCODE\_r\_tuple\_of\_r\_array\_of\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDENDEND\_TO\_r\_struct\_of\_r\_array\_of\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDENDEND(Unknown Source)  
> at \_\_C402etypeEncode.apply(Unknown Source)  
> at is.hail.io.CompiledEncoder.writeRegionValue(Encoder.scala:32)  
> at is.hail.annotations.BroadcastRegionValue$class.broadcast(BroadcastValue.scala:55)  
> at is.hail.annotations.BroadcastRow.broadcast$lzycompute(BroadcastValue.scala:75)  
> at is.hail.annotations.BroadcastRow.broadcast(BroadcastValue.scala:75)  
> at is.hail.expr.ir.TableMapRows.execute(TableIR.scala:1450)  
> at is.hail.expr.ir.TableMapGlobals.execute(TableIR.scala:1737)  
> at is.hail.expr.ir.Interpret$.run(Interpret.scala:811)  
> at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)  
> at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:56)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)  
> Hail version: 0.2.49-11ae8408bad0  
> Error summary: OutOfMemoryError: Java heap space

May I ask if there’s anything I could do in this case? Thank you very much!

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 16, 2020, 8:47pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/2 "2020-07-16T20:47:34Z")

</div>

what’s the pipeline and what’s the full stack trace?

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 16, 2020, 10:10pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/3 "2020-07-16T22:10:10Z")

</div>

Pipeline:

> #!/usr/bin/python  
> import hail as hl  
> hl.init()  
> mt = hl.import\_plink(bed=‘chr22\_EA\_40\_40.bed’,bim=‘chr22\_EA\_40\_40.bim’,fam=‘chr22\_EA\_40\_40.fam’,quant\_pheno=True)  
> covar = (hl.import\_table(‘EA\_covar.txt’,types={‘IID’:hl.tstr},impute=True).key\_by(‘IID’))  
> mt = mt.annotate\_cols(covar=covar[mt.s])  
> weight = (hl.import\_table(‘EA\_weight.txt’,types={‘IID’:hl.tstr},impute=True).key\_by(‘IID’))  
> mt = mt.annotate\_cols(weight=weight[mt.s])  
> mt = mt.annotate\_rows(gwas=hl.agg.linreg(mt.quant\_pheno,  
> [1,mt.covar.isMale,mt.covar.YEAR,mt.covar.isAxiom,  
> mt.covar.PC1,mt.covar.PC2,mt.covar.PC3,mt.covar.PC4,  
> mt.covar.PC5,mt.covar.PC6,mt.covar.PC7,mt.covar.PC8,  
> mt.covar.PC9,mt.covar.PC10,mt.covar.PC11,mt.covar.PC12,  
> mt.covar.PC13,mt.covar.PC14,mt.covar.PC15,mt.covar.PC16,  
> mt.covar.PC17,mt.covar.PC18,mt.covar.PC19,mt.covar.PC20,  
> mt.GT.n\_alt\_alleles()],weight=mt.weight.weight))  
> mt.gwas.export(‘chr22\_EA\_40\_40\_hail.txt’)

Full error log (including stack trace):

> 2020-07-16 15:45:59 Hail: INFO: Found 372999 samples in fam file.  
> 2020-07-16 15:45:59 Hail: INFO: Found 2290 variants in bim file.  
> 2020-07-16 15:45:59 Hail: INFO: Reading table to impute column types  
> [Stage 0:\> (0 + 1) / 1]2020-07-16 15:46:05 Hail: INFO: Finished type imputation  
> Loading column ‘IID’ as type ‘str’ (user-specified)  
> Loading column ‘isMale’ as type ‘bool’ (imputed)  
> Loading column ‘YEAR’ as type ‘int32’ (imputed)  
> Loading column ‘isAxiom’ as type ‘bool’ (imputed)  
> Loading column ‘PC1’ as type ‘float64’ (imputed)  
> Loading column ‘PC2’ as type ‘float64’ (imputed)  
> Loading column ‘PC3’ as type ‘float64’ (imputed)  
> Loading column ‘PC4’ as type ‘float64’ (imputed)  
> Loading column ‘PC5’ as type ‘float64’ (imputed)  
> Loading column ‘PC6’ as type ‘float64’ (imputed)  
> Loading column ‘PC7’ as type ‘float64’ (imputed)  
> Loading column ‘PC8’ as type ‘float64’ (imputed)  
> Loading column ‘PC9’ as type ‘float64’ (imputed)  
> Loading column ‘PC10’ as type ‘float64’ (imputed)  
> Loading column ‘PC11’ as type ‘float64’ (imputed)  
> Loading column ‘PC12’ as type ‘float64’ (imputed)  
> Loading column ‘PC13’ as type ‘float64’ (imputed)  
> Loading column ‘PC14’ as type ‘float64’ (imputed)  
> Loading column ‘PC15’ as type ‘float64’ (imputed)  
> Loading column ‘PC16’ as type ‘float64’ (imputed)  
> Loading column ‘PC17’ as type ‘float64’ (imputed)  
> Loading column ‘PC18’ as type ‘float64’ (imputed)  
> Loading column ‘PC19’ as type ‘float64’ (imputed)  
> Loading column ‘PC20’ as type ‘float64’ (imputed)  
> 2020-07-16 15:46:05 Hail: INFO: Reading table to impute column types  
> 2020-07-16 15:46:05 Hail: INFO: Finished type imputation  
> Loading column ‘IID’ as type ‘str’ (user-specified)  
> Loading column ‘weight’ as type ‘float64’ (imputed)  
> [Stage 3:\> (0 + 1) / 1]Traceback (most recent call last):  
> File “test.py”, line 17, in   
> mt.gwas.export(‘chr22\_EA\_40\_40\_hail.txt’)  
> File “”, line 2, in export  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 614, in wrapper  
> return _original\_func(\*args, \*\*kwargs_)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/expr/expressions/base\_expression.py”, line 944, in export  
> ds.export(output=path, delimiter=delimiter, header=header)  
> File “”, line 2, in export  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 614, in wrapper  
> return _original\_func(\*args, \*\*kwargs_)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/table.py”, line 1038, in export  
> ir.TableWrite(self.\_tir, ir.TableTextWriter(output, types\_file, header, parallel, delimiter)))  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark\_backend.py”, line 296, in execute  
> result = json.loads(self.\_jhc.backend().executeJSON(jir))  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java\_gateway.py”, line 1257, in **call**  
> answer, self.gateway\_client, self.target\_id, self.name)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark\_backend.py”, line 41, in deco  
> ‘Error summary: %s’ % (deepest, full, hail. **version** , deepest)) from None  
> hail.utils.java.FatalError: OutOfMemoryError: Java heap space
> 
> > Java stack trace:
> > 
> > java.lang.OutOfMemoryError: Java heap space
> > 
> > at java.util.Arrays.copyOf(Arrays.java:3236)
> > 
> > at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> > 
> > at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> > 
> > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> > 
> > at is.hail.io.StreamBlockOutputBuffer.writeBlock(OutputBuffers.scala:293)
> > 
> > at is.hail.io.LZ4OutputBlockBuffer.writeBlock(OutputBuffers.scala:313)
> > 
> > at is.hail.io.BlockingOutputBuffer.writeBlock(OutputBuffers.scala:190)
> > 
> > at is.hail.io.BlockingOutputBuffer.writeDouble(OutputBuffers.scala:234)
> > 
> > at is.hail.io.LEB128OutputBuffer.writeDouble(OutputBuffers.scala:170)
> > 
> > at \_\_C402etypeEncode.\_\_m408ENCODE\_r\_float64\_TO\_r\_float64(Unknown Source)
> > 
> > at \_\_C402etypeEncode.\_\_m407write\_fields\_group\_0(Unknown Source)
> > 
> > at \_\_C402etypeEncode.\_\_m406ENCODE\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDEND\_TO\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDEND(Unknown Source)
> > 
> > at \_\_C402etypeEncode.\_\_m405ENCODE\_r\_array\_of\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDEND\_TO\_r\_array\_of\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDEND(Unknown Source)
> > 
> > at \_\_C402etypeEncode.\_\_m404write\_fields\_group\_0(Unknown Source)
> > 
> > at \_\_C402etypeEncode.\_\_m403ENCODE\_r\_tuple\_of\_r\_array\_of\_r\_tuple\_of\_r\_float64ANDo\_tuple\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_tuple\_of\_r\_float64ENDENDEND\_TO\_r\_struct\_of\_r\_array\_of\_r\_struct\_of\_r\_float64ANDo\_struct\_of\_r\_boolANDr\_int32ANDr\_boolANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ANDr\_float64ENDANDo\_struct\_of\_r\_float64ENDENDEND(Unknown Source)
> > 
> > at \_\_C402etypeEncode.apply(Unknown Source)
> > 
> > at is.hail.io.CompiledEncoder.writeRegionValue(Encoder.scala:32)
> > 
> > at is.hail.annotations.BroadcastRegionValue$class.broadcast(BroadcastValue.scala:55)
> > 
> > at is.hail.annotations.BroadcastRow.broadcast$lzycompute(BroadcastValue.scala:75)
> > 
> > at is.hail.annotations.BroadcastRow.broadcast(BroadcastValue.scala:75)
> > 
> > at is.hail.expr.ir.TableMapRows.execute(TableIR.scala:1450)
> > 
> > at is.hail.expr.ir.TableMapGlobals.execute(TableIR.scala:1737)
> > 
> > at is.hail.expr.ir.Interpret$.run(Interpret.scala:811)
> > 
> > at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53)
> > 
> > at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)
> > 
> > at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)
> > 
> > at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)
> > 
> > at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:56)
> > 
> > at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)
> > 
> > at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)
> > 
> > at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
> > 
> > at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)
> 
> Hail version: 0.2.49-11ae8408bad0
> 
> Error summary: OutOfMemoryError: Java heap space

Please let me know if you need any further information, thanks!

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 17, 2020, 10:06am UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/4 "2020-07-17T10:06:23Z")

</div>

A few questions -

1. what runtime are you using? A Spark cluster, or running locally? Spark allocates a very small amount of memory by default in local mode.

2. What are the dimensions of the PLINK file you’re importing? Number of variants, number of samples?

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 17, 2020, 4:47pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/5 "2020-07-17T16:47:26Z")

</div>

1. Yes I’m running on a Spark cluster.

2. There are 372999 samples, and the total number of variants is 7000000. However, I’ve cutted the chromosomes into pieces before running GWAS, and the largest piece contains ~15000 variants.

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 17, 2020, 4:50pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/6 "2020-07-17T16:50:36Z")

</div>

Can we see the full pipeline? May need to look at the hail log file too if that doesn’t illuminate things.

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 17, 2020, 4:55pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/7 "2020-07-17T16:55:58Z")

</div>

Do you mean the pipeline I used to cut chromosomes into pieces? I did that with R and PLINK, not hail.

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 17, 2020, 4:58pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/8 "2020-07-17T16:58:01Z")

</div>

oops, sorry, you posted it above.

I think the problem may be related to the scalability of Hail’s import\_plink function. Could you try running PLINK to create a VCF from those input files, and importing that?

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 17, 2020, 8:29pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/9 "2020-07-17T20:29:43Z")

</div>

I tried to import VCF files instead and run GWAS. However, similar error occured:

> 2020-07-17 15:21:17 Hail: INFO: Reading table to impute column types  
> [Stage 0:\> (0 + 1) / 1]2020-07-17 15:21:26 Hail: INFO: Finished type imputation  
> Loading column ‘IID’ as type ‘str’ (user-specified)  
> Loading column ‘isMale’ as type ‘bool’ (imputed)  
> Loading column ‘YEAR’ as type ‘int32’ (imputed)  
> Loading column ‘isAxiom’ as type ‘bool’ (imputed)  
> Loading column ‘PC1’ as type ‘float64’ (imputed)  
> Loading column ‘PC2’ as type ‘float64’ (imputed)  
> Loading column ‘PC3’ as type ‘float64’ (imputed)  
> Loading column ‘PC4’ as type ‘float64’ (imputed)  
> Loading column ‘PC5’ as type ‘float64’ (imputed)  
> Loading column ‘PC6’ as type ‘float64’ (imputed)  
> Loading column ‘PC7’ as type ‘float64’ (imputed)  
> Loading column ‘PC8’ as type ‘float64’ (imputed)  
> Loading column ‘PC9’ as type ‘float64’ (imputed)  
> Loading column ‘PC10’ as type ‘float64’ (imputed)  
> Loading column ‘PC11’ as type ‘float64’ (imputed)  
> Loading column ‘PC12’ as type ‘float64’ (imputed)  
> Loading column ‘PC13’ as type ‘float64’ (imputed)  
> Loading column ‘PC14’ as type ‘float64’ (imputed)  
> Loading column ‘PC15’ as type ‘float64’ (imputed)  
> Loading column ‘PC16’ as type ‘float64’ (imputed)  
> Loading column ‘PC17’ as type ‘float64’ (imputed)  
> Loading column ‘PC18’ as type ‘float64’ (imputed)  
> Loading column ‘PC19’ as type ‘float64’ (imputed)  
> Loading column ‘PC20’ as type ‘float64’ (imputed)  
> 2020-07-17 15:21:26 Hail: INFO: Reading table to impute column types  
> [Stage 1:\> (0 + 1) / 1]2020-07-17 15:21:27 Hail: INFO: Finished type imputation  
> Loading column ‘IID’ as type ‘str’ (user-specified)  
> Loading column ‘weight’ as type ‘float64’ (imputed)  
> 2020-07-17 15:21:27 Hail: INFO: Reading table to impute column types  
> 2020-07-17 15:21:27 Hail: INFO: Finished type imputation  
> Loading column ‘IID’ as type ‘str’ (user-specified)  
> Loading column ‘EA’ as type ‘int32’ (imputed)  
> [Stage 6:========================\> (11 + 14) / 25]2020-07-17 15:23:27 Hail: INFO: Coerced sorted dataset  
> [Stage 7:\> (0 + 24) / 25]Traceback (most recent call last):  
> File “test2.py”, line 19, in   
> mt.gwas.export(‘chr22\_1\_40\_hail.txt’)  
> File “\<decorator-gen-577\>”, line 2, in export  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 614, in wrapper  
> return _original\_func(\*args, \*\*kwargs_)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/expr/expressions/base\_expression.py”, line 944, in export  
> ds.export(output=path, delimiter=delimiter, header=header)  
> File “”, line 2, in export  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 614, in wrapper  
> return _original\_func(\*args, \*\*kwargs_)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/table.py”, line 1038, in export  
> ir.TableWrite(self.\_tir, ir.TableTextWriter(output, types\_file, header, parallel, delimiter)))  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark\_backend.py”, line 296, in execute  
> result = json.loads(self.\_jhc.backend().executeJSON(jir))  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/py4j/java\_gateway.py”, line 1257, in **call**  
> answer, self.gateway\_client, self.target\_id, self.name)  
> File “/ua/xzhong35/.local/lib/python3.6/site-packages/hail/backend/spark\_backend.py”, line 41, in deco  
> ‘Error summary: %s’ % (deepest, full, hail. **version** , deepest)) from None  
> hail.utils.java.FatalError: OutOfMemoryError: Java heap space
> 
> Java stack trace:  
> org.apache.spark.SparkException: Job aborted.  
> at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:100)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1096)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1067)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:957)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1499)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1478)  
> at is.hail.utils.richUtils.RichRDD$.writeTable$extension(RichRDD.scala:78)  
> at is.hail.expr.ir.TableValue.export(TableValue.scala:98)  
> at is.hail.expr.ir.TableTextWriter.apply(TableWriter.scala:337)  
> at is.hail.expr.ir.Interpret$.run(Interpret.scala:811)  
> at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)  
> at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:56)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)  
> at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:51)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12)  
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)  
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)  
> at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12)  
> at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:28)  
> at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$\_execute(SparkBackend.scala:318)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:305)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:304)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:20)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:18)  
> at is.hail.utils.package$.using(package.scala:602)  
> at is.hail.annotations.Region$.scoped(Region.scala:18)  
> at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:18)  
> at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:230)  
> at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:304)  
> at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:324)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
> at py4j.Gateway.invoke(Gateway.java:282)  
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
> at py4j.commands.CallCommand.execute(CallCommand.java:79)  
> at py4j.GatewayConnection.run(GatewayConnection.java:238)  
> at java.lang.Thread.run(Thread.java:748)
> 
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 12 in stage 7.0 failed 1 times, most recent failure: Lost task 12.0 in stage 7.0 (TID 43, localhost, executor driver): org.apache.spark.SparkException: Task failed while writing rows  
> at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)  
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)  
> at org.apache.spark.scheduler.Task.run(Task.scala:121)  
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)  
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)  
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at java.lang.Thread.run(Thread.java:748)  
> Caused by: java.lang.OutOfMemoryError: Java heap space  
> at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)  
> at java.lang.StringCoding.decode(StringCoding.java:193)  
> at java.lang.StringCoding.decode(StringCoding.java:254)  
> at java.lang.String.(String.java:546)  
> at is.hail.expr.ir.GenericLine.toString(GenericLines.scala:322)  
> at is.hail.io.vcf.MatrixVCFReader$$anonfun$21$$anonfun$apply$10$$anonfun$apply$11.apply(LoadVCF.scala:1732)  
> at is.hail.io.vcf.MatrixVCFReader$$anonfun$21$$anonfun$apply$10$$anonfun$apply$11.apply(LoadVCF.scala:1731)  
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:464)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at is.hail.utils.richUtils.RichContextRDD$$anonfun$cleanupRegions$1$$anon$1.hasNext(RichContextRDD.scala:31)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:128)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)  
> at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)  
> at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)  
> … 10 more
> 
> Driver stacktrace:  
> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)  
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)  
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)  
> at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)  
> at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)  
> at scala.Option.foreach(Option.scala:257)  
> at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)  
> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)  
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)  
> at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)  
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)  
> at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1096)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1094)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1067)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1032)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:957)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1499)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)  
> at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
> at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)  
> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1478)  
> at is.hail.utils.richUtils.RichRDD$.writeTable$extension(RichRDD.scala:78)  
> at is.hail.expr.ir.TableValue.export(TableValue.scala:98)  
> at is.hail.expr.ir.TableTextWriter.apply(TableWriter.scala:337)  
> at is.hail.expr.ir.Interpret$.run(Interpret.scala:811)  
> at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)  
> at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)  
> at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:56)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)  
> at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13)  
> at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)  
> at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13)  
> at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:51)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14)  
> at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12)  
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)  
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)  
> at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12)  
> at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:28)  
> at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$\_execute(SparkBackend.scala:318)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:305)  
> at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:304)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:20)  
> at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:18)  
> at is.hail.utils.package$.using(package.scala:602)  
> at is.hail.annotations.Region$.scoped(Region.scala:18)  
> at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:18)  
> at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:230)  
> at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:304)  
> at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:324)  
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> at java.lang.reflect.Method.invoke(Method.java:498)  
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
> at py4j.Gateway.invoke(Gateway.java:282)  
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
> at py4j.commands.CallCommand.execute(CallCommand.java:79)  
> at py4j.GatewayConnection.run(GatewayConnection.java:238)  
> at java.lang.Thread.run(Thread.java:748)
> 
> org.apache.spark.SparkException: Task failed while writing rows  
> at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)  
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)  
> at org.apache.spark.scheduler.Task.run(Task.scala:121)  
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)  
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)  
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at java.lang.Thread.run(Thread.java:748)
> 
> java.lang.OutOfMemoryError: Java heap space  
> at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149)  
> at java.lang.StringCoding.decode(StringCoding.java:193)  
> at java.lang.StringCoding.decode(StringCoding.java:254)  
> at java.lang.String.(String.java:546)  
> at is.hail.expr.ir.GenericLine.toString(GenericLines.scala:322)  
> at is.hail.io.vcf.MatrixVCFReader$$anonfun$21$$anonfun$apply$10$$anonfun$apply$11.apply(LoadVCF.scala:1732)  
> at is.hail.io.vcf.MatrixVCFReader$$anonfun$21$$anonfun$apply$10$$anonfun$apply$11.apply(LoadVCF.scala:1731)  
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:464)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at is.hail.utils.richUtils.RichContextRDD$$anonfun$cleanupRegions$1$$anon$1.hasNext(RichContextRDD.scala:31)  
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)  
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:128)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)  
> at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)  
> at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)  
> at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)  
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)  
> at org.apache.spark.scheduler.Task.run(Task.scala:121)  
> at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)  
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)  
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
> at java.lang.Thread.run(Thread.java:748)
> 
> Hail version: 0.2.49-11ae8408bad0  
> Error summary: OutOfMemoryError: Java heap space

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 17, 2020, 8:43pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/10 "2020-07-17T20:43:50Z")

</div>

OK, thanks. This is super confusing.

Can you access the Spark web UI to look at memory settings? I’ve never seen an OOM from a VCF like this.

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 21, 2020, 2:31pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/11 "2020-07-21T14:31:09Z")

</div>

Sorry for the delayed response, it took me a while to figure out how to access the Spark web UI. Is this what you’re looking for? Otherwise, may I ask where can I find the memory settings? Thanks!

 ![Screen Shot 2020-07-21 at 9.27.52 AM](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/712a6f4964113d92ceb6cc9916cfbd249d4c19ea.png)

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 21, 2020, 2:33pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/12 "2020-07-21T14:33:58Z")

</div>

This indicates that you’re running on one machine with 32 cores, and have less than 400 MB (!) allocated for the entire process. This should solve things:

> [@How do I increase the memory or RAM available to the JVM when I start Hail through Python?](https://discuss.hail.is/t/how-do-i-increase-the-memory-or-ram-available-to-the-jvm-when-i-start-hail-through-python/133/2):
>
> You can set the memory using an environment variable: PYSPARK\_SUBMIT\_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython This will start an ipython notebook with 8 GB of memory. If you want ipython to always start with 8 GB of memory, you can add this to your .bashrc (or the equivalent file for your shell): export PYSPARK\_SUBMIT\_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 21, 2020, 2:46pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/13 "2020-07-21T14:46:28Z")

</div>

Thanks. I often put the code in a Python script and executed it with the “python3” command line argument. In this case, am I supposed to put

> export PYSPARK\_SUBMIT\_ARGS=“–driver-memory 8g pyspark-shell”

in all my Python scripts? My apology if the question sounds stupid, but I’m not very familiar with Python…

---

<div class="post-metadata">

### Author: ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [July 21, 2020, 2:49pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/14 "2020-07-21T14:49:30Z")

</div>

no, this is a shell command. You can also do:

```
PYSPARK_SUBMIT_ARGS="–driver-memory 8g pyspark-shell" python3

```

Also, you’ll probably want more than 8G for a machine with 32 cores. I’m not sure how much memory your machine has, though.

---

<div class="post-metadata">

### Author: ![EZXYZ](https://avatars.discourse-cdn.com/v4/letter/e/13edae/32.png) [@EZXYZ](https://discuss.hail.is/u/EZXYZ)
#### Post date: [July 21, 2020, 3:07pm UTC](https://discuss.hail.is/t/heap-out-of-memory/1536/15 "2020-07-21T15:07:19Z")

</div>

Sounds good, this is really helpful. Thanks a lot!