# Export\_vcf OutOfMemoryError: Java heap space despite --driver-memory 8g

**URL:** <https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679>\
**Category:** Hail Query & hailctl\
**Created:** [October 5, 2018, 1:47pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679 "2018-10-05T13:47:43Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 9, 2019, 1:39pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/21 "2019-01-09T13:39:10Z")

</div>

Table.select (And MT.select\_rows / select\_cols) keep key fields implicitly - so `rows = ds.rows().select('rsid')` preserves the locus/alleles key.

What I want to do here is do the expensive and temperamental shuffle-join once with a small dataset, save it, and use that with the bgen.

---

<div class="post-metadata">

**Author:** ![hhx037](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/hhx037/32/160_2.png) [@hhx037](https://discuss.hail.is/u/hhx037)\
**Post date:** [January 9, 2019, 2:15pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/22 "2019-01-09T14:15:57Z")

</div>

OK, I see, but how do I then use it with the bgen?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 9, 2019, 3:08pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/23 "2019-01-09T15:08:49Z")

</div>

then you can read it in and join on key:

```auto
mt = hl.import_bgen(..._
inclusion_ht = hl.read_table('inclusion.ht')
mt = mt.filter_rows(~inclusion_ht[mt.row_key].inclusion)

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 9, 2019, 3:08pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/24 "2019-01-09T15:08:55Z")

</div>

then you can read it in and join on key:

```auto
mt = hl.import_bgen(..._
inclusion_ht = hl.read_table('inclusion.ht')
mt = mt.filter_rows(~inclusion_ht[mt.row_key].inclusion)

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [January 9, 2019, 3:09pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/25 "2019-01-09T15:09:25Z")

</div>

then you can read it in and join on key:

```auto
mt = hl.import_bgen(..._
inclusion_ht = hl.read_table('inclusion.ht')
mt = mt.filter_rows(~inclusion_ht[mt.row_key].inclusion)

```

---

<div class="post-metadata">

**Author:** ![hhx037](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/hhx037/32/160_2.png) [@hhx037](https://discuss.hail.is/u/hhx037)\
**Post date:** [January 9, 2019, 3:12pm UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/26 "2019-01-09T15:12:21Z")

</div>

OK… I wasn’t sure if using filter\_rows was still the way to go. Thank you, I’ll try that and let you know how it goes.

---

<div class="post-metadata">

**Author:** ![hhx037](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/hhx037/32/160_2.png) [@hhx037](https://discuss.hail.is/u/hhx037)\
**Post date:** [January 11, 2019, 9:48am UTC](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679/27 "2019-01-11T09:48:03Z")

</div>

Unfortunately, it seems the shuffling issue persists even on the smaller matrix, it crashes when it’s executed (i.e. when saving the matrix):

> variant\_exclusion\_table = hl.import\_table(“/mnt/output/HRC.vcfs/HRC\_variants.tsv.bgz”, no\_header=True, key=‘f0’)  
> variant\_exclusion\_table = variant\_exclusion\_table.key\_by(‘f0’)  
> rows = ds.rows().select(‘rsid’)  
> rows = rows.annotate(included = ~hl.is\_defined(variant\_exclusion\_table[rows.rsid]))  
> rows.write(“mnt/output/HRC.vcfs/UK10K\_inclusion\_status.ht”)

```
[Stage 6:> (0 + 46) / 2901]Traceback (most recent call last):
  File "/mnt/output/regression1/UK10K.py", line 17, in <module>
    rows.write("/mnt/output/HRC.vcfs/UK10K_inclusion_status.ht")
  File "<decorator-gen-694>", line 2, in write
  File "/usr/local/hail/build/distributions/hail-python.zip/hail/typecheck/check.py", line 560, in wrapper
  File "/usr/local/hail/build/distributions/hail-python.zip/hail/table.py", line 1163, in write
  File "/usr/local/hail/build/distributions/hail-python.zip/hail/backend/backend.py", line 25, in interpret
  File "/usr/local/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/usr/local/hail/build/distributions/hail-python.zip/hail/utils/java.py", line 210, in deco
hail.utils.java.FatalError: OutOfMemoryError: Java heap space

```

[Previous page](https://discuss.hail.is/t/export-vcf-outofmemoryerror-java-heap-space-despite-driver-memory-8g/679.md?page=1)
