# VCF matrix table memory error?

**URL:** <https://discuss.hail.is/t/vcf-matrix-table-memory-error/2243>\
**Category:** Hail Query & hailctl\
**Created:** [September 15, 2021, 9:32pm UTC](https://discuss.hail.is/t/vcf-matrix-table-memory-error/2243 "2021-09-15T21:32:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\
**Post date:** [September 15, 2021, 9:32pm UTC](https://discuss.hail.is/t/vcf-matrix-table-memory-error/2243/1 "2021-09-15T21:32:15Z")

</div>

I’ve imported a large vcf into a matrix table with import\_vcf(). When I try to run mt.show(5), I get this very long java stack trace, and at the bottom it says

> Error summary: IOException: No space left on device

I’ve tried changing the n\_paritions in the import\_vcf() command to 50, 1000, 10000, always the same error. I’m not sure how to tackle this problem. I eventually want to annotate this vcf using gnomAD popmax data, but I can’t even seem to get it to display the first 5 rows of the matrix table without overloading the memory. (I’m using a remote linux server with plenty of “horsepower”)

What are some strategies for handling large tables such as this? Would writing the table to a “directory.mt” make any difference?

Full Stack trace:

mt=hl.import\_vcf(vcf, force\_bgz=True, reference\_genome=rg, n\_partitions=10000)  
mt.show(5)  
2021-09-15 13:43:54 Hail: INFO: Ordering unsorted dataset with shuffle) / 10000]  
2021-09-15 13:44:01 Hail: INFO: Ordering unsorted dataset with network shuffle0]  
2021-09-15 13:44:08 Hail: INFO: Ordering unsorted dataset with shuffle) / 10000]  
2021-09-15 13:44:15 Hail: INFO: Ordering unsorted dataset with network shuffle0]  
Traceback (most recent call last): (291 + 48) / 10000]  
File “”, line 1, in  
File “”, line 2, in show  
File “/usr/local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 577, in wrapper  
return \*\*original\_func(_args_ , \*_kwargs_ )  
File “/usr/local/lib/python3.6/site-packages/hail/matrixtable.py”, line 2629, in show  
return handler(MatrixTable.\_Show(t, n\_rows, actual\_n\_cols, displayed\_n\_cols, width, truncate, types))  
File “/usr/local/lib/python3.6/site-packages/hail/matrixtable.py”, line 2538, in **str**  
s = self.table\_show. **str** ()  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 1294, in **str**  
return self.\_ascii\_str()  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 1320, in \_ascii\_str  
rows, has\_more, dtype = self.data()  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 1304, in data  
rows, has\_more = t.\_take\_n(self.n)  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 1451, in \_take\_n  
rows = self.take(n + 1)  
File “”, line 2, in take  
File “/usr/local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 577, in wrapper  
return \*\*original\_func(_args_ , \*_kwargs_ )  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 2121, in take  
return self.head(n).collect(\_localize)  
File “”, line 2, in collect  
File “/usr/local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 577, in wrapper  
return \*\*original\_func(_args_ , \*_kwargs_ )  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 2121, in take  
return self.head(n).collect(\_localize)  
File “”, line 2, in collect  
File “/usr/local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 577, in wrapper  
return \*\*original\_func(_args_ , \*_kwargs_ )  
File “/usr/local/lib/python3.6/site-packages/hail/table.py”, line 1920, in collect  
return Env.backend().execute(e.\_ir)  
File “/usr/local/lib/python3.6/site-packages/hail/backend/py4j\_backend.py”, line 98, in execute  
raise e  
File “/usr/local/lib/python3.6/site-packages/hail/backend/py4j\_backend.py”, line 74, in execute  
result = json.loads(self.\_jhc.backend().executeJSON(jir))  
File “/usr/local/lib/python3.6/site-packages/py4j/java\_gateway.py”, line 1305, in **call**  
answer, self.gateway\_client, self.target\_id, self.name)  
File “/usr/local/lib/python3.6/site-packages/hail/backend/py4j\_backend.py”, line 32, in deco  
‘Error summary: %s’ % (deepest, full, hail. **version** , deepest), error\_id) from None  
hail.utils.java.FatalError: IOException: No space left on device

Java stack trace:  
org.apache.spark.SparkException: Job aborted due to stage failure: Task 309 in stage 4.0 failed 1 times, most recent failure: Lost task 309.0 in stage 4.0 (TID 40309) (drgn-6 executor driver): java.io.IOException: No space left on device  
at java.io.FileOutputStream.writeBytes(Native Method)  
at java.io.FileOutputStream.write(FileOutputStream.java:326)  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)  
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)  
at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.$anonfun$writeIndexFileAndCommit$2(IndexShuffleBlockResolver.scala:302)  
at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.java:23)  
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)  
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)  
at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:258)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.$anonfun$writeIndexFileAndCommit$1(IndexShuffleBlockResolver.scala:300)  
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)  
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:305)  
at org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile(LocalDiskSingleSpillMapOutputWriter.java:53)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.mergeSpills(UnsafeShuffleWriter.java:280)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:224)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:180)  
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)  
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)  
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)  
at org.apache.spark.scheduler.Task.run(Task.scala:131)  
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)  
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)  
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
at java.lang.Thread.run(Thread.java:748)  
Suppressed: java.io.IOException: No space left on device  
at java.io.FileOutputStream.writeBytes(Native Method)  
at java.io.FileOutputStream.write(FileOutputStream.java:326)  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)  
at java.io.DataOutputStream.flush(DataOutputStream.java:123)  
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.$anonfun$writeIndexFileAndCommit$3(IndexShuffleBlockResolver.scala:305)  
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1448)  
… 15 more  
Suppressed: java.io.IOException: No space left on device  
at java.io.FileOutputStream.writeBytes(Native Method)  
at java.io.FileOutputStream.write(FileOutputStream.java:326)  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)  
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)  
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)  
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)  
… 17 more

Driver stacktrace:  
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258)  
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2207)  
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2206)  
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)  
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)  
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)  
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2206)  
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1079)  
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1079)  
at scala.Option.foreach(Option.scala:407)  
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1079)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2445)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387)  
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376)  
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)  
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2217)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)  
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)  
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)  
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)  
at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)  
at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)  
at is.hail.backend.spark.SparkBackend.parallelizeAndComputeWithIndex(SparkBackend.scala:286)  
at is.hail.backend.BackendUtils.collectDArray(BackendUtils.scala:28)  
at \_\_C1078Compiled.\_\_m1093split\_TailLoop(Emit.scala)  
at \_\_C1078Compiled.\_\_m1079split\_ToArray(Emit.scala)  
at \_\_C1078Compiled.apply(Emit.scala)  
at is.hail.expr.ir.CompileAndEvaluate$.anonfun\_apply$6(CompileAndEvaluate.scala:67)  
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:67)  
at is.hail.expr.ir.CompileAndEvaluate$.$anonfun$apply$1(CompileAndEvaluate.scala:18)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:18)  
at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:740)  
at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:859)  
at is.hail.expr.ir.lowering.LowerTableIR$.apply(LowerTableIR.scala:1198)  
at is.hail.expr.ir.lowering.LowerToCDA$.lower(LowerToCDA.scala:68)  
at is.hail.expr.ir.lowering.LowerToCDA$.apply(LowerToCDA.scala:17)  
at is.hail.expr.ir.lowering.LowerToDistributedArrayPass.transform(LoweringPass.scala:76)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:26)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:66)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:52)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:71)  
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:68)  
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:15)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:15)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:13)  
at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:12)  
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:63)  
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:14)  
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:12)  
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)  
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)  
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)  
at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12)  
at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:46)  
at is.hail.backend.spark.SparkBackend.\_execute(SparkBackend.scala:381)  
at is.hail.backend.spark.SparkBackend.$anonfun$execute$1(SparkBackend.scala:365)  
at is.hail.expr.ir.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:47)  
at is.hail.utils.package$.using(package.scala:638)  
at is.hail.expr.ir.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:47)  
at is.hail.utils.package$.using(package.scala:638)  
at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17)  
at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:46)  
at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275)  
at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:362)  
at is.hail.backend.spark.SparkBackend.$anonfun$executeJSON$1(SparkBackend.scala:406)  
at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52)  
at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:404)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:282)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:238)  
at java.lang.Thread.run(Thread.java:748)

java.io.IOException: No space left on device  
at java.io.FileOutputStream.writeBytes(Native Method)  
at java.io.FileOutputStream.write(FileOutputStream.java:326)  
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)  
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)  
at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.$anonfun$writeIndexFileAndCommit$2(IndexShuffleBlockResolver.scala:302)  
at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.java:23)  
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)  
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)  
at scala.collection.mutable.ArrayOps$ofLong.foreach(ArrayOps.scala:258)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.$anonfun$writeIndexFileAndCommit$1(IndexShuffleBlockResolver.scala:300)  
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)  
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)  
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:305)  
at org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile(LocalDiskSingleSpillMapOutputWriter.java:53)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.mergeSpills(UnsafeShuffleWriter.java:280)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:224)  
at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:180)  
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)  
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)  
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)  
at org.apache.spark.scheduler.Task.run(Task.scala:131)  
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)  
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)  
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.75-52791e9868e4  
Error summary: IOException: No space left on device