# Hail cluster mode output file error

**URL:** <https://discuss.hail.is/t/hail-cluster-mode-output-file-error/2419>\
**Category:** Hail Query & hailctl\
**Created:** [December 29, 2021, 7:27am UTC](https://discuss.hail.is/t/hail-cluster-mode-output-file-error/2419 "2021-12-29T07:27:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ksmpooh](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ksmpooh/32/619_2.png) [@ksmpooh](https://discuss.hail.is/u/ksmpooh)\
**Post date:** [December 29, 2021, 7:27am UTC](https://discuss.hail.is/t/hail-cluster-mode-output-file-error/2419/1 "2021-12-29T07:27:23Z")

</div>

Hello Everyone.

I try to set hail on a spark cluster using 2 workstations. (File system HDFS)

Workstation 1 : Spark (master node and worker node), HDFS (NameNode and datanode)  
Workstation 2 : Spark (worker node), HDFS (datanode)

Spark version : 3.1.2  
Hadoop version : 3.3.0

It worked well when it was just a master node.

First test : just a master node.  
-code-

> ~/hadoop-3.3.0/sbin/start-dfs.sh  
> jps  
> 92803 SecondaryNameNode  
> 92293 NameNode  
> 96950 Jps  
> 92486 DataNode  
> sudo ~/spark-3.1.2/sbin/start-master.sh  
> sudo ~/spark-3.1.2/sbin/start-worker.sh spark://genome101:7077  
> sudo python3  
> import hail as hl  
> hl.init(master=‘spark://genome101:7077’)  
> mt = hl.balding\_nichols\_model(n\_populations=3,  
> n\_samples=500,  
> n\_variants=500\_000,  
> n\_partitions=32)  
> mt = mt.annotate\_cols(drinks\_coffee = hl.rand\_bool(0.33))  
> gwas = hl.linear\_regression\_rows(y=mt.drinks\_coffee,  
> x=mt.GT.n\_alt\_alleles(),  
> covariates=[1.0])  
> gwas.export(‘hdfs://genome101:9000/user/test/test.tsv’)  
> 2021-12-29 13:40:35 Hail: INFO: merging 32 files totalling 42.8M…26 + 6) / 32]  
> 2021-12-29 13:40:36 Hail: INFO: while writing:  
> hdfs://genome101:9000/user/test/test.tsv  
> merge time: 604.267ms

* * *

But it makes error to do it with a worker node

second test : master and worker node.  
-code-

> ~/hadoop-3.3.0/sbin/start-dfs.sh ## master  
> jps ## master  
> 92803 SecondaryNameNode  
> 92293 NameNode  
> 96950 Jps  
> 92486 DataNode

> jps ## worker  
> 103746 Jps  
> 103365 DataNode

master

> sudo ~/spark-3.1.2/sbin/start-master.sh  
> sudo ~/spark-3.1.2/sbin/start-worker.sh spark://genome101:7077

worker

> sudo ~/spark-3.1.2/sbin/start-worker.sh spark://genome101:7077

master

> sudo python3  
> import hail as hl  
> hl.init(master=‘spark://genome101:7077’)  
> mt = hl.balding\_nichols\_model(n\_populations=3,  
> n\_samples=500,  
> n\_variants=500\_000,  
> n\_partitions=32)  
> mt = mt.annotate\_cols(drinks\_coffee = hl.rand\_bool(0.33))  
> gwas = hl.linear\_regression\_rows(y=mt.drinks\_coffee,  
> x=mt.GT.n\_alt\_alleles(),  
> covariates=[1.0])  
> gwas.export(‘hdfs://genome101:9000/user/test/test.tsv’)

* * *

======================error message======================  
Traceback (most recent call last):============================\> (30 + 2) / 32]  
File “”, line 1, in   
File “”, line 2, in export  
File “/usr/local/lib/python3.8/dist-packages/hail/typecheck/check.py”, line 577, in wrapper  
return _original\_func(\*args, \*\*kwargs_)  
File “/usr/local/lib/python3.8/dist-packages/hail/table.py”, line 1045, in export  
Env.backend().execute(  
File “/usr/local/lib/python3.8/dist-packages/hail/backend/py4j\_backend.py”, line 110, in execute  
raise e  
File “/usr/local/lib/python3.8/dist-packages/hail/backend/py4j\_backend.py”, line 86, in execute  
result\_tuple = self.\_jhc.backend().executeEncode(jir, stream\_codec)  
File “/usr/local/lib/python3.8/dist-packages/py4j/java\_gateway.py”, line 1304, in **call**  
return\_value = get\_return\_value(  
File “/usr/local/lib/python3.8/dist-packages/hail/backend/py4j\_backend.py”, line 29, in deco  
raise FatalError(‘%s\n\nJava stack trace:\n%s\n’  
hail.utils.java.FatalError: HailException: Expected 32 part files but found 16

Java stack trace:  
is.hail.utils.HailException: Expected 32 part files but found 16  
at is.hail.utils.ErrorHandling.fatal(ErrorHandling.scala:11)  
at is.hail.utils.ErrorHandling.fatal$(ErrorHandling.scala:11)  
at is.hail.utils.package$.fatal(package.scala:78)  
at is.hail.io.fs.FS.copyMerge(FS.scala:264)  
at is.hail.io.fs.FS.copyMerge$(FS.scala:238)  
at is.hail.io.fs.HadoopFS.copyMerge(HadoopFS.scala:70)  
at is.hail.utils.richUtils.RichRDD$.writeTable$extension(RichRDD.scala:118)  
at is.hail.expr.ir.TableValue.export(TableValue.scala:138)  
at is.hail.expr.ir.TableTextWriter.apply(TableWriter.scala:355)  
at is.hail.expr.ir.Interpret$.run(Interpret.scala:852)  
at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:57)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:20)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:67)  
at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:72)  
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:69)  
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16)  
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81)  
at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14)  
at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13)  
at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:64)  
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:15)  
at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:13)  
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)  
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)  
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)  
at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:13)  
at is.hail.expr.ir.CompileAndEvaluate$.\_apply(CompileAndEvaluate.scala:47)  
at is.hail.backend.spark.SparkBackend.\_execute(SparkBackend.scala:381)  
at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$2(SparkBackend.scala:417)  
at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:47)  
at is.hail.utils.package$.using(package.scala:638)  
at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:47)  
at is.hail.utils.package$.using(package.scala:638)  
at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17)  
at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46)  
at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275)  
at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$1(SparkBackend.scala:414)  
at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52)  
at is.hail.backend.spark.SparkBackend.executeEncode(SparkBackend.scala:413)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:282)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:238)  
at java.lang.Thread.run(Thread.java:748)  
Hail version: 0.2.78-2f7f9e231aaa  
Error summary: HailException: Expected 32 part files but found 16

Anyone knows this problem?

Thank you

---

<div class="post-metadata">

**Author:** ![ksmpooh](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/ksmpooh/32/619_2.png) [@ksmpooh](https://discuss.hail.is/u/ksmpooh)\
**Post date:** [January 17, 2022, 2:03am UTC](https://discuss.hail.is/t/hail-cluster-mode-output-file-error/2419/2 "2022-01-17T02:03:29Z")

</div>

I Found the solution.

When hl.init(), set “tmp\_dir” option

```auto
hl.init(‘spark://genome101:7077’,tmp_dir = "hdfs://genome101:9000/user/tmp/")

```

than, this error was soluted!