Hi Hail team,
I’m converting a large VDS to VCF via WDL and hitting what looks like a partioning-related error:
Traceback (most recent call last): File “”, line 69, in File “”, line 2, in export_vcf File “/usr/local/lib/python3.11/dist-packages/hail/typecheck/check.py”, line 585, in wrapper return __original_func(*args_, **kwargs_) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/hail/methods/impex.py”, line 636, in export_vcf Env.backend().execute(ir.MatrixWrite(dataset._mir, writer)) File “/usr/local/lib/python3.11/dist-packages/hail/backend/spark_backend.py”, line 217, in execute raise err File “/usr/local/lib/python3.11/dist-packages/hail/backend/spark_backend.py”, line 209, in execute return super().execute(ir, timed) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/hail/backend/backend.py”, line 181, in execute raise e.maybe_user_error(ir) from None File “/usr/local/lib/python3.11/dist-packages/hail/backend/backend.py”, line 179, in execute result, timings = self._rpc(ActionTag.EXECUTE, payload) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/dist-packages/hail/backend/py4j_backend.py”, line 232, in _rpc raise fatal_error_from_java_error_triplet( hail.utils.java.FatalError: IllegalArgumentException: requirement failed Java stack trace: java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:268) at is.hail.rvd.RVDPartitioner.(RVDPartitioner.scala:60) at is.hail.rvd.RVDPartitioner.(RVDPartitioner.scala:39) at is.hail.expr.ir.TableNativeZippedReader.$anonfun$lower$5(TableIR.scala:1992) at scala.Option.map(Option.scala:230) at is.hail.expr.ir.TableNativeZippedReader.lower(TableIR.scala:1992) at is.hail.expr.ir.TableReader.lower(TableIR.scala:611) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1063) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1261) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1197) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2245) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2026) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2026) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2112) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1252) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2026) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2245) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:2245) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:1052) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1656) at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:729) at is.hail.expr.ir.lowering.LowerTableIR$.apply(LowerTableIR.scala:1022) at is.hail.expr.ir.lowering.LowerToCDA$.lower(LowerToCDA.scala:27) at is.hail.expr.ir.lowering.LowerToCDA$.apply(LowerToCDA.scala:11) at is.hail.expr.ir.lowering.LowerToDistributedArrayPass.transform(LoweringPass.scala:94) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:28) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:60) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:65) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:86) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:37) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:98) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:37) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:98) at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:35) at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:34) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:81) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$2(LoweringPipeline.scala:22) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$2$adapted(LoweringPipeline.scala:20) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:20) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:98) at is.hail.backend.ExecuteContext.time(ExecuteContext.scala:183) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:11) at is.hail.expr.ir.CompileAndEvaluate$.anonfun_apply$1(CompileAndEvaluate.scala:48) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:98) at is.hail.backend.ExecuteContext.time(ExecuteContext.scala:183) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:47) at is.hail.backend.spark.SparkBackend.$anonfun$execute$1(SparkBackend.scala:550) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:98) at is.hail.backend.ExecuteContext.time(ExecuteContext.scala:183) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:539) at is.hail.backend.BackendHttpHandler.$anonfun$handle$4(BackendServer.scala:93) at is.hail.utils.package$.using(package.scala:673) at is.hail.backend.ExecuteContext.local(ExecuteContext.scala:220) at is.hail.backend.BackendHttpHandler.$anonfun$handle$3(BackendServer.scala:91) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:15) at is.hail.backend.BackendHttpHandler.$anonfun$handle$2(BackendServer.scala:90) at is.hail.backend.BackendHttpHandler.$anonfun$handle$2$adapted(BackendServer.scala:89) at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:100) at is.hail.utils.package$.using(package.scala:673) at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:100) at is.hail.utils.package$.using(package.scala:673) at is.hail.annotations.RegionPool.scopedRegion(RegionPool.scala:166) at is.hail.backend.ExecuteContext$.$anonfun$scoped$1(ExecuteContext.scala:83) at is.hail.utils.package$.using(package.scala:673) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:13) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:82) at is.hail.backend.spark.SparkBackend.$anonfun$withExecuteContext$1(SparkBackend.scala:406) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:15) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:22) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:387) at is.hail.backend.BackendHttpHandler.handle(BackendServer.scala:89) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:848) at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77) at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:817) at jdk.httpserver/sun.net.httpserver.ServerImpl$DefaultExecutor.execute(ServerImpl.java:201) at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.handle(ServerImpl.java:560) at jdk.httpserver/sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:525) at java.base/java.lang.Thread.run(Thread.java:829) Hail version: 0.2.134-952ae203dbbe Error summary: IllegalArgumentException: requirement failed
Anything I can do about this? Thanks for any advice.