Error when using the write() function

[ehigham]

Hi @marise,

A couple of things to note here:

First, I suspect this is a problem with the classpath. Can you try running hl.init without starting a spark session? If this works, then you’ll need to manually configure spark as Dan suggested in the post below before you create the session. You can do this via pyspark.SparkConf in your python session.

Initialise Hail with existing Spark

Hi @ireneisdoomed ! Are the Hail JARs in the class path of your Spark cluster? There are some details on using arbitrary Spark clusters in the docs. For example, if you’re using spark-submit, you need to specify the JAR paths: HAIL_HOME=$(pip3 show hail | grep Location | awk -F' ' '{print $2 "/hail"}') spark-submit \ --jars $HAIL_HOME/hail-all-spark.jar \ --conf spark.driver.extraClassPath=$HAIL_HOME/hail-all-spark.jar \ --conf spark.executor.extraClassPath=./hail-all-spark.jar \ --co…

Secondly, driver_cores has no effect if you’re creating your own spark session. Also worker_memory applies to the 'batch' backend only.

Hope this helps,