Hi @danielgoldstein , now I got the java.lang.OutOfMemoryError: Java heap space message. Are you sure that 'spark.executor.memory': '1800g' means 1800 GB of RAM per core?
From an old thread I’ve found this, which suggest it’s the total number of RAM?
How do I increase the memory or RAM available to the JVM when I start Hail through Python? - Hail Query & hailctl - Hail Discussion
I actually have no experience with SPARK, so maybe I am on the completely wrong track here.