I’m pretty sure you’re hitting an infuriating, poorly documented “feature” of --jars
/ spark.jars
that I mention in another discuss post. Short answer: set these two configuration parameters appropriately for your worker nodes:
spark.driver.extraClassPath = ./hail.jar
spark.executor.extraClassPath = ./hail.jar