Hi, according to hail documentation currently supported spark version is 3.1.2 (or 3.1.x). Spark released 3.2.0 version in October 2021.
Is there a plan to support this new Spark version (Spark Release 3.2.0 | Apache Spark)?
1 Like
We will probably support Spark 3.2 when major cloud providers have released Spark 3.2 images in general availability.
That said, it’s certainly possible Hail will work out of the box built against 3.2.
Do you have any approx timeline when Spark 3.2.0 will be supported ?
I suspect if you build with Spark 3.2 it’ll work, but we haven’t tried yet. We currently test building against latest 3.0 and 3.1 versions. There are likely no plans to officially support Spark 3.2 for at least next couple of months.
I would be great to make it work on spark 3.2.1, now that dataproc serverless is available. Unfortunatelly, dataproc serverless only allows spark 3.2.1 AFAIK.
Hi @amanas Have you been able to make Hail work with Dataproc Serverless on Spark 3.2 or Spark 3.3? I am trying to build a dependency JAR or bundle it in a custom container image. Any suggestions that you may have will be very helpful. Thank you