Dear Hail Team! Came across this issue and thought to bring it to you guys for help. I built the package from Git source (ver=0.1). Then I tested it on your Tutorial > Overview ( https://hail.is/docs/stable/tutorials/hail-overview.html ). Everything works fine until it gets to df = vds.sample…

We really haven’t tested Hail 0.1 against Spark 2.2.0, and have seen problems in the past. Hail 0.2, which will be in beta in several weeks, will support 2.2.0 as the default version I think.

Spark 2.3.0 was just released on Feb 28th and comes with numerous bug fixes and improvements to SparkSQL and PySpark executions, as well as support for the ‘zstd’ compression method which offers ~38% improvement over the snappy default for Parquet files. Would it be possible to consider making Spark…

I think we’re pretty committed to 2.2.0 as the default version for the 0.2 beta version, and probably won’t start deploying 2.3 jars until Google Dataproc starts offering a 2.3 image. We’re also diverging more and more from SparkSQL, and don’t use DataFrames to write parquet anymore due to poor per…

Also, if you’re getting started with Hail now, I’d definitely recommend using the 0.2 development version !

Hi, “We’re also diverging more and more from SparkSQL, and don’t use DataFrames to write parquet anymore due to poor performance going between the DataFrame and RDD APIs.” ----- diverging towards Datasets or to RDD APIs? Because Spark recommends to use Datasets and Dataframes for any future develop…

Hail 0.2 is built entirely on the RDD API. 0.1 used RDD everywhere except to write/read parquet, and this caused huge problems with memory management and performance. Dataframes don’t work for some kinds of computational tasks Hail needs to support – namely, extremely wide tables (sometimes up to 1…

So , 0.2 is still using RDDs as base abstraction?

Yep! Really no choice, for the problems Hail is designed to solve.

Hi Tim, now that a google dataproc image exists for spark 2.3, how to compile Hail 0.2 with this version of spark ? [image] Thanks Ines

see here: [image] Hail 0.2 class not found exception on EMR Hail Query & hailctl It’s totally possible to compile your own! I haven’t done it in a while (since making that .jar and .zip) so I could be wrong about specifics, but all you need to do is pass versions for Spark…

Potential compatibility issue with Spark 2.2.0?

Help [0.1]

tpoterba August 24, 2018, 5:45pm 11

see here:

Topic		Replies	Views
Hail 0.2.27 not running on Cloudera spark 2.4.0 Hail Query & hailctl	8	671	November 19, 2019
Using Hail on Spark 2.1.1 Azure HDInsight causes error Help [0.1]	16	2009	April 6, 2018
Spark version problem Hail Query & hailctl	3	906	April 18, 2019
AttributeError: 'DataFrame' object has no attribute 'to_spark' Help [0.1]	19	7481	August 1, 2018
Install Hail using Spark Hail Query & hailctl	15	1416	April 13, 2018

Potential compatibility issue with Spark 2.2.0?

Related topics