Hl.missing is absent in 0.2.57

Hail Query & hailctl
1

Hi.

I am trying to represent missing values and here is the code:

ac = hl.if_else(hl.len(nisc[mt.row_key].info.AC) >= mt.a_index,
             nisc[self.mt.row_key].info.AC[mt.a_index - 1], hl.missing(hl.tint32))

But I need to run it with 0.2.57 Hail version and hl.missing is absent in it. What could I use instead of it? Is there any function that is compatible between the versions?

2

hl.null does what you want.

Why are you stuck on 0.2.57?

1 Like
3

Thank you! We did many runs already with 0.2.57 (generating Elasticsearch indices in the end) and have the code written that works with this version. If we switch to the latest one, would we face any issues or it should be just fine? I am just a bit afraid of any compatibility issues between the old and new versions and/or changes in formatting of values that are written to the Elasticsearch indices (its our last step). If its not of concern, we could just switch to the latest and not stuck.

4

Switching to 0.2.64 should be completely hassle free. 0.2.65 (the latest release) uses Spark 3 by default, but can be built from source for Spark 2 as well. Depending on where you’re running this (on-prem cluster, laptop, cloud, etc.), you might have to tweak your installation / build steps, though the Hail API doesn’t change at all.

5

I actually just tried to switch to 0.2.63 on AWS EMR and it gave me an error, the last part of which is:

  File "/usr/local/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply.
: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

Hmm, I see. Maybe its spark then, not sure.

6

This looks like a spark or Scala version mismatch, I think.