Export ElasticSearch error

Hail Query & hailctl
1

Hi,

I am trying to export to elasticsearch but I get an error. I don’t if is for the version of ES or some configuration.

Traceback (most recent call last):
File “”, line 1, in
File “”, line 2, in export_elasticsearch
File “/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py”, line 614, in wrapper
return original_func(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/hail/methods/impex.py”, line 2456, in export_elasticsearch
Env.hail().io.ElasticsearchConnector.export(jdf, host, port, index, index_type, block_size, config, verbose)
File “/usr/local/lib/python3.7/site-packages/py4j/java_gateway.py”, line 1257, in call
answer, self.gateway_client, self.target_id, self.name)
File “/usr/local/lib/python3.7/site-packages/hail/backend/spark_backend.py”, line 42, in deco
‘Error summary: %s’ % (deepest, full, hail.version, deepest)) from None
hail.utils.java.FatalError: EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.9.3

Java stack trace:
is.hail.relocated.org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting ‘es.nodes.wan.only’
at is.hail.relocated.org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:327)
at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:97)
at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:83)
at is.hail.relocated.org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:49)
at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:44)
at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:20)
at is.hail.io.ElasticsearchConnector.export(ElasticsearchConnector.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

is.hail.relocated.org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.9.3
at is.hail.relocated.org.elasticsearch.hadoop.util.EsMajorVersion.parse(EsMajorVersion.java:82)
at is.hail.relocated.org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:578)
at is.hail.relocated.org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:320)
at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:97)
at is.hail.relocated.org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:83)
at is.hail.relocated.org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:49)
at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:44)
at is.hail.io.ElasticsearchConnector$.export(ElasticsearchConnector.scala:20)
at is.hail.io.ElasticsearchConnector.export(ElasticsearchConnector.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.57-582b2e31b8bd
Error summary: EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.9.3

2

For example now I tried directly with spark and it works… with the jar elasticsearch-hadoop-7.9.3.jar. So my question is, which jar are you using to connect to ES?

3

Some help with this issue?. Could please tell me the elasticsearch-hadoop version that uses Hail 0.2?. Now I have to convert to spark with to_spark but this method is really very very slow.

4

Hail uses elasticsearch-spark-20_2.11:6.2.4 (https://github.com/hail-is/hail/search?q=elasticsearch+spark).

That version of ES-hadoop only supports up to Elasticsearch 6.8 (https://www.elastic.co/support/matrix#matrix_compatibility).

More info on support for Spark/Scala versions at https://www.elastic.co/guide/en/elasticsearch/hadoop/master/install.html.

5

Ok, thank you