# Hl.export\_elasticsearch conflict with scala2.12

**URL:** https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826
**Category:** Hail Query & hailctl
**Created:** [December 2, 2020, 5:23am UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826 "2020-12-02T05:23:31Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![mhebrard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mhebrard/32/322_2.png) [@mhebrard](https://discuss.hail.is/u/mhebrard)
#### Post date: [December 2, 2020, 5:23am UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/1 "2020-12-02T05:23:31Z")

</div>

Hi.

I am installing Hail v0.2.60 on spark 3.0.0 / scala 2.12.10

```bash
sudo make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.12.10 SPARK_VERSION=3.0.0

```

I am able to load, process and write data without issues.

The problem rise when I wish to export my table on Elasticsearch. Here I got a error that seems like an incompatibility with scala 2.12.

```python
# Load table
ht_res = hl.read_table('s3://[...].ht')

hl.export_elasticsearch(
    ht_res,
    "[es-URL]",
    [es-port],
    '[index]',
    'documents',
    100,
    config={
        'es.nodes.wan.only':'true',
        'es.batch.write.retry.wait':'60s',
        'es.batch.write.retry.count':'30'
    },
    verbose=True
)

```

> Hail version: 0.2.60-de1845e1c2f6  
> Error summary: NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

---

<div class="post-metadata">

### Author: ![mhebrard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mhebrard/32/322_2.png) [@mhebrard](https://discuss.hail.is/u/mhebrard)
#### Post date: [December 2, 2020, 10:02am UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/2 "2020-12-02T10:02:05Z")

</div>

In the mean time, I am able to install Hail v0.2.60 on spark 2.4.6 / scala 2.11.12

```bash
sudo make install-on-cluster HAIL_COMPILE_NATIVES=1 SCALA_VERSION=2.11.12 SPARK_VERSION=2.4.6

```

In this context `hl.export_elasticsearch()` code above run without issue

---

<div class="post-metadata">

### Author: ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [December 2, 2020, 2:10pm UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/3 "2020-12-02T14:10:41Z")

</div>

You’re right, this is a mistake. We hard code the path to the elasticsearch dependency to use spark 2 and scala 2.11.

I made a github issue here: [https://github.com/hail-is/hail/issues/9767](https://github.com/hail-is/hail/issues/9767)

and assigned it to myself. Will try to address in next few days.

---

<div class="post-metadata">

### Author: ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [December 8, 2020, 4:17pm UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/4 "2020-12-08T16:17:50Z")

</div>

I’m actually not sure how to do this. We use the library here, which is explicitly for spark 2.x:

> **[Maven Repository: org.elasticsearch » elasticsearch-spark-20](https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20)**

I’ve done some quick googling and didn’t immediately find anyone doing this with Spark 3, I’ll have to keep looking.

---

<div class="post-metadata">

### Author: ![nawatts](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@nawatts](https://discuss.hail.is/u/nawatts)
#### Post date: [December 8, 2020, 7:48pm UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/5 "2020-12-08T19:48:03Z")

</div>

There’s an open issue for Spark 3 / Scala 2.12 support in the elasticsearch-hadoop connnector.

> <https://github.com/elastic/elasticsearch-hadoop/issues/1412>
>
> Feature description
> Spark3 is currently in RC. Will there be support for Spark3 in the next release version (v8) or will we...

---

<div class="post-metadata">

### Author: ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)
#### Post date: [December 8, 2020, 7:55pm UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/6 "2020-12-08T19:55:01Z")

</div>

Thanks @nawatts. So I think the answer then is that there’s not currently a way to export to elasticsearch from Spark 3, and until there is Hail likely won’t support it.

---

<div class="post-metadata">

### Author: ![nawatts](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@nawatts](https://discuss.hail.is/u/nawatts)
#### Post date: [December 8, 2020, 8:08pm UTC](https://discuss.hail.is/t/hl-export-elasticsearch-conflict-with-scala2-12/1826/7 "2020-12-08T20:08:14Z")

</div>

Also, [https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html) lists (in the Apache Spark section) supported Spark / Scala versions and their corresponding ES-Hadoop artifact ID.
