# Updating index gives number of documents in the index cannot exceed 2147483519 **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [May 20, 2021, 1:33pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052 "2021-05-20T13:33:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [May 20, 2021, 1:33pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/1 "2021-05-20T13:33:38Z")
We have an **Elasticsearch** index which has **42522080** documents. We try updating it by retrieving **VCF** file from which the index was generated (from ES index metadata), annotating it and then writing it back. The code is presented below: ``` mt = hl.import_vcf(dataset_path, reference_genome='GRCh' + genome_version, force_bgz=True, min_partitions=500) mt = hl.split_multi_hts(mt.annotate_rows(locus_old=mt.locus, alleles_old=mt.alleles), permit_shuffle=True) ... variant_count = mt.count_rows() logger.info("\n==> exporting {} variants to elasticsearch:".format(variant_count)) row_table = mt.rows().flatten() row_table = row_table.drop(row_table.locus, row_table.alleles) hl.export_elasticsearch(row_table, ...) ``` We verified that variant\_count is **42522080** however, when **export\_elasticsearch** function is run the error is produced: > hail.utils.java.FatalError: EsHadoopException: Could not write all entries for bulk operation [93/1000]. Error sample (first [5] error messages): > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > number of documents in the index cannot exceed 2147483519 > Bailing out… So, while writing it into Elasticsearch the number of documents generated and written seems to exceed the Lucene’s capacity (which is strange given that the total number of variants is **42522080** ) > **[Elasticsearch throwing number of documents in the index cannot exceed 2147483519](https://discuss.elastic.co/t/elasticsearch-throwing-number-of-documents-in-the-index-cannot-exceed-2147483519/242512)** > > HI Team, I'm using monthly index to store all my data in elasticsearch. i have enabled DLQ to capture the unformatted log information and in that im getting capture below error. elasticsearch ±Could not index event to Elasticsearch. status:... So, I am wondering whether its the issue of Hail’s **split\_multi\_hts** function, and what I could try in Hail, which parameters, settings to avoid that. I might ask a similar question also on **Elasticsearch** forum since I am not fully sure that it should go here. ---
**Author:** ![nawatts](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@nawatts](https://discuss.hail.is/u/nawatts)\ **Post date:** [May 20, 2021, 2:50pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/2 "2021-05-20T14:50:37Z")
Are you setting [es.mapping.id](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#cfg-mapping) in the config passed to `hl.export_elasticsearch`? If not, duplicate documents may be written to the ES index (for example, if a task running on a preemptible machine is preempted midway through processing a partition). Also, does the mapping for this index include any [nested fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html)? Values for nested fields are stored as separate documents. Thus, when using nested fields, more than one ES document may be created for a row in a Hail Table. ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [May 21, 2021, 2:26pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/3 "2021-05-21T14:26:05Z")
> [@nawatts](#): > > [es.mapping.id](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#cfg-mapping) Yeah, I am using `es.mapping.id`, so its nested fields most probably. Is there a way to calculate the number of all documents that will be generated for a `MatrixTable`, so that I could set the number of primary shards to a different number? . ---
**Author:** ![nawatts](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@nawatts](https://discuss.hail.is/u/nawatts)\ **Post date:** [May 21, 2021, 2:41pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/4 "2021-05-21T14:41:25Z")
It would depend on the mapping used for your ES index. Generally, nested fields are useful for arrays of objects. Is there a field in your table that contains a collection of structs and is mapped as nested? ---
**Author:** ![NLSVTN](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/nlsvtn/32/339_2.png) [@NLSVTN](https://discuss.hail.is/u/NLSVTN)\ **Post date:** [May 21, 2021, 3:46pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/5 "2021-05-21T15:46:24Z")
I see there 3 nested fields: one has 7 array fields, another - 26, third - 5. Would the number be then `(7 + 26 + 5 + 1) * 42522080`? But then its still less than `2147483519` limit. There are no `nested` within `nested`, just `3` `nested` at the same level. Or each item in each array will be like a document? ---
**Author:** ![nawatts](https://avatars.discourse-cdn.com/v4/letter/n/839c29/32.png) [@nawatts](https://discuss.hail.is/u/nawatts)\ **Post date:** [May 21, 2021, 4:05pm UTC](https://discuss.hail.is/t/updating-index-gives-number-of-documents-in-the-index-cannot-exceed-2147483519/2052/6 "2021-05-21T16:05:40Z")
Each element in the array/set/collection would be a separate document for fields mapped as nested. So if a Table row has a field containing an array of length 3, there are 4 documents created for that row.