# Hl.vds.new\_combiner gives unexpected TypeError

**URL:** <https://discuss.hail.is/t/hl-vds-new-combiner-gives-unexpected-typeerror/3822>\
**Category:** Hail Query & hailctl\
**Created:** [May 5, 2024, 5:38pm UTC](https://discuss.hail.is/t/hl-vds-new-combiner-gives-unexpected-typeerror/3822 "2024-05-05T17:38:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![f-ferraro](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/f-ferraro/32/1139_2.png) [@f-ferraro](https://discuss.hail.is/u/f-ferraro)\
**Post date:** [May 5, 2024, 5:38pm UTC](https://discuss.hail.is/t/hl-vds-new-combiner-gives-unexpected-typeerror/3822/1 "2024-05-05T17:38:17Z")

</div>

Hi,

Thank you so much for making hail available to the community!

I am running hail locally on my institution’s cluster.  
I have ~500 gVCFs generated in the same way.  
I created a VDS with `hl.vds.new_combiner`, which worked very well and I could extract variants as desired.

I am now trying to add the remaining ~470 with

```python
combiner = hl.vds.new_combiner(
    output_path="/mnt/data/db_merged.vds",
    temp_path="/mnt/data/projects/.tmp",
    gvcf_paths=gvcfs_f,
    vds_paths=["/mnt/data/db.vds"],
    use_genome_default_intervals=True,
    reference_genome="GRCh38"
)

```

However eventually I get the following error:

```python
Traceback (most recent call last):==========================>(2570 + 16) / 2586]
  File "/mnt/data/projects/add_to_vds_db.py", line 31, in <module>
    combiner.run()
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/vds/combiner/variant_dataset_combiner.py", line 356, in run
    self.step()
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/vds/combiner/variant_dataset_combiner.py", line 430, in step
    self._step_vdses()
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/vds/combiner/variant_dataset_combiner.py", line 484, in _step_vdses
    combined = combine_variant_datasets(vdss)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/vds/combiner/combine.py", line 380, in combine_variant_datasets
    reference = combine_references([vds.reference_data for vds in vdss])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/vds/combiner/combine.py", line 374, in combine_references
    ts = hl.Table.multi_way_zip_join([localize(mt) for mt in mts], 'data', 'g')
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<decorator-gen-1278>", line 2, in multi_way_zip_join
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data/appl/conda/miniconda3/envs/hail/lib/python3.12/site-packages/hail/table.py", line 4634, in multi_way_zip_join
    raise TypeError(
TypeError: All input tables to multi_way_zip_join must have the same row type
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>, LAD: array<int32>}>}
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>}>}
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>}>}
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>}>}
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>}>}
  struct{locus: locus<GRCh38>, __entries: array<struct{END: int32, DP: int32, GQ: int32, ICNT: array<int32>, MIN_DP: int32, SPL: array<int32>}>}

```

I guess the issue is that the vds also has a LAD field but I’ve not figured out how to remove it before merging the other gVCFs. Could you please help me out? 🙂  
Also, I cannot directly overwrite the output, right?

---

<div class="post-metadata">

**Author:** ![patrick-schultz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/patrick-schultz/32/265_2.png) [@patrick-schultz](https://discuss.hail.is/u/patrick-schultz)\
**Post date:** [May 6, 2024, 7:33pm UTC](https://discuss.hail.is/t/hl-vds-new-combiner-gives-unexpected-typeerror/3822/2 "2024-05-06T19:33:44Z")

</div>

Hi @f-ferraro!

If the vds you already created has an LAD field, but the new gvcfs don’t, I think you can just replace it with one with LAD removed. I would create the modified vds while keeping the old one until you know it has worked.

```python
vds = hl.vds.read_vds("/mnt/data/db.vds")
hl.vds.VariantDataset(
  vds.reference_data.drop('LAD'),
  vds.variant_data,
).write("/mnt/data/db_modified.vds")

```

You can drop LAD from the variant data too if necessary. Then just run the combiner again using the modified vds.

Flagging @chrisvittal to double check this.

---

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [May 7, 2024, 4:31pm UTC](https://discuss.hail.is/t/hl-vds-new-combiner-gives-unexpected-typeerror/3822/3 "2024-05-07T16:31:21Z")

</div>

This is correct. You need to align the types of the inputs, and write it to a temporary path.