Duplicate sample IDs accepted silently by new_combiner and what is the supported way to drop one column?

Hail Query & hailctl
1

Hi,

Up front: this is not a bug report. The combiner did exactly what my inputs told it to do, and I have already traced the root cause on my side. What I am asking about is design and supported usage

two questions:

A. Why does new_combiner accept duplicate sample IDs across its inputs without any warning or error?
B. What is the officially supported way to drop one column of a duplicated pair from an already-merged VDS?

Environment:

  • Hail version: <FILL IN, e.g. 0.2.13x>
  • Platform: Dataproc on GCP, Spark
  • Reference genome: GRCh38

What happened:

I built 45 batch VDSes from single-sample gVCFs (25 samples per batch, the last one 20), then merged them in a separate step using vds_paths only:

combiner = hl.vds.new_combiner(
    output_path=VDS_OUT,
    temp_path=MERGE_TEMP,
    gvcf_paths=[],
    vds_paths=batch_vds_paths,   # 45 batch VDS folders
    reference_genome='GRCh38',
    use_genome_default_intervals=True,
    branch_factor=100,
    target_records=30000,
    force=True,
)
combiner.run()

The merge completed without any error or warning. The output VDS has 1120 columns but only 1035 unique sample IDs: 85 samples each appear as exactly two columns with an identical s.

Root cause, already diagnosed:

I collected variant_data.cols().s from each of the 45 input batch VDSes. Every batch is internally clean (no duplicate IDs within a batch), but 85 sample IDs appear in two different batches. The overlap comes in contiguous blocks — for example batch_0017 columns 0-9 hold the same sample IDs as batch_0033 columns 15-24. So the same gVCFs were listed twice when batches were assigned upstream, and the combiner faithfully carried both copies through. My input error.

Question A silent acceptance of duplicate sample IDs

Is allowing duplicate s across vds_paths and gvcf_paths a deliberate design decision, or simply an unchecked case? I can imagine reasons to allow it (the combiner is a mechanical union and should not impose semantics on sample names), but I would like to understand the intended contract: is uniqueness entirely the caller’s responsibility?

If it is, would a preflight validation or even just a warning be a reasonable addition? Collecting and comparing column names across inputs is cheap relative to the merge itself: in my case the check takes minutes, while the merge takes hours on a 168-core cluster. The duplication passed silently all the way through that merge and only surfaced much later during relatedness analysis, where the duplicated IDs appeared as self-pairs in the pc_relate output. A warning at plan time would have saved the entire run.

Related: is a VDS containing duplicate sample IDs considered valid input to the rest of the hl.vds API? I would like to know whether any operation would silently misbehave rather than fail loudly.

Question B supported way to remove one duplicate column

Re-running the combiner on a corrected input list is expensive for us, so I would prefer to repair the existing output if that is safe and supported.

hl.vds.filter_samples matches on sample ID, so it keeps both columns of a duplicated pair and cannot express “keep only one of these two”. The approach I have in mind is index-based filtering applied to both components, then reconstructing the VDS:

vds = hl.vds.read_vds(VDS_OUT)

vd = vds.variant_data.add_col_index()
vd = vd.filter_cols(keep_idx.contains(vd.col_idx)).drop('col_idx')

rd = vds.reference_data.add_col_index()
rd = rd.filter_cols(keep_idx.contains(rd.col_idx)).drop('col_idx')

hl.vds.VariantDataset(rd, vd).write(DEDUP_VDS)

Specifically:

  1. Is constructing a VDS this way hl.vds.VariantDataset(rd, vd) from separately filtered components supported, or does it bypass invariants the format relies on?

  2. Is column order guaranteed to be identical between reference_data and variant_data, so that filtering both on the same column indices keeps them aligned? Is that an invariant I can rely on, or an implementation detail that could change between versions?

  3. Is there an existing helper I have missed something that selects columns by index rather than by sample ID?

  4. Would the resulting VDS be fully equivalent to one produced by re-running the combiner on a deduplicated input list, or are there fields (reference block structure, entry-level local allele indices, metadata) that would differ in a way that matters downstream?

Thanks.