# VariantDatasetCombiner - dataset contains both multiallelic variants and duplicated loci for review **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [January 3, 2023, 2:50pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021 "2023-01-03T14:50:08Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 2:50pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/1 "2023-01-03T14:50:08Z")
Hello, I am trying to run VariantDatasetCombiner: path\_to\_input\_list = ‘input\_files.txt’ # a file with one GVCF path per line gvcfs = with hl.hadoop\_open(path\_to\_input\_list, ‘r’) as f: for line in f: print(line) gvcfs.append(line.strip()) combiner = hl.vds.new\_combiner( output\_path=‘dataset.vds’, temp\_path=‘my-temp-bucket’, gvcf\_paths=gvcfs, use\_genome\_default\_intervals=True, reference\_genome=‘GRCh38’, ) combiner.run() But an error showed up: Current key: { locus: { contig: chrM, position: 302 }, alleles: [3; A, AC, ACC] } Previous key: { locus: { contig: chrM, position: 302 }, alleles: [3; A, AC, C] } This error can occur after a split\_multi if the dataset contains both multiallelic variants and duplicated loci. How can I tell the combiner not to split\_multi or how should I preprocess the gvfc files? In addition I would like to mention that using hl.experimental.run\_combiner was working with the same files: import hail as hl path\_to\_input\_list = ‘input\_files.txt’ # a file with one GVCF path per line inputs = with hl.hadoop\_open(path\_to\_input\_list, ‘r’) as f: for line in f: print(line) inputs.append(line.strip()) output\_file = ‘output.mt’ # output destination temp\_bucket = ‘my-temp-bucket’ # bucket for storing intermediate files hl.experimental.run\_combiner(inputs, out\_file=output\_file, tmp\_path=temp\_bucket, reference\_genome=‘GRCh38’, use\_genome\_default\_intervals=True, overwrite=True) But I need to use methods that VariantDataset includes in my analysis. I would appreciate your help. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [January 3, 2023, 2:57pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/2 "2023-01-03T14:57:57Z")
Can you share the full stack trace to this error? I’m certainly concerned to see this error popping out of the VDS combiner. ---
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 3:02pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/3 "2023-01-03T15:02:47Z")
It is too long I send it in a text file. And thank you for trying to help me. 🙂 [error\_hail.txt](https://discuss.hail.is/uploads/short-url/wTz9EcYJRbNhaBTj4dOsodPxgff.txt) (40.6 KB) ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [January 3, 2023, 3:16pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/4 "2023-01-03T15:16:38Z")
Is it possible that your GVCFs have duplicate loci? Could you use tabix to look up `chrM:302` for each GVCF and ensure that each GVCF has at most one record for that site? ---
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 3:34pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/5 "2023-01-03T15:34:03Z")
1. gvcf: ![image](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/d1fbc41f2194c8019849b6286678b4ed8c363a38.png) ---
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 3:34pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/6 "2023-01-03T15:34:43Z")
2.gvcf: ![image](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/29630fdd059178971d8c91fbcda4e99959a15d14.png) ---
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 3:35pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/7 "2023-01-03T15:35:43Z")
3.gvcf do not have chrM:302 ![image](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/547af2e687ccac782d432e2b6702e4617e395f12.png) ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [January 3, 2023, 3:36pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/8 "2023-01-03T15:36:52Z")
Ack, this is the reason. Hail’s combiner data model assumes that GVCFs use multiallelic rather than split biallelic representations for multiple polymorphisms at the same locus. What caller are these from? Is there an unsplit (multiallelic) version you could use? ---
**Author:** ![gameforcela12](https://avatars.discourse-cdn.com/v4/letter/g/e19b73/32.png) [@gameforcela12](https://discuss.hail.is/u/gameforcela12)\ **Post date:** [January 3, 2023, 3:38pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/9 "2023-01-03T15:38:27Z")
To be honest I do not know, I will need to get this information from leader of my project. Probably tomorrow. And if we do not have another variant of gvfc, is there any way around this? ---
**Author:** ![joonan30](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/joonan30/32/989_2.png) [@joonan30](https://discuss.hail.is/u/joonan30)\ **Post date:** [June 30, 2023, 2:48pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/10 "2023-06-30T14:48:58Z")
I got the same error on chrM 302 locus. It’s a multi allele. My gVCF is from Dragen v.4.0. Is there any function to use specific contigs or exclude chrM in `hl.vds.new_combiner`. I thought `contig_recoding` might work on specific contig but it did not work (though it meant to work for rename contigs). ```python combiner = hl.vds.new_combiner( output_path='/Users/joonan/tmp/output/dataset.vds', temp_path='/Users/joonan/tmp/temp', gvcf_paths=gvcfs, reference_genome='GRCh38', contig_recoding = {'chr1':'chr1'}, use_genome_default_intervals=True, ) ``` Thanks, ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [June 30, 2023, 3:32pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/11 "2023-06-30T15:32:24Z")
There is not currently a way to filter GVCFs during combination (I’ve created [an issue for this](https://github.com/hail-is/hail/issues/13223)). Can you confirm that your GVCF file has multiple lines with the 302 locus? Do you know if Dragen has a configuration that prevent this? In the short-term, I think your best bet is to filter the GVCFs by hand with awk or something. Apologies! ---
**Author:** ![joonan30](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/joonan30/32/989_2.png) [@joonan30](https://discuss.hail.is/u/joonan30)\ **Post date:** [June 30, 2023, 3:45pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/12 "2023-06-30T15:45:28Z")
Hi Dan, Thanks for prompt reply. I am not sure whether Dragen has a configuration for this but re-processing with a new configuration for thousands samples would be difficult (regarding cost). awk or bcftools are something I can work on but not really fast as Hail. Lot of pain. It would be great if you have some features to include specific contigs in future dev. Best, ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [July 13, 2023, 6:14pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/13 "2023-07-13T18:14:28Z")
Hey @joonan30 ! I’m tracking this down a bit more to hopefully save others this trouble in the future. Did you use a mitochondrial specific version of the DRAGEN caller or did this use the regular settings on data from chrM? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [July 24, 2023, 6:14pm UTC](https://discuss.hail.is/t/variantdatasetcombiner-dataset-contains-both-multiallelic-variants-and-duplicated-loci-for-review/3021/14 "2023-07-24T18:14:51Z")
@joonan30 , any chance you could share the settings you used for DRAGEN?