Hi all, hope you guys are well! I had a simple question. I have a set ~900 sample-specific VCFs that I would like to import and combine in Hail. I tried on a small list of two vcfs (if this command is correct) and receiving the below error vcf_list = [“gs://maze-data-external-private/answer-als/…

Are these GVCFs (with reference blocks)? If so, you should use the vcf combiner . If not, are they from genotype arrays (same variants for every VCF)? If the answer to both is no, we can still merge, but it’s going to be lossy and you’ll end up with some missing data where a given sample didn’t have…

These are all per-sample annotated VCFs from whole genome sequencing data. There are about 900 of them. I figured Hail was the quickest way to combine, but let me know if I’m mistaken!

[image] beryl_maze: These are all per-sample annotated VCFs from whole genome sequencing data So they have non-overlapping variants, yes? Also, what is the handling of INFO fields you want here?

Yes, they have non-overlapping variants, that will be singletons and which we definitely want to preserve. In terms of info field, my main interest is combining into a single VCF to be able to do variant QC, so I’d like to maintain the GQ, DP - but I think that’s calculated from the entries right? …

Also just to be extra clear, this is what the VCF itself looks like for example. [image]

Here’s something that may work: def multi_way_union_mts(mts: list, tmp_dir: str, chunk_size: int) -> hl.MatrixTable: """Joins MatrixTables in the provided list :param list mts: list of MatrixTables to join together :param str tmp_dir: path to temporary directory for intermediate results…

great thanks Tim! I tried this on the small list of two VCFs and it works. Will try on the larger list now. appreciate it!

things like variantqc/sampleqc won’t work out of the box, I think, because you need to fill in missing homrefs. Something like: mt = mt.annotate_entries(present_in_original = True) mt = mt.unfilter_entries() # densify the missing entries mt = mt.annotate_entries(GT = hl.coalesce(mt.GT, hl.call(0, 0…

Hi @tpoterba , I have testing this function to merge many single sample .mts to one big .mt. I have noticed I am missing certain rows after the merge. For example this is how individual .mt looks like: ---------------------------------------- Global fields: None --------------------------------…

Importing many sample-specific VCFs

Hail Query & hailctl

beryl_maze April 7, 2021, 3:00pm 4

I also realize this might be a duplicate of Merge vcfs like bcftools merge - #5 by njbernstein but I couldn’t quite see where in Konrad’s code he was referring to

Topic		Replies	Views
Store multiple vcfs into single MatrixTable Hail Query & hailctl	10	806	September 9, 2020
Merging single sample MatrixTables into one big MatrixTable Hail Query & hailctl	3	786	November 18, 2022
Possible vcf_combiner issue Hail Query & hailctl	19	1357	June 15, 2020
Import multi sample vcf Hail Query & hailctl	6	595	October 24, 2022
FatalError: StackOverflowError: null when combining single sample vcfs Hail Query & hailctl	3	602	September 8, 2021

Importing many sample-specific VCFs

Related topics