Hello, I am wondering if you have any suggestions for how to proceed with using hail to sampleqc roughly 500k WES samples on the UK biobank database. The database contains WES data in PLINK format (fractioned by chr), BGEN format (fractioned by chr), and VCFs (fractioned both by individual partici…

OK, I’ve been meaning to write most of this out anyway, so this is a good chance to do that. I start with some background and definitions before giving you practical advice. We view sequencing as a series of transformations of data: DNA ==sequencer=> FASTQ ==alignment=> BAM/CRAM ==variant caller=> …

Hey @mgarcia ! A couple things: No matter what, this will be a very expensive analysis. 500k samples is a huge data set. Quality control is an iterative and explorative process. Hail is complex software, you’ll probably make mistakes as you learn to use it. Where/how did you get a VCF partitioned…

Hi @danking , Thank you for your response. To answer your first question, I have attached this excerpt from the UK Biobank that references the first 200k exomes they released (same protocol for the full 500k): Primary and secondary analysis for the UKB 200k release was performed with an updated Fun…

OK, I understand your situation. The answer will take a bit of writing to properly describe and I’m a bit busy tomorrow. Hopefully I’ll have time tomorrow to type it all out

[image] mgarcia: This might be a novice question, but why exactly is it preferable to use a GVCF in these large scale analyses over a VCF? I understand that GVCFs keeps records of all sites, whether or not there exists a variant call, but how exactly is that advantageous? Just to be crystal c…

Thank you so much for your incredibly detailed response! Yes, there is a PVCF partitioned by variant that appears to be our best choice moving forward. We greatly appreciate the help.

Performing SampleQC using Hail on ~500k WES samples

Hail Query & hailctl

Topic		Replies	Views
Working with large VCFs (e.g. from UK Biobank) is slow Hail Query & hailctl	12	2071	August 23, 2024
Poor performance for QC filtering on medium sized genotype data Hail Query & hailctl	20	2266	February 8, 2020
Filter variants by sample id in gVCF Help [0.1]	20	1574	February 27, 2019
Importing many sample-specific VCFs Hail Query & hailctl	12	1268	December 12, 2022
Import multi sample vcf Hail Query & hailctl	6	595	October 24, 2022

Performing SampleQC using Hail on ~500k WES samples

Related topics