Loading many datasets into single VDS?

Help [0.1]
1

Hi Hail gurus, thanks for this great tool.

I have a big collection of VCF files, one per individual, sitting in a bunch of S3 folders. Each VCF contains several million SNPs.
Is there a way of loading all of them into one big VDS?

I tried importing them one-by-one with import_vcf, then joining using VariantDataset.join(), but join() only accepts one VDS - I need to join a list.

Thanks, any help greatly appreciated,
Oron

2

Hi Oron,
Unfortunately, this isn’t going to be easy. There are two problems:

  1. The join function (which is named terribly by the way, it’s really a union_samples function) only accepts two datasets, and does work proportional to the samples joined, meaning that doing N joins for N VCFs does O(N**2) work! This is obviously unacceptable and we need to rethink it for 0.2.
  2. Join does an inner join, so most likely you’d end up with no variants after doing this*! See here for a bit of recent discussion on why it’s not trivial to join VCFs with non-overlapping samples and variants: Joining Variant Datasets - Missed variant in inner join - Outer join

*if your VCFs are what I think they are, which is a gVCF with stripped reference-block information.

We should really have a way to import a list of gVCFs. I think Cloudera was thinking about contributing this at one point but don’t know the status there.

3

ok, not the answer I was hoping for, but thanks Tim for the quick and concise reply!

4

This is not a trivial problem to scale. Here is a GATK4 video on Scaling up joint calling with GenomicsDB that discusses the issue and the solution.

So the GATK tools are used to create a single joint-genotyped VCF from multiple gVCFs. The single VCF which is then converted into the VDS with Hail.