Hi Hail team, I previously have been able to subset very small vcfs around 55 samples, but not have had to do any subsetting at a large scale. I am currently trying to use hail to subset large ~40k sample vcfs (by chromosome) into 48 subsetted vcfs. This would end up being 22 (chromosomes) * 48 (s…

When you say it “stopped” without completion, was there some kind of error thrown? Do you have the hail log file? Also, are you a Broad user / what cloud provider are you using? I suspect the hail experience on an on prem cluster isn’t going to great.

Hi @johnc1231 , I have attached the log file. I did not see any errors throw in there. Yes, I am in DSP at Broad and use google cloud. Me and another team member previously came to hail office hours to discuss subsetting vcfs in the cloud using hail and we were told it would be easier to do this on…

Hey Sam, you probably need to allocate more memory to the Java process . How many variants are in the VCFs? I do not remember how we arrived at the on-prem solution. It’s definitely a little complicated to securely get your GCP Service Account keys into a docker container. A VM would not have this i…

Hi @danking , I used to have it on my calendar but can’t seem to find it… When is your next hail office hours?

It’s at 1pm in Serengetti on Thursday February 27th

@danking – any idea on how to raise memory for the on-prem backend of Hail?

Subset large vcf into multiple vcfs

Hail Query & hailctl

danking February 27, 2020, 3:07pm 8

Do you mean Hail in local mode? You’ll need to use PYSPARK_SUBMIT_ARGS.

Topic		Replies	Views
Cluster Size for Subsetting in Hail Hail Query & hailctl	3	413	March 10, 2020
Filtering samples from VDS in Google cloud Hail Batch & General Cloud	11	606	August 6, 2024
Trying to annotate vcf subset and then filter according to properties Hail Query & hailctl	9	124	March 21, 2025
Performing SampleQC using Hail on ~500k WES samples Hail Query & hailctl	7	666	July 12, 2023
Working with large VCFs (e.g. from UK Biobank) is slow Hail Query & hailctl	12	2071	August 23, 2024

Subset large vcf into multiple vcfs

Related topics