# How to speed up export\_vcf? **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [September 23, 2021, 3:55pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264 "2021-09-23T15:55:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 23, 2021, 3:55pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/1 "2021-09-23T15:55:02Z")
I have a workflow that essentially annotates a large vcf with gnomAD popmax values, filters by a threshold and exports back to vcf. The process so far has been prohibitively slow (about 15 minutes for just chr21, and about 1.5 hours for chr1). For my latest run, I’m attempting to do all the chromosomes at once. I ran the script late last night, and it’s still running now, roughly 10 hours later. Can anyone suggest ways to speed up my pipeline? (I’ll include the code in a separate post to avoid the spam filter). -Thanks in advance ---
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 23, 2021, 3:57pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/2 "2021-09-23T15:57:42Z")
```auto import hail as hl import os rg = hl.get_reference('GRCh38') vcf="/path/to/3225171004313.GTAGAGCA.hard-filtered.vcf.gz" check=f"/path/to/WGS/{os.path.basename(vcf)[:-2]}mt" mt=hl.read_matrix_table(check) ht=hl.read_table("/path/to/GNOMAD/gnomad.genomes.v3.1.1.sites.ht") ht=ht.filter(~hl.is_missing(ht.popmax.AF)) #remove all popmax=NA n=mt.annotate_rows(popmax=ht[mt.locus, mt.alleles].popmax.AF) #add popmax values n=n.filter_rows(n.popmax < 0.01, keep=True) #filter n=n.repartition(25) exp_vcf=f"/path/to/WGS/{os.path.basename(vcf).replace('hard', 'gnomad')}" hl.export_vcf(n, exp_vcf, parallel='separate_header') ``` ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 23, 2021, 4:01pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/3 "2021-09-23T16:01:20Z")
I’d recommend removing the repartition. That’s a very expensive operation. ---
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 23, 2021, 9:30pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/4 "2021-09-23T21:30:45Z")
I removed it, but I didn’t notice any improvement in runtime (however, definitely good to know that the partitioning isn’t necessary). My latest idea is to reduce the size of the gnomad hail table by selecting the popmax column only. Then I’ll use that to annotate and filter my vcfs. Question: In my script, after I do ```auto ht=ht.select("popmax") ``` should I checkpoint the resulting ht object? Would that improve my performance at all- versus just using the ht directly to annotate? (downstream would look something like this:) ```auto n=mt.annotate_rows(popmax=ht[mt.locus, mt.alleles].popmax.AF) n=n.filter_rows(n.popmax < 0.01, keep=True) hl.export_vcf(n, exp_vcf) ``` ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 24, 2021, 12:39pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/5 "2021-09-24T12:39:50Z")
I think we need a bit more information about these inputs and your system. 1. What are you running this pipeline on? A single server, or a cluster? 2. How big is the VCF? How many data partitions does it have in matrixtable form (`mt.n_partitions()`)? 3. How big is the gnomAD table in bytes and how many partitions? There’s a known performance issue with executing joins where the partitioning of the two sides differs greatly. ---
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 24, 2021, 4:48pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/6 "2021-09-24T16:48:23Z")
Got my answer. I essentially reduced the size of the gnomAD table down to only the necessary information. Annotation and filtering was done in about 15 minutes (previously took 1.5 hours). For any future users that look this up: eliminate all unnecessary columns from gnomAD table (ht.select(), ht.checkpoint()), you’ll see a big improvement in runtime. Thanks for the brainstorming @tpoterba you guys have built a first class package here 👌 ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 24, 2021, 5:11pm UTC](https://discuss.hail.is/t/how-to-speed-up-export-vcf/2264/7 "2021-09-24T17:11:18Z")
> [@mrdata](#): > > For any future users that look this up: eliminate all unnecessary columns from gnomAD table (ht.select(), ht.checkpoint()), you’ll see a big improvement in runtime. Our compiler should be doing most of this automatically. I think what you’ve done is essentially work around the performance issues with the join that I’ve described, though – glad it works!