# Slow Performance using filter\_intervals

**URL:** <https://discuss.hail.is/t/slow-performance-using-filter-intervals/784>\
**Category:** Hail Query & hailctl\
**Created:** [November 28, 2018, 5:22am UTC](https://discuss.hail.is/t/slow-performance-using-filter-intervals/784 "2018-11-28T05:22:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cburghard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cburghard/32/519_2.png) [@cburghard](https://discuss.hail.is/u/cburghard)\
**Post date:** [November 28, 2018, 5:22am UTC](https://discuss.hail.is/t/slow-performance-using-filter-intervals/784/1 "2018-11-28T05:22:11Z")

</div>

I’m testing on a small (1.2GB) dataset and I would like to filter variants from a vcf file overlapping many short intervals. When I filter by a single interval, it’s taking \>12 s to count.

This seems to be much slower than I would expect based on the results in this [blog post](http://discuss.hail.is/t/using-filter-intervals-to-query-the-16gb-gnomad-sites-dataset-in-300ms-on-your-laptop/148) (16GB on a laptop in 300ms.) I’ve got 40 cores and 256GB memory, and have seen high CPU and memory use. How is my use different than the example? Why is it taking so long when it says it is only accessing 1 of 38 partitions?

Here’s my code, and the matrix table description. Apologies if I’m missing something basic.

`%time wes_genotypes.count()`  
`CPU times: user 20 ms, sys: 10.6 ms, total: 30.6 ms Wall time: 34.2 s`  
`(622351, 520)`

``  
`intervals = [hl.parse_locus_interval('1:10027520-10027542')]`  
`exon_vars = hl.filter_intervals(wes_genotypes,intervals,keep=True)`  
`%time exon_vars.count()`

`CPU times: user 5.01 ms, sys: 0 ns, total: 5.01 ms Wall time: 12.8 s`  
`(1, 520)`  
``

`wes_genotypes.describe()`  
(omitting large number of INFO row columns)

`----------------------------------------`  
`Global fields:`  
`None`  
`----------------------------------------`  
`Column fields:`  
`'s': str`  
`----------------------------------------`  
`Row fields:`  
`'locus': locus<GRCh37>`  
`'alleles': array<str>`  
`'rsid': str`  
`'qual': float64`  
`'filters': set<str>`  
`'info': struct {`  
`...`  
`----------------------------------------`  
`Entry fields:`  
`'AD': array<int32>`  
`'DP': int32`  
`'GQ': int32`  
`'GT': call`  
`'MIN_DP': int32`  
`'PL': array<int32>`  
`'SB': array<int32>`  
`----------------------------------------`  
`Column key: ['s']`  
`Row key: ['locus', 'alleles']`  
`----------------------------------------`

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 28, 2018, 1:13pm UTC](https://discuss.hail.is/t/slow-performance-using-filter-intervals/784/2 "2018-11-28T13:13:58Z")

</div>

Thanks for flagging this. If there are performance bugs hiding here, I want to get them fixed.

The performance of filter\_intervals is proportional to the number of partitions kept, but also the operations upstream and downstream.

There are two reasons that we could get computations under a second in that blog post:

1. The simplicity of the pipeline. It was:

2. **tiny partitions**. In the edge case that you are querying an interval which overlaps only one variant, then the execution will take as long as it takes for one core to execute all pieces of the pipeline for that one partition.

I have a couple followups -

1. What is the upstream pipeline? Just import\_vcf, or are there qc/filter steps above? These will definitely slow Hail down.

2. could you try importing with more partitions in import\_vcf (`min_partitions=1000?`) and see how that matters?

---

<div class="post-metadata">

**Author:** ![cburghard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cburghard/32/519_2.png) [@cburghard](https://discuss.hail.is/u/cburghard)\
**Post date:** [November 28, 2018, 4:26pm UTC](https://discuss.hail.is/t/slow-performance-using-filter-intervals/784/3 "2018-11-28T16:26:24Z")

</div>

My pipeline has no additional steps, just import, filter, and count.

I tried min\_partitions=1000 and it sped things up. With 1MB partitions it takes ~600ms. Thanks for pointing out the importance of partition size.

I’ll update when I try with larger data.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 28, 2018, 4:27pm UTC](https://discuss.hail.is/t/slow-performance-using-filter-intervals/784/4 "2018-11-28T16:27:57Z")

</div>

The speed here shouldn’t really vary with data size – however, we see problems with more than ~100K partitions in Spark, so if you do have large data, tiny partitions won’t be an option.