# Most efficient way to filter and densify VDS

**URL:** <https://discuss.hail.is/t/most-efficient-way-to-filter-and-densify-vds/3810>\
**Category:** Hail Query & hailctl\
**Created:** [April 24, 2024, 5:29pm UTC](https://discuss.hail.is/t/most-efficient-way-to-filter-and-densify-vds/3810 "2024-04-24T17:29:54Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![patrick-schultz](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/patrick-schultz/32/265_2.png) [@patrick-schultz](https://discuss.hail.is/u/patrick-schultz)\
**Post date:** [May 6, 2024, 8:51pm UTC](https://discuss.hail.is/t/most-efficient-way-to-filter-and-densify-vds/3810/2 "2024-05-06T20:51:30Z")

</div>

Hi @jbs!

> I was advised that `filter_rows()` is more efficient than `filter_intervals()` as a way of reducing cost.

That’s not right. `filter_intervals` will avoid reading any partitions that don’t overlap any of the intervals, while `filter_rows` has to scan through every row and apply the filter condition (though our optimizer tries very hard to rewrite `filter_rows` to `filter_intervals` whenever possible). Also, that snippet only filters the variant data, not the reference data, which will likely cause downstream performance issues.

> I saw [here](https://discuss.hail.is/t/filtering-samples-from-vds-in-google-cloud/3718) that Hail doesn’t support filtering a subset of the full 245k AoU sample set very well in the VDS. In terms of the optimal order of operations here, do you recommend filtering by samples first, writing a smaller VDS, then filtering by intervals before densifying?

You want to filter the variants first. The reason filtering samples is expensive is because a VDS (like a matrixtable) is stored “variant major”, i.e. for each variant it stores an array of all the entries for all the samples. Filtering the samples means filtering every one of those entries arrays, whereas filtering variants just requires skipping over the filtered ones. By filtering variants first, only the entries arrays for the kept variants need to be rewritten.

So in short, I think you were doing everything right before. One thought: can you run `vds.reference_data.describe()` and check if there’s a global field `ref_block_max_length`? That’s important for making `filter_intervals` on the reference data efficient.

@chrisvittal Can you think of anything else to suggest? Is the cost for the first run in the ballpark of what you would expect?

Actually, you probably want to write out the vds after filtering before densifying. That may actually be the cause of the high cost. Does that sound right, Chris?

---

_[View the full topic](https://discuss.hail.is/t/most-efficient-way-to-filter-and-densify-vds/3810)._