# VDS annotation triggers shuffle

**URL:** <https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450>\
**Category:** Help \[0.1\]\
**Created:** [April 4, 2018, 12:29pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450 "2018-04-04T12:29:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cristinaluengoagullo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cristinaluengoagullo/32/254_2.png) [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\
**Post date:** [April 4, 2018, 12:29pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/1 "2018-04-04T12:29:48Z")

</div>

Hi,

I’m importing and annotating a vcf with just one chromosome, and I get the following message when annotating (with annotate\_variants\_table):

Hail: INFO: Ordering unsorted dataset with network shuffle

The vcf file is ordered by position, and so is the annotation table. Does this mean that Hail performs an ordering somewhere when annotating?  
Thanks,

Cristina.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [April 4, 2018, 12:31pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/2 "2018-04-04T12:31:21Z")

</div>

Hail does ordered joins almost everywhere, which means that both sides need to be ordered. This message is probably referring to the shuffle that orders the annotation table.

---

<div class="post-metadata">

**Author:** ![cristinaluengoagullo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cristinaluengoagullo/32/254_2.png) [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\
**Post date:** [April 4, 2018, 12:33pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/3 "2018-04-04T12:33:39Z")

</div>

Thank you so much for the very fast reply! But then, does this mean that even if the annotation table is ordered beforehand, a shuffle will be triggered anyway? Is there a way to avoid having this shuffles on already sorted data?  
Thanks again!

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [April 4, 2018, 12:35pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/4 "2018-04-04T12:35:06Z")

</div>

no, if it’s ordered then there shouldn’t be a shuffle in annotate\_variants\_table.

I do recall that filter\_variants\_table may always do a shuffle, though.

---

<div class="post-metadata">

**Author:** ![cristinaluengoagullo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cristinaluengoagullo/32/254_2.png) [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\
**Post date:** [April 4, 2018, 12:44pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/5 "2018-04-04T12:44:04Z")

</div>

Ok thanks! Last question though (maybe silly, but just in case), the right ordering is by chromosome and then position, right? Or is there any other factor that must be considered?

I’m asking also because depending on the vcf that I import, I get either this message:

INFO: Coerced sorted dataset

Or this one:

INFO: Coerced almost-sorted dataset

Even if I have previously sorted both vcf with bedtools sort.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [April 4, 2018, 12:56pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/6 "2018-04-04T12:56:04Z")

</div>

Ordering is actually by the full variant (`TVariant`) in 0.1. This means if you have a multiallelic variant split over two lines, it could be out of order. In this case, it won’t trigger a full shuffle but a local sort (`Coerced almost-sorted dataset`)

---

<div class="post-metadata">

**Author:** ![cristinaluengoagullo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/cristinaluengoagullo/32/254_2.png) [@cristinaluengoagullo](https://discuss.hail.is/u/cristinaluengoagullo)\
**Post date:** [April 4, 2018, 1:04pm UTC](https://discuss.hail.is/t/vds-annotation-triggers-shuffle/450/7 "2018-04-04T13:04:04Z")

</div>

Ok thank you very much!
