# Variant annotation in MatrixTable **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [June 26, 2020, 12:25am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493 "2020-06-26T00:25:04Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [June 26, 2020, 12:25am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/1 "2020-06-26T00:25:05Z")
Hi, I have a table ( `ht` ), see `ht.describe()` ```auto ---------------------------------------- Global fields: None ---------------------------------------- Row fields: 'chr': str 'start': int32 'end': int32 'range': str 'window_id': str 'gene_name': str 'n_variants': int32 'gene_id': str 'interval': interval> ---------------------------------------- Key: ['interval'] ``` I want to annotate my variants (in my `mt` ) and create a _structure_ rows filed which will have all the information. One variant could overlap with multiple intervals, and hence I probably need to create `array **Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 29, 2020, 11:45am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/2 "2020-06-29T11:45:30Z") The familiar syntax below is _almost_ what you want: ```python mt = mt.annotate_rows(interval_data = ht[mt.row_key]) ``` However, this will give you one distinct struct per row of the matrix table. Instead, you can call through to the [`Table.index`](https://hail.is/docs/0.2/hail.Table.html#hail.Table.index) method, and use the `all_matches` flag: ```python mt = mt.annotate_rows(interval_data = ht.index(mt.row_key, all_matches=True)) ``` ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [June 29, 2020, 1:10pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/3 "2020-06-29T13:10:55Z")
Thank you very much @tpoterba. It was exactly what I want. I am having another issue now - my `ht` looks like - ![Screenshot 2020-06-29 at 14.00.20](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/51209f48e0f33584cf8048fe7b8799106668f205.jpeg) I have a variant `chr1:948594` in my `mt`, which was surprisingly not annotated with the `ht` information, althouth the position is present in the `ht`, - see the last line - the end position is the actual variant position. Here is my code - ```auto mt = mt.annotate_rows(windows = ht.index(mt.row_key[0], all_matches=True)) mt = mt.rows() mt.select(mt.windows).show() ``` Which gives me the output like - ![Screenshot 2020-06-29 at 14.10.16](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/573432b787409f524c1cf1139489d758cf261fea.jpeg) As you can see the last variant was not correctly annotated. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 29, 2020, 1:27pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/4 "2020-06-29T13:27:28Z")
The printed representation of these intervals indicates using [interval notation](https://www.varsitytutors.com/hotmath/hotmath_help/topics/interval-notation) that they are inclusive of the left endpoint, but exclusive of the right endpoint. You’ll need to re-create these intervals to include the right endpoint, if you want `chr1:946174-chr1:948594` to include `chr1:948594`. ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [June 29, 2020, 1:57pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/5 "2020-06-29T13:57:33Z")
Ahhh Okie. Thanks Tim. ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 2, 2020, 11:30pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/6 "2020-07-02T23:30:28Z")
Hi, I have a question regarding [`Table.index`](https://hail.is/docs/0.2/hail.Table.html#hail.Table.index) I have a Hail table (`ht`) `ht.describe()` ```auto ---------------------------------------- Global fields: None ---------------------------------------- Row fields: 'CHR': int32 'START': int32 'END': int32 'Window_boundary': str 'Window_Name': str 'Gene_ID': str 'No_of_Variants': int32 'Range': str 'interval': interval> ---------------------------------------- Key: ['interval', 'Gene_ID'] ---------------------------------------- ``` Now I need to annotate my `mt` The key of my `mt` is - ```auto ---------------------------------------- Column key: ['s'] Row key: ['locus', 'gene_set'] ---------------------------------------- ``` As @tpoterba mentioned earlier to this thread, I am trying this - ```auto mt = mt.annotate_rows(interval_data = ht.index(mt.row_key, all_matches=True)) ``` But it gives me an Error mentioned below - **Error:** ```auto ExpressionException: Key type mismatch: cannot index table with given expressions: Table key: interval>, str Index Expressions: locus, str ``` However, ```auto mt = mt.annotate_rows(interval_data = ht.index(mt.row_key[0], all_matches=True)) ``` works. But I need to match both condition (i) the variants (`mt.locus`) should be present in the **interval** (`ht.interval`) and (ii) for matching genes, the `mt.gene_set` should match with `ht.Gene_ID`. Note that my `mt` already have some duplicated `mt.locus` with different `mt.gene_set` but the key is unique. Could you please help me. Thanks ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 3, 2020, 12:01am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/7 "2020-07-03T00:01:09Z")
I think the solution here is to take `gene_set` out of the table key. ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 3, 2020, 12:06am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/8 "2020-07-03T00:06:20Z")
But then how do I match both the conditions - locus and gene\_set ? ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 3, 2020, 12:07am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/9 "2020-07-03T00:07:29Z")
And also `mt.locus` is repeated. Same locus for multiple genes ! ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 3, 2020, 11:49am UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/10 "2020-07-03T11:49:12Z")
I see. Interval joins are a bit special, and only are possible when joining interval to , and don’t work when you have additional key fields after the interval. I think what I’d do is (1) key the interval table by only the interval, and (b) filter the result after: ```python mt = mt.annotate_rows(interval_matches = interval_ht.index(mt.locus, all_matches=True).filter( lambda x: x.gene_set == mt.gene_set)) ``` ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 3, 2020, 12:43pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/11 "2020-07-03T12:43:22Z")
I am not sure, this will give me what I want - Let’s take an example - `chr1:1000` is a LoF variant for **GeneA** and **GeneB** , and `chr1:1100` is a LoF variant for **GeneC** The interval file says - ```auto chr1:100-2000 GeneA rank1 chr1:100-2000 GeneA rank2 chr1:100-2000 GeneB rank1 chr1:100-2000 GeneC rank1 ``` I want to annotate my MatrixTable in a way - ```auto chr1:1000 [GeneA, GeneB] ## this we get from all_matches=True chr1:1100 [GeneC] ``` But what you proposed, wouldn’t I get ```auto chr1:1000 [GeneA, GeneB, GeneC] chr1:1100 [GeneA, GeneB, GeneC] ``` Eventually, I want to explode the Gene, and then `chr1:1000` will be annotated with GeneC, and `chr1:1100 ` will be annotated with GeneA and GeneB which are wrong. Sorry I am being very complicated ☹ ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 3, 2020, 1:08pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/12 "2020-07-03T13:08:51Z")
Ahh wait - So what you proposed @tpoterba - it actually filter Genes right So with the above mentioned example - it gives - ```auto chr1:1000 [GeneA, GeneB] ## this we get from all_matches=True chr1:1100 [GeneC] ``` Right ?? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 3, 2020, 3:26pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/13 "2020-07-03T15:26:50Z")
yep, exactly. ---
**Author:** ![kousikbioinfo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kousikbioinfo/32/302_2.png) [@kousikbioinfo](https://discuss.hail.is/u/kousikbioinfo)\ **Post date:** [July 3, 2020, 8:59pm UTC](https://discuss.hail.is/t/variant-annotation-in-matrixtable/1493/14 "2020-07-03T20:59:19Z")
Thank you very much @tpoterba for the confirmation