# Problem with annotate **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [April 18, 2020, 7:33am UTC](https://discuss.hail.is/t/problem-with-annotate/1376 "2020-04-18T07:33:18Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![mhebrard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mhebrard/32/322_2.png) [@mhebrard](https://discuss.hail.is/u/mhebrard)\ **Post date:** [April 18, 2020, 7:33am UTC](https://discuss.hail.is/t/problem-with-annotate/1376/1 "2020-04-18T07:33:18Z")
I have some issue when tryig to copy data from `mt.variant_qc` to `mt.info` Context: I have a matrix table create from VCF. I filter out some samples, then compute `variant_qc`. I wish to update the fields `AC, AN, AF` from `variant_qc` to `info`. Code: ```auto # Correct AC, AN, AF mt = mt.annotate_rows( info = mt.info.annotate( AC = mt.variant_qc.AC, AN = mt.variant_qc.AN, AF = mt.variant_qc.AF ) ) ``` Problem: When I check the mt “in memory” I see the good information ```auto mt.info.AC.show() # +---------------+------------------------------+--------------+ # | locus | alleles | | # +---------------+------------------------------+--------------+ # | locus | array | array | # +---------------+------------------------------+--------------+ # | chr1:10128 | ["ACCCTAACCCTAACCCTAAC","A"] | [2578,2] | ``` But when I write the `mt` and load it again, the fields are empty ```auto # Write table mt.write('s3://....genotypes.mt', overwrite=True) # load full mt mt_load = hl.read_matrix_table('s3://....genotypes.mt') mt_load.info.AC.show() # +---------------+------------------------------+--------------+ # | locus | alleles | | # +---------------+------------------------------+--------------+ # | locus | array | array | # +---------------+------------------------------+--------------+ # | chr1:10128 | ["ACCCTAACCCTAACCCTAAC","A"] | NA | ``` I have no idea why I lost the data … any help ? Thanks ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 18, 2020, 2:20pm UTC](https://discuss.hail.is/t/problem-with-annotate/1376/2 "2020-04-18T14:20:57Z")
This is quite strange. What version are you using? ---
**Author:** ![johnc1231](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/johnc1231/32/286_2.png) [@johnc1231](https://discuss.hail.is/u/johnc1231)\ **Post date:** [April 18, 2020, 4:18pm UTC](https://discuss.hail.is/t/problem-with-annotate/1376/3 "2020-04-18T16:18:30Z")
Isn’t the problem here just the nested annotate? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 18, 2020, 4:19pm UTC](https://discuss.hail.is/t/problem-with-annotate/1376/4 "2020-04-18T16:19:56Z")
The problem is that after write/read, a row that was defined is now missing, I think. ---
**Author:** ![mhebrard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mhebrard/32/322_2.png) [@mhebrard](https://discuss.hail.is/u/mhebrard)\ **Post date:** [April 19, 2020, 2:11am UTC](https://discuss.hail.is/t/problem-with-annotate/1376/5 "2020-04-19T02:11:59Z")
I use Hail version 0.2.32-a5876a0a2853 ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 22, 2020, 1:43am UTC](https://discuss.hail.is/t/problem-with-annotate/1376/6 "2020-04-22T01:43:00Z")
Can you try updating and seeing if the error persists? If so, we’ll need to get some more information to debug, like the log file. ---
**Author:** ![mhebrard](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mhebrard/32/322_2.png) [@mhebrard](https://discuss.hail.is/u/mhebrard)\ **Post date:** [April 22, 2020, 8:49am UTC](https://discuss.hail.is/t/problem-with-annotate/1376/7 "2020-04-22T08:49:17Z")
For now I found some workaround. Some details on the context: I had 2 matrix table obtained from different VCF. Then I use generate the union matrix table `mt_union = mt_left.union(mt_right)`. The resulting mt contain the info field of `mt_left`. the rows that are only present in `mt_right` have an info field empty. That is (not ideal but) expected behaviour of `union()`. Then I try to updates fields nested into info using `annotate(info = mt.info.annotate(...` And at that point my guess is that when `mt.info` is empty then the annotate function ignore the line. It is not explicitely stated for `annotate()` function but I remember seen mention of “NA rows are ignored” in some place in the doc To wor around that I make a first pass when I force data into info for each rows `mt.annotate_rows( info = hl.struct(mode = 'union') )` Then I make a second pass to update multiple fields of info. That way, there is no more empty info and all rows get updated. That solve my imediate problem, but do not explain why in the previous post I am able to recover the data when I work “in memory” but not after a write/load of the mt. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 22, 2020, 11:36am UTC](https://discuss.hail.is/t/problem-with-annotate/1376/8 "2020-04-22T11:36:10Z")
> And at that point my guess is that when `mt.info` is empty then the annotate function ignore the line. It is not explicitely stated for `annotate()` function but I remember seen mention of “NA rows are ignored” in some place in the doc the `mt.info.annotate` is using [`StructExpression.annotate`](https://hail.is/docs/0.2/hail.expr.StructExpression.html#hail.expr.StructExpression.annotate), which will return a missing value if the input is missing. In order to force a defined value, you can use `hl.struct` as follows: ```python mt = mt.annotate_rows(info = hl.struct( **mt.info,** annotation_fields)) ``` > [@mhebrard](#): > > That solve my imediate problem, but do not explain why in the previous post I am able to recover the data when I work “in memory” but not after a write/load of the mt. This is the thing I’m concerned about. It has the smell of an optimizer error (where Hail’s optimizing compiler makes an invalid transformation of the pipeline to execute).