# Can I liftover a hail table from GRCH37 to GRCH38 **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [February 12, 2019, 10:24pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861 "2019-02-12T22:24:08Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 12, 2019, 10:24pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/1 "2019-02-12T22:24:08Z")
Dear Hail team: I’m wondering if it is possible to liftover a table from GRCH37 to GRCH38? I realized there is a function to liftover a locus, but can I liftover an entire hail table? Thanks! Best regards, Wei ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [February 13, 2019, 6:25pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/2 "2019-02-13T18:25:37Z")
Konrad is planning to add a bit about this in the how-to, but here’s his example: Lift over the locus coordinates in a Table or MatrixTable from reference genome `'GRCh37'` to `'GRCh38'`: ```python >>> rg37 = hl.get_reference('GRCh37') # doctest: +SKIP >>> rg38 = hl.get_reference('GRCh38') # doctest: +SKIP >>> rg37.add_liftover('gs://hail-common/references/grch37_to_grch38.over.chain.gz', rg38) # doctest: +SKIP >>> ht = ht.annotate(new_locus=hl.liftover(ht.locus, 'GRCh38'), old_locus=ht.locus) # doctest: +SKIP >>> ht = ht.filter(hl.is_defined(ht.new_locus)) # doctest: +SKIP >>> ht = ht.key_by(locus=ht.new_locus) # doctest: +SKIP ``` ---
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 13, 2019, 8:02pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/3 "2019-02-13T20:02:04Z")
Hi Tim, Thank you very much! That solves the problem. Best regards, Wei ---
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 14, 2019, 10:05pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/4 "2019-02-14T22:05:47Z")
Hi Tim, Can I follow up with a related question? I successfully lifted over an hail table using your code. Now I want to annotate a MatrixTable using this Table with the following command: > mt = mt.annotate\_rows(gnomAD = gn[mt.locus, mt.alleles]) But I got the following error message. My suspicion is that there are some variants in MatrixTable that are not included in the Table, which cause an error when calling “gn[mt.locus, mt.alleles]”. Could you please let me know how to fix this issue? > Hail version: 0.2.8-70304a52d33d > Error summary: SparkException: Job aborted due to stage failure: Task 340 in stage 49.0 failed 4 times, most recent failure: Lost task 340.3 in stage 49.0 (TID 174347, ip-10-66-50-98.goldfinch.lan, executor 62): ExecutorLostFailure (executor 62 exited caused by one of the running tasks) Reason: Container marked as failed: container\_1547607392313\_0230\_01\_173099 on host: ip-10-66-50-98.goldfinch.lan. Exit status: 137. Diagnostics: Container killed on request. Exit code is 137 > Container exited with a non-zero exit code 137 > Killed by external signal Thank you very much! Wei ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [February 14, 2019, 10:36pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/5 "2019-02-14T22:36:28Z")
this error (137) is usually a Spark out-of-memory exception. The code you’re running seems perfectly fine. What kind of cluster are you running on? ---
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 15, 2019, 1:55pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/6 "2019-02-15T13:55:50Z")
Thanks for your reply! I’m using a 24-node cluster from AWS, with 192 VCores and 576 Gb memory. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [February 15, 2019, 1:58pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/7 "2019-02-15T13:58:04Z")
what is the full pipeline you are running? ---
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 15, 2019, 4:12pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/8 "2019-02-15T16:12:59Z")
### Load data > mt = hl.read\_matrix\_table(‘all.SNVs.vqsr.mt’) ### gnomAD annots > gn = hl.read\_table(‘gnomad.genomes.r2.1.sites.ht’) > rg37 = hl.get\_reference(‘GRCh37’) > rg38 = hl.get\_reference(‘GRCh38’) > rg37.add\_liftover(‘grch37\_to\_grch38.over.chain.gz’, rg38) > gn = gn.annotate(new\_locus=hl.liftover(gn.locus, ‘GRCh38’), old\_locus=gn.locus) > gn = gn.filter(hl.is\_defined(gn.new\_locus)) > gn = gn.key\_by(locus=gn.new\_locus, alleles=gn.alleles) > mt = mt.annotate\_rows(gnomAD = gn[mt.locus, mt.alleles]) > mt.rows().show() ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 16, 2019, 3:22pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/9 "2019-02-16T15:22:09Z")
How many partitions in the MT and the table? Generally, you can fix memory issues by increasing the number of partitions. I’d try doubling the number of partitions on both. ---
**Author:** ![Wei\_Zhang](https://avatars.discourse-cdn.com/v4/letter/w/3d9bf3/32.png) [@Wei\_Zhang](https://discuss.hail.is/u/Wei_Zhang)\ **Post date:** [February 17, 2019, 3:12pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/10 "2019-02-17T15:12:54Z")
I have 67376 partitions for the MT and 10000 partitions for the table. Thanks for the suggestion. I will try to increase these numbers. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 26, 2019, 10:02am UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/11 "2019-04-26T10:02:36Z")
A post was split to a new topic: [Liftover NullPointerException](https://discuss.hail.is/t/liftover-nullpointerexception/944) ---
**Author:** ![kumarveerapen](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kumarveerapen/32/355_2.png) [@kumarveerapen](https://discuss.hail.is/u/kumarveerapen)\ **Post date:** [May 10, 2022, 9:03pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/12 "2022-05-10T21:03:57Z")
Hi, Hail Team! As you all know, I think Hail is fantastic and I am planning on using it for a dataset that is in hg19 that I’d like to liftover to hg38. One of my questions is: what is the difference between the Hail liftover function vs the one from Picard tools? Any thoughts? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 11, 2022, 3:33pm UTC](https://discuss.hail.is/t/can-i-liftover-a-hail-table-from-grch37-to-grch38/861/13 "2022-05-11T15:33:42Z")
We just call samtools directly to do the liftover. I don’t know what else Picard does, but Hail does not address the alleles, we only deal with the locus itself.