# Annotate variants with genes **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [April 27, 2023, 3:38pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360 "2023-04-27T15:38:44Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [April 27, 2023, 3:38pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/1 "2023-04-27T15:38:44Z")
Hello team, I need to annotate each variants in a matrixtable with gene symbol and intervals for GRCh38. Is there a hail function to map genes with variants based on variant locus? Thanks very much for your help! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [April 27, 2023, 4:03pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/2 "2023-04-27T16:03:35Z")
There is no such function. Your options: 1. Download a GRCh38 locus to gene mapping file, import as a table, and use that to annotate your matrix table. 2. Run `hl.vep` using a Dataproc cluster. 3. Use the Hail Annotation Database to annotate `gencode` which has gene names. The Annotation Database is only available in Amazon EMR and Google Dataproc. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [May 10, 2023, 9:27pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/3 "2023-05-10T21:27:25Z")
@danking Thank you for the solution. I have used the option 3 you suggested to annotate the variants with genes using the gencode database. Some of the variants in my matrixtable are not mapped to any genes, since the locus of these variants are outside of any gene intervals. We would like to find the closest gene for each of these variants that are not mapped. Is there a function in hail for finding the closest gene based on the variant locus? Thanks very much for your help. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 12, 2023, 2:42am UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/4 "2023-05-12T02:42:06Z")
There is no such function in Hail. If I was going to build that, I would take the ~30,000 genes and (just in Python or w/e) evenly expand their locus intervals until the entire genome is covered by at least one gene. Then I’d import that as a Hail table and use that to annotate my data. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [May 12, 2023, 5:50pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/5 "2023-05-12T17:50:15Z")
Thanks Dan for the suggestion. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [May 22, 2023, 6:47pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/6 "2023-05-22T18:47:33Z")
@danking I am using the hail function below to load the gencode database on the DNAnexus platform. gencode = hl.experimental.load\_dataset(name=‘gencode’, version=‘v31’, reference\_genome=‘GRCh38’, region=‘us’, cloud=‘aws’) This function was running fine before last week. However, when I ran this function last Wednesday, I got the error below. It seems that the error is related to some AWS authentication issue in the S3 bucket in the US region (FatalError: ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider). I didn’t have this error before, and this happened after I installed the AWS Client Workspaces on my laptop. I tried to uninstall the AWS Client Workspaces and re-ran the above hail function again, but I still got the error. Your help in resolving this issue will be greatly appreciated. * * * FatalError Traceback (most recent call last) in 3 reference\_genome=‘GRCh38’, 4 region=‘us’, ----\> 5 cloud=‘aws’) /opt/conda/lib/python3.6/site-packages/hail/experimental/datasets.py in load\_dataset(name, version, reference\_genome, region, cloud) 119 dataset = \_read\_dataset(path) 120 except hl.utils.java.FatalError: → 121 dataset = \_read\_dataset(path.replace(‘s3://’, ‘s3a://’)) 122 else: 123 dataset = \_read\_dataset(path) /opt/conda/lib/python3.6/site-packages/hail/experimental/datasets.py in \_read\_dataset(path) 11 return hl.read\_table(path) 12 elif path.endswith(‘.mt’): —\> 13 return hl.read\_matrix\_table(path) 14 elif path.endswith(‘.bm’): 15 return hl.linalg.BlockMatrix.read(path) in read\_matrix\_table(path, \_intervals, \_filter\_intervals, \_drop\_cols, \_drop\_rows, \_n\_partitions) /opt/conda/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) 575 def wrapper(_original\_func, \*args, \*\*kwargs): 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) → 577 return _original\_func(\*args, \*\*kwargs_) 578 579 return wrapper /opt/conda/lib/python3.6/site-packages/hail/methods/impex.py in read\_matrix\_table(path, \_intervals, \_filter\_intervals, \_drop\_cols, \_drop\_rows, \_n\_partitions) 2113 :class:`.MatrixTable` 2114 “”" → 2115 for rg\_config in Env.backend().load\_references\_from\_dataset(path): 2116 hl.ReferenceGenome.\_from\_config(rg\_config) 2117 /opt/conda/lib/python3.6/site-packages/hail/backend/spark\_backend.py in load\_references\_from\_dataset(self, path) 326 327 def load\_references\_from\_dataset(self, path): → 328 return json.loads(Env.hail().variant.ReferenceGenome.fromHailDataset(self.fs.\_jfs, path)) 329 330 def from\_fasta\_file(self, name, fasta\_file, index\_file, x\_contigs, y\_contigs, mt\_contigs, par): /cluster/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py in **call** (self, \*args) 1255 answer = self.gateway\_client.send\_command(command) 1256 return\_value = get\_return\_value( → 1257 answer, self.gateway\_client, self.target\_id, self.name) 1258 1259 for temp\_arg in temp\_args: /opt/conda/lib/python3.6/site-packages/hail/backend/py4j\_backend.py in deco(\*args, \*\*kwargs) 29 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’ 30 ‘Hail version: %s\n’ —\> 31 ‘Error summary: %s’ % (deepest, full, hail. **version** , deepest), error\_id) from None 32 except pyspark.sql.utils.CapturedException as e: 33 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’ FatalError: ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider Java stack trace: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at is.hail.io.fs.HadoopFS.fileStatus(HadoopFS.scala:164) at is.hail.io.fs.FS$class.isDir(FS.scala:175) at is.hail.io.fs.HadoopFS.isDir(HadoopFS.scala:70) at is.hail.expr.ir.RelationalSpec$.readMetadata(AbstractMatrixTableSpec.scala:30) at is.hail.expr.ir.RelationalSpec$.readReferences(AbstractMatrixTableSpec.scala:73) at is.hail.variant.ReferenceGenome$.fromHailDataset(ReferenceGenome.scala:581) at is.hail.variant.ReferenceGenome.fromHailDataset(ReferenceGenome.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at is.hail.io.fs.HadoopFS.fileStatus(HadoopFS.scala:164) at is.hail.io.fs.FS$class.isDir(FS.scala:175) at is.hail.io.fs.HadoopFS.isDir(HadoopFS.scala:70) at is.hail.expr.ir.RelationalSpec$.readMetadata(AbstractMatrixTableSpec.scala:30) at is.hail.expr.ir.RelationalSpec$.readReferences(AbstractMatrixTableSpec.scala:73) at is.hail.variant.ReferenceGenome$.fromHailDataset(ReferenceGenome.scala:581) at is.hail.variant.ReferenceGenome.fromHailDataset(ReferenceGenome.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) Hail version: 0.2.78-b17627756568 Error summary: ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 22, 2023, 6:50pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/7 "2023-05-22T18:50:49Z")
If you’re running Hail through DNANexus, this might be a bug in how they’re configuring Hail / Spark (especially if it was working last week). Looks like the culprit is a missing JAR for the AWS auth functionality. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [May 22, 2023, 7:04pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/8 "2023-05-22T19:04:01Z")
Thanks @tpoterba for the quick reply. I am using the code below before the above function call. from pyspark.sql import SparkSession import hail as hl builder = ( SparkSession .builder .enableHiveSupport() ) spark = builder.getOrCreate() hl.init(sc=spark.sparkContext) I got the message below: pip-installed Hail requires additional configuration options in Spark referring to the path to the Hail Python module directory HAIL\_DIR, e.g. /path/to/python/site-packages/hail: spark.jars=HAIL\_DIR/hail-all-spark.jar spark.driver.extraClassPath=HAIL\_DIR/hail-all-spark.jar spark.executor.extraClassPath=./hail-all-spark.jarRunning on Apache Spark version 2.4.4 SparkUI available at [http://ip-10-60-152-12.eu-west-2.compute.internal:8081](http://ip-10-60-152-12.eu-west-2.compute.internal:8081/) Welcome to \_\_ \_\_ \<\>\_\_ / /_/ /\_\_ \_\_/ / / \_\_ / \_ `/ / / /_/ /_/\_,_/_/_/ version 0.2.78-b17627756568 LOGGING: writing to /opt/notebooks/hail-20230518-1658-0.2.78-b17627756568.log Is there anything I can do in my end to resolve the error? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 22, 2023, 7:09pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/9 "2023-05-22T19:09:12Z")
No, you need to contact DNANexus and ask them to fix this issue. They must’ve changed the Hail installation. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [May 22, 2023, 7:12pm UTC](https://discuss.hail.is/t/annotate-variants-with-genes/3360/10 "2023-05-22T19:12:14Z")
@danking Thanks for your quick reply. I will contact the DNANexus to fix this issue.