# is.hail.kryo.HailKryoRegistrator ClassNotFoundException **URL:** \ **Category:** Help \[0.1\]\ **Created:** [April 26, 2018, 3:07pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484 "2018-04-26T15:07:45Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![atebbe](https://avatars.discourse-cdn.com/v4/letter/a/e5b9ba/32.png) [@atebbe](https://discuss.hail.is/u/atebbe)\ **Post date:** [April 26, 2018, 3:07pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/1 "2018-04-26T15:07:45Z")
We are getting an exception when calling the hl.import\_vcf() function. The hl.init() function completes without issue, which makes me think the hail jar is loading properly. The import\_vcf function is the first one we are running that creates a job on the cluster. We are running this on an AWS EMR cluster with hail (master branch), and spark 2.2.1. Exception from task attempt logs: org.apache.spark.SparkException: Failed to register classes with Kryo Caused by: java.lang.ClassNotFoundException: is.hail.kryo.HailKryoRegistrator Relevant settings in spark-defaults.conf spark.driver.extraClassPath …:./hail-all-spark.jar spark.executor.extraClassPath …:./hail-all-spark.jar spark.kryo.registrator is.hail.kryo.HailKryoRegistrator spark.serializer org.apache.spark.serializer.KryoSerializer ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 26, 2018, 4:14pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/2 "2018-04-26T16:14:57Z")
I suspect that this problem is related to the hail jar not being properly visible on Spark workers. `hl.init()` will fail if the driver isn’t configured correctly, but if the workers are misconfigured, I’d expect a `java.lang.ClassNotFoundException` in the first place that loads Hail classes on worker machines. I think maybe you need to pass --jars or the appropriate config in the spark-defaults to get the jar to ship correctly to the workers. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [April 27, 2018, 2:18pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/3 "2018-04-27T14:18:18Z")
Hi @atebbe, I’m sorry you’re running into this issue! Since `hl.init()` executed successfully, I suspect the Hail jar is located in the correct location on the driver node. However, the `import_vcf` function must actually communicate with the executors (worker nodes). Based on the error message, I suspect `hail-all-spark.jar` is not located in the current working directory of the Spark processes on your executors. If you are using `spark-shell`, are you also passing the `--jars` parameter? If you’re not using `spark-shell` what command are you using to start interacting with the cluster? ---
**Author:** ![atebbe](https://avatars.discourse-cdn.com/v4/letter/a/e5b9ba/32.png) [@atebbe](https://discuss.hail.is/u/atebbe)\ **Post date:** [April 27, 2018, 5:13pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/4 "2018-04-27T17:13:01Z")
We are interacting with hail using the Apache Toree - Pyspark kernel for Jupyter. The jar is on the namenode of the cluster in /home/hadoop. The following is the first cell of our notebooks: sc.addFile(’/home/hadoop/hail-all-spark.jar’) sc.addPyFile(’/home/hadoop/hail-python.zip’) import hail as hl hl.init(sc) My assumption was sc.addFile was adding the jar to hdfs. This worked fine with 0.1 - this is our first attempt with 0.2. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 27, 2018, 5:36pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/5 "2018-04-27T17:36:15Z")
Does `sc.addJar` have different semantics from `sc.addFile`? Maybe try that? ---
**Author:** ![atebbe](https://avatars.discourse-cdn.com/v4/letter/a/e5b9ba/32.png) [@atebbe](https://discuss.hail.is/u/atebbe)\ **Post date:** [April 30, 2018, 12:25pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/6 "2018-04-30T12:25:35Z")
I get an error that addJar does not exist sc.addJar(’/home/hadoop/hail-all-spark.jar’) sc.addPyFile(’/home/hadoop/hail-python.zip’) … Name: org.apache.toree.interpreter.broker.BrokerException Message: Traceback (most recent call last): File “/tmp/kernel-PySpark-583e9b04-451e-4331-9e04-300634d28644/pyspark\_runner.py”, line 194, in eval(compiled\_code) File “”, line 1, in AttributeError: ‘SparkContext’ object has no attribute ‘addJar’ ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [April 30, 2018, 2:15pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/7 "2018-04-30T14:15:37Z")
Ah, okay. addJar must have been something from Spark 1.X that was removed in 2.0. This syntax looks right… ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [May 4, 2018, 2:41pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/8 "2018-05-04T14:41:36Z")
Sorry for the long delay on my reply, @atebbe Let’s recall your spark class path settings: ```auto spark.driver.extraClassPath …:./hail-all-spark.jar spark.executor.extraClassPath …:./hail-all-spark.jar ``` These assert that, on both the driver and the executors, the jar is located in the working directory of the driver process. If you ssh to one of your executors and find the spark job working directory (try looking in `/var/run/spark/work`), I suspect you will not find `hail-all-spark.jar` in that directory. While you’re at it, can you open a terminal in your Jupyter notebook and verify that the `hail-all-spark.jar` is indeed in the working directory of your executor? [This StackOverflow post](https://stackoverflow.com/questions/37132559/add-jars-to-a-spark-job-spark-submit) suggests that `addFile` is inappropriate for “runtime dependencies”. So. Assuming the jar is indeed missing from the working directory of your executors, we need to figure out how to get it there. First, try `sc._jsc.addJar` instead of `sc.addFile`. If that fails, `Apache Toree` [suggests using](https://toree.incubator.apache.org/docs/current/user/faq/) the `%AddJar` magics invocation to add a jar. ---
**Author:** ![atebbe](https://avatars.discourse-cdn.com/v4/letter/a/e5b9ba/32.png) [@atebbe](https://discuss.hail.is/u/atebbe)\ **Post date:** [May 4, 2018, 7:09pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/9 "2018-05-04T19:09:53Z")
Thanks for following up on this. sc.\_jsc.addJar did the trick! My worker nodes don’t have /var/run/spark. I searched for the jar on the entire filesystem of the worker node and did not find it. Is it recommended to use \_jsc? Thanks, Adam ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 4, 2018, 7:10pm UTC](https://discuss.hail.is/t/is-hail-kryo-hailkryoregistrator-classnotfoundexception/484/10 "2018-05-04T19:10:38Z")
I don’t know why they don’t expose it in Python! Clearly it’s necessary…