# HailContext(sc) error **URL:** \ **Category:** Help \[0.1\]\ **Created:** [May 14, 2017, 3:07am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212 "2017-05-14T03:07:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![eilalan](https://avatars.discourse-cdn.com/v4/letter/e/ecc23a/32.png) [@eilalan](https://discuss.hail.is/u/eilalan)\ **Post date:** [May 14, 2017, 3:07am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/1 "2017-05-14T03:07:26Z")
## [6]: hc = hail.HailContext() Py4JJavaError Traceback (most recent call last) in () ----\> 1 hc = hail.HailContext() /home/eila/hail/python/hail/context.pyc in **init** (self, sc, appName, master, local, log, quiet, append, parquet\_compression, min\_block\_size, branching\_factor, tmp\_dir) 69 self.\_jhc = scala\_object(self.\_hail, ‘HailContext’).apply( 70 jsc, appName, joption(master), local, log, quiet, append, —\> 71 parquet\_compression, min\_block\_size, branching\_factor, tmp\_dir) 72 73 self.\_jsc = self.\_jhc.sc() /usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/java\_gateway.py in **call** (self, \*args) 1131 answer = self.gateway\_client.send\_command(command) 1132 return\_value = get\_return\_value( -\> 1133 answer, self.gateway\_client, self.target\_id, [self.name](http://self.name)) 1134 1135 for temp\_arg in temp\_args: /usr/lib/spark/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py in get\_return\_value(answer, gateway\_client, target\_id, name) 317 raise Py4JJavaError( 318 “An error occurred while calling {0}{1}{2}.\n”. –\> 319 format(target\_id, “.”, name), value) 320 else: 321 raise Py4JError( Py4JJavaError: An error occurred while calling o3.apply. : org.apache.spark.SparkException: Found both spark.executor.extraClassPath and SPARK\_CLASSPATH. Use only the former. at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:543) at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:541) at scala.collection.immutable.List.foreach(List.scala:381) at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:541) at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:529) at scala.Option.foreach(Option.scala:257) at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:529) at org.apache.spark.SparkContext.(SparkContext.scala:365) at is.hail.HailContext$.configureAndCreateSparkContext(HailContext.scala:86) at is.hail.HailContext$.apply(HailContext.scala:161) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) In [7]: ---
**Author:** ![eilalan](https://avatars.discourse-cdn.com/v4/letter/e/ecc23a/32.png) [@eilalan](https://discuss.hail.is/u/eilalan)\ **Post date:** [May 14, 2017, 3:08am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/2 "2017-05-14T03:08:39Z")
Hi, sorry that I didn’t continue the older thread -I received a server error in numerous times. Anyway, I tried to regenerate env. with a larger machine resources. everything went smooth + I was able to install the anaconda env. this time. the only issue that i encounter was with ipython installation. it was looking for python.h and to resolve that, I install the python-dev version. when i run the test code on ssh to the master machine: > > > import hail =\> no issues > > > hc = hail.HailContext(sc) =\> fires error from Py4JJavaError (see in the above post) Could you please let me know what might it be. Thanks, eilalan ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 14, 2017, 3:09am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/3 "2017-05-14T03:09:03Z")
What was your dataproc submission command? I haven’t seen this error before on GCP. ---
**Author:** ![eilalan](https://avatars.discourse-cdn.com/v4/letter/e/ecc23a/32.png) [@eilalan](https://discuss.hail.is/u/eilalan)\ **Post date:** [May 14, 2017, 3:14am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/4 "2017-05-14T03:14:41Z")
i wanted to test the installation. Is this the right way to do so, or should i run submit command using gcloud? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 14, 2017, 3:18am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/5 "2017-05-14T03:18:52Z")
Definitely test using `gcloud dataproc jobs submit pyspark`. I think dataproc probably sets up some Spark config automatically, which makes the Hail ‘getting started’ setup unnecessary (and looks like it causes conflicts!). Generally, the only time we SSH into dataproc VMs is when we need to get local log files. Nearly everything goes through `gcloud`. ---
**Author:** ![eilalan](https://avatars.discourse-cdn.com/v4/letter/e/ecc23a/32.png) [@eilalan](https://discuss.hail.is/u/eilalan)\ **Post date:** [May 14, 2017, 3:25am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/6 "2017-05-14T03:25:42Z")
excellent. tested and working ---
**Author:** ![eilalan](https://avatars.discourse-cdn.com/v4/letter/e/ecc23a/32.png) [@eilalan](https://discuss.hail.is/u/eilalan)\ **Post date:** [May 14, 2017, 3:29am UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/7 "2017-05-14T03:29:27Z")
How do you shut down a cluster? by shutting down the VMs? The online suggestion was to delete them. do you know of any other option? Thanks! ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [May 14, 2017, 6:47pm UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/8 "2017-05-14T18:47:00Z")
You can shut down the cluster using the UI or the command line tool: [https://cloud.google.com/dataproc/docs/guides/manage-cluster](https://cloud.google.com/dataproc/docs/guides/manage-cluster) Shutting down the VMs one by one sounds incredibly painful, don’t do that! ---
**Author:** ![jbloom](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/jbloom/32/109_2.png) [@jbloom](https://discuss.hail.is/u/jbloom)\ **Post date:** [May 14, 2017, 6:47pm UTC](https://discuss.hail.is/t/hailcontext-sc-error/212/9 "2017-05-14T18:47:04Z")
Hi! You can do it with the GUI or on the command line. With regard to general questions about Google Cloud Platform, I’d suggest checking out their extensive docs: [https://cloud.google.com/sdk/docs/](https://cloud.google.com/sdk/docs/)