# Pca failed due to not enough executor memory **URL:** \ **Category:** Help \[0.1\]\ **Created:** [September 13, 2017, 2:30am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285 "2017-09-13T02:30:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jerryliu2005](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/jerryliu2005/32/97_2.png) [@jerryliu2005](https://discuss.hail.is/u/jerryliu2005)\ **Post date:** [September 13, 2017, 2:30am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/1 "2017-09-13T02:30:43Z")
Hello, My pca test failed with the following error. The job is running on a Cloudera cluster using hail devel-7beaca7. I had to use the dev version as our VCF doesn’t have the PL field which was required for v.0.1 to load GT properly. What would you suggest I do? Where do I “boosting spark.yarn.executor.memoryOverhead” as the error message suggested? Thanks very much! Jerry \*\*\*\*\*\* Hail command and error msg below \*\*\*\*\*\*\*\*\*\* pca=vds.pca(‘scores=sa.pca’, loadings=None, eigenvalues=None, k=5, as\_array=False) 2017-09-12 22:16:44 Hail: INFO: Running PCA with 5 components… [Stage 0:\> (0 + 128) / 140]--------------------------------------------------------------------------- FatalError Traceback (most recent call last) in () ----\> 1 pca=vds.pca(‘scores=sa.pca’, loadings=None, eigenvalues=None, k=5, as\_array=False) in pca(self, scores, loadings, eigenvalues, k, as\_array) PATH/hail-devel-7beaca7/build/distributions/hail-python.zip/hail/java.pyc in handle\_py4j(func, \*args, \*\*kwargs) 119 raise FatalError(’%s\n\nJava stack trace:\n%s\n’ 120 ‘Hail version: %s\n’ –\> 121 ‘Error summary: %s’ % (deepest, full, Env.hc().version, deepest)) 122 except py4j.protocol.Py4JError as e: 123 if e.args[0].startswith(‘An error occurred while calling’): FatalError: SparkException: Job aborted due to stage failure: Task 11 in stage 0.0 failed 4 times, most recent failure: Lost task 11.3 in stage 0.0 (TID 159, [itmiprd1cdhd02.itmi.inova.org](http://itmiprd1cdhd02.itmi.inova.org), executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 1.7 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Driver stacktrace: Java stack trace: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 0.0 failed 4 times, most recent failure: Lost task 11.3 in stage 0.0 (TID 159, [itmiprd1cdhd02.itmi.inova.org](http://itmiprd1cdhd02.itmi.inova.org), executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 1.7 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.collect(RDD.scala:934) at is.hail.stats.ToHWENormalizedIndexedRowMatrix$.apply(ComputeRRM.scala:104) at is.hail.methods.SamplePCA$.variantsSvdAndScores(SamplePCA.scala:51) at is.hail.methods.SamplePCA$.apply(SamplePCA.scala:31) at is.hail.variant.VariantDatasetFunctions$.pca$extension(VariantDataset.scala:508) at is.hail.variant.VariantDatasetFunctions.pca(VariantDataset.scala:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Hail version: devel-7beaca7 Error summary: SparkException: Job aborted due to stage failure: Task 11 in stage 0.0 failed 4 times, most recent failure: Lost task 11.3 in stage 0.0 (TID 159, [itmiprd1cdhd02.itmi.inova.org](http://itmiprd1cdhd02.itmi.inova.org), executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 1.7 GB of 1.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Driver stacktrace: ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 13, 2017, 10:45am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/2 "2017-09-13T10:45:46Z")
Hi Jerry, The YARN defaults aren’t super great for a lot of the memory-intensive linear algebra routines we use. One way to make things a bit better is to increase the number of cores per executor from 1 to 4, so that all the overhead per JVM drops by a factor of 4. You can do this by setting `spark.executor.cores=4` in the spark configuration. ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [June 2, 2020, 7:00pm UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/3 "2020-06-02T19:00:46Z")
@tpoterba I am having the same error as OP and I would like to increase the # of executor cores but I am using Hail in Terra. I know there is some collaboration between Hail and Terra with regards to a Terra Base Image (which I am currently using). Do you know if there is a custom image I can use in Terra that includes Hail & Spark, with the executor.cores parameter changed as you describe? Is it simple enough to change this parameter in a Dockerfile if I was to make a custom image for my Terra notebook? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [June 3, 2020, 1:37am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/4 "2020-06-03T01:37:25Z")
You should be able to pass this parameter in hail initialization: ```python import hail as hl hl.init(spark_conf={'spark.executor.cores': '4'}) ``` ---
**Author:** ![acererak](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@acererak](https://discuss.hail.is/u/acererak)\ **Post date:** [June 3, 2020, 4:09pm UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/5 "2020-06-03T16:09:35Z")
Thanks @tpoterba. I later figured that out and was able to increase the # of executor cores but unfortunately I am still getting SparkExceptions on this PCA run. I have created a new line of inquiry at [PCA job aborted from SparkException](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433) so as not to sidetrack this older thread. Do you think the issue is in Spark configurations or in the cluster resources? I am currently running with 24 nodes (12 of which are preemptible) but I am experimenting with increasing compute resources even further. ---
**Author:** ![kumarveerapen](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kumarveerapen/32/355_2.png) [@kumarveerapen](https://discuss.hail.is/u/kumarveerapen)\ **Post date:** [July 16, 2020, 3:41am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/6 "2020-07-16T03:41:34Z")
@tpoterba any updates about this? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [July 16, 2020, 10:34am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/7 "2020-07-16T10:34:43Z")
we’re actively discussing here: [PCA job aborted from SparkException](https://discuss.hail.is/t/pca-job-aborted-from-sparkexception/1433/40) ---
**Author:** ![kumarveerapen](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/kumarveerapen/32/355_2.png) [@kumarveerapen](https://discuss.hail.is/u/kumarveerapen)\ **Post date:** [July 16, 2020, 11:59am UTC](https://discuss.hail.is/t/pca-failed-due-to-not-enough-executor-memory/285/8 "2020-07-16T11:59:19Z")
Gosh! My apologies for not reading this thread as well as I should have! Thanks, Tim! 🙂