I am running a job like this .
bsub -n 2000 -e bsub.err -o bsub.out lsf-spark202-submit.sh --conf spark.driver.maxResultSize=0 --conf spark.reducer.maxSizeInFlight=100m --conf spark.shuffle.file.buffer=64k --conf spark.akka.frameSize=256 --conf spark.akka.threads=10 --conf spark.driver.cores=32 --conf spark.executor.memory=150g --conf spark.local.dir=/gpfs/ngsdata/sparkcache --conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator --conf spark.driver.memory=100g --conf “spark.driver.extraClassPath=/gpfs/software/spark/hail-all-spark.jar” --conf “spark.executor.extraClassPath=/gpfs/software/spark/hail-all-spark.jar” --jars /gpfs/software/spark/hail-all-spark.jar --py-files /gpfs/software/spark/hail-python.zip hailimport.py
Where I am trying to import a VCF file into VDS. VCF file is pretty big its consisting of around 3000 WGS. I am getting below error.
2017-09-22 14:52:59 TaskSchedulerImpl: INFO: Removed TaskSet 5.0, whose tasks have all completed, from pool
2017-09-22 14:52:59 DAGScheduler: INFO: ResultStage 5 (fold at ReadRowsRDD.scala:307) finished in 549.695 s
2017-09-22 14:52:59 DAGScheduler: INFO: Job 4 finished: fold at ReadRowsRDD.scala:307, took 1244.862619 s
2017-09-22 14:52:59 Hail: INFO: wrote 58713573 records
2017-09-22 14:53:00 SparkContext: INFO: Invoking stop() from shutdown hook
2017-09-22 14:53:00 ServerConnector: INFO: Stopped ServerConnector@59a045bc{HTTP/1.1}{0.0.0.0:4040}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@72d12d55{/stages/stage/kill,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5f04598a{/api,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@29593992{/,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@2699fad3{/static,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@345d7a8{/executors/threadDump/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@179c26fd{/executors/threadDump,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@710fc80a{/executors/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@7e1b6c47{/executors,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5b39fe98{/environment/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@1e9f1d53{/environment,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@799ec0be{/storage/rdd/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@3072beb6{/storage/rdd,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@3fbf5b9b{/storage/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@4845e46a{/storage,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@118b05db{/stages/pool/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@2045f0fa{/stages/pool,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@6ed32bc2{/stages/stage/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5f30d4f4{/stages/stage,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@404e3627{/stages/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@15ab7e7c{/stages,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@19e506e6{/jobs/job/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@63ff83d1{/jobs/job,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@77644a84{/jobs/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5eca411f{/jobs,null,UNAVAILABLE}
2017-09-22 14:53:00 SparkUI: INFO: Stopped Spark web UI at http://172.32.65.103:4040
2017-09-22 14:53:00 StandaloneSchedulerBackend: INFO: Shutting down all executors
2017-09-22 14:53:00 CoarseGrainedSchedulerBackend$DriverEndpoint: INFO: Asking each executor to shut down
2017-09-22 14:53:00 MapOutputTrackerMasterEndpoint: INFO: MapOutputTrackerMasterEndpoint stopped!
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.103:51440 in memory (size: 1616.3 KB, free: 53.2 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(driver, 172.32.65.103, 51440),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.94:48993 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(33, 172.32.65.94, 48993),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.97:51849 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(54, 172.32.65.97, 51849),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.103:54951 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(38, 172.32.65.103, 54951),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.122:42720 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(19, 172.32.65.122, 42720),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.71:35570 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(9, 172.32.65.71, 35570),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.127:41786 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(49, 172.32.65.127, 41786),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.111:57754 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(16, 172.32.65.111, 57754),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.72:35595 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(48, 172.32.65.72, 35595),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.113:55277 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(7, 172.32.65.113, 55277),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.116:60369 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(15, 172.32.65.116, 60369),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.128:58200 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(13, 172.32.65.128, 58200),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.132:43076 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(52, 172.32.65.132, 43076),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.99:43106 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(42, 172.32.65.99, 43106),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.108:57121 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(35, 172.32.65.108, 57121),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.130:51636 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(43, 172.32.65.130, 51636),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.107:38782 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0, 172.32.65.107, 38782),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.110:41985 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(20, 172.32.65.110, 41985),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.109:58089 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(37, 172.32.65.109, 58089),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.104:52445 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(59, 172.32.65.104, 52445),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.92:33948 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(3, 172.32.65.92, 33948),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.30:54100 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(11, 172.32.65.30, 54100),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.35:56913 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(53, 172.32.65.35, 56913),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.33:38020 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(51, 172.32.65.33, 38020),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.29:48911 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(30, 172.32.65.29, 48911),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.96:57033 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(2, 172.32.65.96, 57033),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.64:34028 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(27, 172.32.65.64, 34028),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.112:49582 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(10, 172.32.65.112, 49582),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.32:38945 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(45, 172.32.65.32, 38945),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.123:35704 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(41, 172.32.65.123, 35704),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.134:56513 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(44, 172.32.65.134, 56513),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.133:34881 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(18, 172.32.65.133, 34881),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.125:57159 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(56, 172.32.65.125, 57159),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.106:47423 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(22, 172.32.65.106, 47423),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.120:52585 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(28, 172.32.65.120, 52585),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.131:53616 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(62, 172.32.65.131, 53616),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.105:41990 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(21, 172.32.65.105, 41990),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.27:44031 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(39, 172.32.65.27, 44031),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.31:35351 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(6, 172.32.65.31, 35351),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.47:40482 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(25, 172.32.65.47, 40482),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.124:57339 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(8, 172.32.65.124, 57339),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.126:56958 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(47, 172.32.65.126, 56958),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.129:42369 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(26, 172.32.65.129, 42369),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.49:41447 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(58, 172.32.65.49, 41447),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.16:43966 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(60, 172.32.65.16, 43966),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.53:34089 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(17, 172.32.65.53, 34089),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.101:39076 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(24, 172.32.65.101, 39076),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.58:55720 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(50, 172.32.65.58, 55720),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.98:42298 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(23, 172.32.65.98, 42298),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.73:37621 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(36, 172.32.65.73, 37621),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.28:47033 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(12, 172.32.65.28, 47033),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.66:50770 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(29, 172.32.65.66, 50770),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.79:51415 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(34, 172.32.65.79, 51415),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.121:55074 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(55, 172.32.65.121, 55074),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.59:32978 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(32, 172.32.65.59, 32978),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.12:38496 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(57, 172.32.65.12, 38496),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.75:53325 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(4, 172.32.65.75, 53325),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.51:33173 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(14, 172.32.65.51, 33173),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.20:57858 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1, 172.32.65.20, 57858),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.62:38664 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, 172.32.65.62, 38664),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.48:45631 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(40, 172.32.65.48, 45631),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.70:42445 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(31, 172.32.65.70, 42445),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.26:44344 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(46, 172.32.65.26, 44344),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.13:48556 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(5, 172.32.65.13, 48556),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 MemoryStore: INFO: MemoryStore cleared
2017-09-22 14:53:00 BlockManager: INFO: BlockManager stopped
2017-09-22 14:53:00 BlockManagerMaster: INFO: BlockManagerMaster stopped
2017-09-22 14:53:00 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: INFO: OutputCommitCoordinator stopped!
2017-09-22 14:53:00 SparkContext: INFO: Successfully stopped SparkContext
2017-09-22 14:53:00 ShutdownHookManager: INFO: Shutdown hook called
2017-09-22 14:53:00 ShutdownHookManager: INFO: Deleting directory /gpfs/projects/hpcinfra/sparktmp/spark-30c78559-d3c3-41cd-a889-70c04ebe88da
2017-09-22 14:53:00 ShutdownHookManager: INFO: Deleting directory /gpfs/projects/hpcinfra/sparktmp/spark-30c78559-d3c3-41cd-a889-70c04ebe88da/pyspark-3b1acc69-cea7-4b64-877c-597c85be4d04
I am not sure why this error is coming ? Is it because of big size of VCF file ?
Thanks
Najeeb