SparkContext: INFO: Invoking stop() from shutdown hook


#1

I am running a job like this .

bsub -n 2000 -e bsub.err -o bsub.out lsf-spark202-submit.sh --conf spark.driver.maxResultSize=0 --conf spark.reducer.maxSizeInFlight=100m --conf spark.shuffle.file.buffer=64k --conf spark.akka.frameSize=256 --conf spark.akka.threads=10 --conf spark.driver.cores=32 --conf spark.executor.memory=150g --conf spark.local.dir=/gpfs/ngsdata/sparkcache --conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator --conf spark.driver.memory=100g --conf “spark.driver.extraClassPath=/gpfs/software/spark/hail-all-spark.jar” --conf “spark.executor.extraClassPath=/gpfs/software/spark/hail-all-spark.jar” --jars /gpfs/software/spark/hail-all-spark.jar --py-files /gpfs/software/spark/hail-python.zip hailimport.py

Where I am trying to import a VCF file into VDS. VCF file is pretty big its consisting of around 3000 WGS. I am getting below error.

2017-09-22 14:52:59 TaskSchedulerImpl: INFO: Removed TaskSet 5.0, whose tasks have all completed, from pool
2017-09-22 14:52:59 DAGScheduler: INFO: ResultStage 5 (fold at ReadRowsRDD.scala:307) finished in 549.695 s
2017-09-22 14:52:59 DAGScheduler: INFO: Job 4 finished: fold at ReadRowsRDD.scala:307, took 1244.862619 s
2017-09-22 14:52:59 Hail: INFO: wrote 58713573 records
2017-09-22 14:53:00 SparkContext: INFO: Invoking stop() from shutdown hook
2017-09-22 14:53:00 ServerConnector: INFO: Stopped ServerConnector@59a045bc{HTTP/1.1}{0.0.0.0:4040}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@72d12d55{/stages/stage/kill,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5f04598a{/api,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@29593992{/,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@2699fad3{/static,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@345d7a8{/executors/threadDump/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@179c26fd{/executors/threadDump,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@710fc80a{/executors/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@7e1b6c47{/executors,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5b39fe98{/environment/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@1e9f1d53{/environment,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@799ec0be{/storage/rdd/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@3072beb6{/storage/rdd,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@3fbf5b9b{/storage/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@4845e46a{/storage,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@118b05db{/stages/pool/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@2045f0fa{/stages/pool,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@6ed32bc2{/stages/stage/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5f30d4f4{/stages/stage,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@404e3627{/stages/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@15ab7e7c{/stages,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@19e506e6{/jobs/job/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@63ff83d1{/jobs/job,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@77644a84{/jobs/json,null,UNAVAILABLE}
2017-09-22 14:53:00 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@5eca411f{/jobs,null,UNAVAILABLE}
2017-09-22 14:53:00 SparkUI: INFO: Stopped Spark web UI at http://172.32.65.103:4040
2017-09-22 14:53:00 StandaloneSchedulerBackend: INFO: Shutting down all executors
2017-09-22 14:53:00 CoarseGrainedSchedulerBackend$DriverEndpoint: INFO: Asking each executor to shut down
2017-09-22 14:53:00 MapOutputTrackerMasterEndpoint: INFO: MapOutputTrackerMasterEndpoint stopped!
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.103:51440 in memory (size: 1616.3 KB, free: 53.2 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(driver, 172.32.65.103, 51440),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.94:48993 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(33, 172.32.65.94, 48993),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.97:51849 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(54, 172.32.65.97, 51849),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.103:54951 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(38, 172.32.65.103, 54951),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.122:42720 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(19, 172.32.65.122, 42720),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.71:35570 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(9, 172.32.65.71, 35570),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.127:41786 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(49, 172.32.65.127, 41786),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.111:57754 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(16, 172.32.65.111, 57754),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.72:35595 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(48, 172.32.65.72, 35595),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.113:55277 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(7, 172.32.65.113, 55277),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.116:60369 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(15, 172.32.65.116, 60369),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.128:58200 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(13, 172.32.65.128, 58200),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.132:43076 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(52, 172.32.65.132, 43076),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.99:43106 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(42, 172.32.65.99, 43106),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.108:57121 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(35, 172.32.65.108, 57121),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.130:51636 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(43, 172.32.65.130, 51636),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.107:38782 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0, 172.32.65.107, 38782),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.110:41985 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(20, 172.32.65.110, 41985),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.109:58089 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(37, 172.32.65.109, 58089),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.104:52445 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(59, 172.32.65.104, 52445),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.92:33948 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(3, 172.32.65.92, 33948),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.30:54100 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(11, 172.32.65.30, 54100),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.35:56913 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(53, 172.32.65.35, 56913),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.33:38020 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(51, 172.32.65.33, 38020),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.29:48911 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(30, 172.32.65.29, 48911),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.96:57033 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(2, 172.32.65.96, 57033),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.64:34028 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(27, 172.32.65.64, 34028),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.112:49582 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(10, 172.32.65.112, 49582),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.32:38945 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(45, 172.32.65.32, 38945),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.123:35704 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(41, 172.32.65.123, 35704),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.134:56513 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(44, 172.32.65.134, 56513),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.133:34881 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(18, 172.32.65.133, 34881),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.125:57159 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(56, 172.32.65.125, 57159),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.106:47423 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(22, 172.32.65.106, 47423),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.120:52585 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(28, 172.32.65.120, 52585),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.131:53616 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(62, 172.32.65.131, 53616),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.105:41990 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(21, 172.32.65.105, 41990),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.27:44031 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(39, 172.32.65.27, 44031),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.31:35351 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(6, 172.32.65.31, 35351),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.47:40482 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(25, 172.32.65.47, 40482),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.124:57339 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(8, 172.32.65.124, 57339),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.126:56958 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(47, 172.32.65.126, 56958),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.129:42369 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(26, 172.32.65.129, 42369),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.49:41447 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(58, 172.32.65.49, 41447),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.16:43966 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(60, 172.32.65.16, 43966),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.53:34089 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(17, 172.32.65.53, 34089),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.101:39076 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(24, 172.32.65.101, 39076),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.58:55720 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(50, 172.32.65.58, 55720),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.98:42298 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(23, 172.32.65.98, 42298),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.73:37621 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(36, 172.32.65.73, 37621),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.28:47033 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(12, 172.32.65.28, 47033),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.66:50770 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(29, 172.32.65.66, 50770),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.79:51415 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(34, 172.32.65.79, 51415),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.121:55074 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(55, 172.32.65.121, 55074),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.59:32978 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(32, 172.32.65.59, 32978),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.12:38496 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(57, 172.32.65.12, 38496),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.75:53325 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(4, 172.32.65.75, 53325),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.51:33173 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(14, 172.32.65.51, 33173),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.20:57858 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1, 172.32.65.20, 57858),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.62:38664 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, 172.32.65.62, 38664),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.48:45631 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(40, 172.32.65.48, 45631),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.70:42445 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(31, 172.32.65.70, 42445),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.26:44344 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(46, 172.32.65.26, 44344),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 BlockManagerInfo: INFO: Removed broadcast_37_piece0 on 172.32.65.13:48556 in memory (size: 1616.3 KB, free: 79.8 GB)
2017-09-22 14:53:00 LiveListenerBus: ERROR: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(5, 172.32.65.13, 48556),broadcast_37_piece0,StorageLevel(1 replicas),0,0))
2017-09-22 14:53:00 MemoryStore: INFO: MemoryStore cleared
2017-09-22 14:53:00 BlockManager: INFO: BlockManager stopped
2017-09-22 14:53:00 BlockManagerMaster: INFO: BlockManagerMaster stopped
2017-09-22 14:53:00 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: INFO: OutputCommitCoordinator stopped!
2017-09-22 14:53:00 SparkContext: INFO: Successfully stopped SparkContext
2017-09-22 14:53:00 ShutdownHookManager: INFO: Shutdown hook called
2017-09-22 14:53:00 ShutdownHookManager: INFO: Deleting directory /gpfs/projects/hpcinfra/sparktmp/spark-30c78559-d3c3-41cd-a889-70c04ebe88da
2017-09-22 14:53:00 ShutdownHookManager: INFO: Deleting directory /gpfs/projects/hpcinfra/sparktmp/spark-30c78559-d3c3-41cd-a889-70c04ebe88da/pyspark-3b1acc69-cea7-4b64-877c-597c85be4d04

I am not sure why this error is coming ? Is it because of big size of VCF file ?

Thanks
Najeeb


#2

Hi Najeeb,

Two things:
first, I think the job finished successfully! This line:

 2017-09-22 14:52:59 Hail: INFO: wrote 58713573 records

is used to indicate write completion.

– However–

You’re using the devel branch, not the latest version of the 0.1 stable branch. We’re not supporting the development branch, the interface is changing all the time, and the file formats can become invalid from one day to another. You should definitely back up to 0.1 until we make another stable release.

This will involve needing to rerun the pipeline, because the VDS file format is totally different. I’m sorry!


#3

Thanks !! yes I will rerun using 0.1 stable vesion.
Even though job finished succesfully I dont have any data inside folder.
[nsyed@ngslogin1 liftOver]$ ls -lR /gpfs/ngsdata/scratch/liftOver/QGP2935.JC_GATK37.dbSnp.eff.vds/rowstore/
/gpfs/ngsdata/scratch/liftOver/QGP2935.JC_GATK37.dbSnp.eff.vds/rowstore/:
total 0
Can you suggest on this Please


#4

Thanks !! using new Hail 01 solved my problme !!

Thanks alot!!