*lost* node error parts In the 200 gvcfs combiner() log file 2020-07-13 21:36:36 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 5. 2020-07-13 21:36:36 DAGScheduler: INFO: Executor lost: 5 (epoch 0) 2020-07-13 21:36:36 BlockManagerMasterEndpoint: INFO: Trying to remove executor 5 from BlockManagerMaster. 2020-07-13 21:36:36 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(5, hm32-w-3.c.myproj.internal, 33389, None) 2020-07-13 21:36:36 BlockManagerMaster: INFO: Removed 5 successfully in removeExecutor 2020-07-13 21:36:36 YarnScheduler: ERROR: Lost executor 5 on hm32-w-3.c.myproj.internal: Container marked as failed: container_1594240183345_0007_01_000009 on host: hm32-w-3.c.myproj.internal. Exit status: -100. Diagnostics: Container released on a *lost* node. 2020-07-13 21:36:36 YarnSchedulerBackend$YarnSchedulerEndpoint: WARN: Requesting driver to remove executor 5 for reason Container marked as failed: container_1594240183345_0007_01_000009 on host: hm32-w-3.c.myproj.internal. Exit status: -100. Diagnostics: Container released on a *lost* node. 2020-07-13 21:36:36 YarnSchedulerBackend$YarnSchedulerEndpoint: WARN: Requesting driver to remove executor 6 for reason Container marked as failed: container_1594240183345_0007_01_000010 on host: hm32-w-3.c.myproj.internal. Exit status: -100. Diagnostics: Container released on a *lost* node. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 262 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 259 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 258 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 261 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 264 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 263 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 257 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 260 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 ExecutorAllocationManager: INFO: Existing executor 5 has been removed (new total is 5) 2020-07-13 21:36:36 BlockManagerMasterEndpoint: INFO: Trying to remove executor 5 from BlockManagerMaster. 2020-07-13 21:36:36 BlockManagerMaster: INFO: Removal of executor 5 requested 2020-07-13 21:36:36 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Asked to remove non-existent executor 5 2020-07-13 21:36:36 YarnScheduler: ERROR: Lost executor 6 on hm32-w-3.c.myproj.internal: Container marked as failed: container_1594240183345_0007_01_000010 on host: hm32-w-3.c.myproj.internal. Exit status: -100. Diagnostics: Container released on a *lost* node. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 254 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 253 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 256 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 250 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 249 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 252 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 255 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 TaskSetManager: INFO: Task 251 failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task. 2020-07-13 21:36:36 DAGScheduler: INFO: Executor lost: 6 (epoch 0) 2020-07-13 21:36:36 ExecutorAllocationManager: WARN: Attempted to mark unknown executor 5 idle 2020-07-13 21:36:36 BlockManagerMasterEndpoint: INFO: Trying to remove executor 6 from BlockManagerMaster. 2020-07-13 21:36:36 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(6, hm32-w-3.c.myproj.internal, 45975, None) 2020-07-13 21:36:36 BlockManagerMaster: INFO: Removed 6 successfully in removeExecutor 2020-07-13 21:36:36 ExecutorAllocationManager: INFO: Existing executor 6 has been removed (new total is 4)