2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 87.2 in stage 7.0 (TID 61378) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 87, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 66.2 in stage 7.0 (TID 61379) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 66, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 24.2 in stage 7.0 (TID 61380) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 24, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 0.2 in stage 7.0 (TID 61381) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 0, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 39.2 in stage 7.0 (TID 61382) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 39, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 45.2 in stage 7.0 (TID 61383) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 45, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 15.2 in stage 7.0 (TID 61384) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 15, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 78.2 in stage 7.0 (TID 61385) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 78, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 63.2 in stage 7.0 (TID 61386) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 63, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 12.2 in stage 7.0 (TID 61387) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 12, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 30.2 in stage 7.0 (TID 61388) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 30, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:03:13 TaskSetManager: INFO: Starting task 36.2 in stage 7.0 (TID 61389) (ip-xx-xx-xx-xx.ec2.internal, executor 15, partition 36, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:46 HeartbeatReceiver: WARN: Removing executor 11 with no recent heartbeats: 134331 ms exceeds timeout 120000 ms 2022-03-03 01:05:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 11 2022-03-03 01:05:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 11 2022-03-03 01:05:46 YarnScheduler: ERROR: Lost executor 11 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 94.1 in stage 7.0 (TID 61316) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 7.1 in stage 7.0 (TID 61307) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 10.1 in stage 7.0 (TID 61298) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 82.1 in stage 7.0 (TID 61325) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 25.1 in stage 7.0 (TID 61310) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 46.1 in stage 7.0 (TID 61301) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 58.1 in stage 7.0 (TID 61319) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 55.1 in stage 7.0 (TID 61295) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 52.1 in stage 7.0 (TID 61312) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 49.1 in stage 7.0 (TID 61321) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 4.1 in stage 7.0 (TID 61294) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 61.1 in stage 7.0 (TID 61303) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 22.1 in stage 7.0 (TID 61315) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 28.1 in stage 7.0 (TID 61297) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 88.1 in stage 7.0 (TID 61306) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 91.1 in stage 7.0 (TID 61324) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 34.1 in stage 7.0 (TID 61309) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 67.1 in stage 7.0 (TID 61318) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 19.1 in stage 7.0 (TID 61300) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 40.1 in stage 7.0 (TID 61317) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 1.1 in stage 7.0 (TID 61299) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 43.1 in stage 7.0 (TID 61308) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 76.1 in stage 7.0 (TID 61320) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 37.1 in stage 7.0 (TID 61302) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 16.1 in stage 7.0 (TID 61311) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 79.1 in stage 7.0 (TID 61305) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 64.1 in stage 7.0 (TID 61296) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 73.1 in stage 7.0 (TID 61323) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 13.1 in stage 7.0 (TID 61314) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 70.1 in stage 7.0 (TID 61304) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 31.1 in stage 7.0 (TID 61313) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 TaskSetManager: WARN: Lost task 85.1 in stage 7.0 (TID 61322) (ip-xx-xx-xx-xx.ec2.internal executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 134331 ms 2022-03-03 01:05:46 DAGScheduler: INFO: Executor lost: 11 (epoch 1) 2022-03-03 01:05:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 11 from BlockManagerMaster. 2022-03-03 01:05:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(11, ip-xx-xx-xx-xx.ec2.internal, 34447, None) 2022-03-03 01:05:46 BlockManagerMaster: INFO: Removed 11 successfully in removeExecutor 2022-03-03 01:05:51 TransportChannelHandler: WARN: Exception in connection from /10.30.19.149:49698 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:05:55 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.30.19.149:49792) with ID 16, ResourceProfileId 0 2022-03-03 01:05:55 ExecutorMonitor: INFO: New executor 16 has registered (new total is 4) 2022-03-03 01:05:55 BlockManagerMasterEndpoint: INFO: Registering block manager ip-xx-xx-xx-xx.ec2.internal:39153 with 26.6 GiB RAM, BlockManagerId(16, ip-xx-xx-xx-xx.ec2.internal, 39153, None) 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 85.2 in stage 7.0 (TID 61390) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 85, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 31.2 in stage 7.0 (TID 61391) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 31, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 70.2 in stage 7.0 (TID 61392) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 70, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 13.2 in stage 7.0 (TID 61393) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 13, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 73.2 in stage 7.0 (TID 61394) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 73, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 64.2 in stage 7.0 (TID 61395) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 64, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 79.2 in stage 7.0 (TID 61396) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 79, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 16.2 in stage 7.0 (TID 61397) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 16, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 37.2 in stage 7.0 (TID 61398) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 37, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 76.2 in stage 7.0 (TID 61399) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 76, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 43.2 in stage 7.0 (TID 61400) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 43, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 1.2 in stage 7.0 (TID 61401) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 1, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 40.2 in stage 7.0 (TID 61402) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 40, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 19.2 in stage 7.0 (TID 61403) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 19, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 67.2 in stage 7.0 (TID 61404) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 67, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 34.2 in stage 7.0 (TID 61405) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 34, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 91.2 in stage 7.0 (TID 61406) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 91, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 88.2 in stage 7.0 (TID 61407) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 88, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 28.2 in stage 7.0 (TID 61408) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 28, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 22.2 in stage 7.0 (TID 61409) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 22, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 61.2 in stage 7.0 (TID 61410) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 61, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 4.2 in stage 7.0 (TID 61411) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 4, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 49.2 in stage 7.0 (TID 61412) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 49, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 52.2 in stage 7.0 (TID 61413) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 52, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 55.2 in stage 7.0 (TID 61414) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 55, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 58.2 in stage 7.0 (TID 61415) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 58, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 46.2 in stage 7.0 (TID 61416) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 46, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 25.2 in stage 7.0 (TID 61417) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 25, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 82.2 in stage 7.0 (TID 61418) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 82, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 10.2 in stage 7.0 (TID 61419) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 10, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 7.2 in stage 7.0 (TID 61420) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 7, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 TaskSetManager: INFO: Starting task 94.2 in stage 7.0 (TID 61421) (ip-xx-xx-xx-xx.ec2.internal, executor 16, partition 94, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece12 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:55 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 2.4 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:39153 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:56 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece12 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 2.4 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:40773 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 2.4 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:05:57 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:41867 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:13:46 HeartbeatReceiver: WARN: Removing executor 16 with no recent heartbeats: 164812 ms exceeds timeout 120000 ms 2022-03-03 01:13:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 16 2022-03-03 01:13:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 16 2022-03-03 01:13:46 YarnScheduler: ERROR: Lost executor 16 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 16.2 in stage 7.0 (TID 61397) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 91.2 in stage 7.0 (TID 61406) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 58.2 in stage 7.0 (TID 61415) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 34.2 in stage 7.0 (TID 61405) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 55.2 in stage 7.0 (TID 61414) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 79.2 in stage 7.0 (TID 61396) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 85.2 in stage 7.0 (TID 61390) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 25.2 in stage 7.0 (TID 61417) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 76.2 in stage 7.0 (TID 61399) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 28.2 in stage 7.0 (TID 61408) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 13.2 in stage 7.0 (TID 61393) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 40.2 in stage 7.0 (TID 61402) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 4.2 in stage 7.0 (TID 61411) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 7.2 in stage 7.0 (TID 61420) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 1.2 in stage 7.0 (TID 61401) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 61.2 in stage 7.0 (TID 61410) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 10.2 in stage 7.0 (TID 61419) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 64.2 in stage 7.0 (TID 61395) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 67.2 in stage 7.0 (TID 61404) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 52.2 in stage 7.0 (TID 61413) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 37.2 in stage 7.0 (TID 61398) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 88.2 in stage 7.0 (TID 61407) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 46.2 in stage 7.0 (TID 61416) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 70.2 in stage 7.0 (TID 61392) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 82.2 in stage 7.0 (TID 61418) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 31.2 in stage 7.0 (TID 61391) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 43.2 in stage 7.0 (TID 61400) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 22.2 in stage 7.0 (TID 61409) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 49.2 in stage 7.0 (TID 61412) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 94.2 in stage 7.0 (TID 61421) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 73.2 in stage 7.0 (TID 61394) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 TaskSetManager: WARN: Lost task 19.2 in stage 7.0 (TID 61403) (ip-xx-xx-xx-xx.ec2.internal executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 164812 ms 2022-03-03 01:13:46 DAGScheduler: INFO: Executor lost: 16 (epoch 1) 2022-03-03 01:13:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 16 from BlockManagerMaster. 2022-03-03 01:13:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(16, ip-xx-xx-xx-xx.ec2.internal, 39153, None) 2022-03-03 01:13:46 BlockManagerMaster: INFO: Removed 16 successfully in removeExecutor 2022-03-03 01:13:51 TransportChannelHandler: WARN: Exception in connection from /10.30.19.149:49792 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:13:55 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.30.19.149:49866) with ID 17, ResourceProfileId 0 2022-03-03 01:13:55 ExecutorMonitor: INFO: New executor 17 has registered (new total is 4) 2022-03-03 01:13:55 BlockManagerMasterEndpoint: INFO: Registering block manager ip-xx-xx-xx-xx.ec2.internal:39167 with 26.6 GiB RAM, BlockManagerId(17, ip-xx-xx-xx-xx.ec2.internal, 39167, None) 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 19.3 in stage 7.0 (TID 61422) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 19, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 73.3 in stage 7.0 (TID 61423) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 73, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 94.3 in stage 7.0 (TID 61424) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 94, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 49.3 in stage 7.0 (TID 61425) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 49, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 22.3 in stage 7.0 (TID 61426) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 22, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 43.3 in stage 7.0 (TID 61427) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 43, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 31.3 in stage 7.0 (TID 61428) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 31, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 82.3 in stage 7.0 (TID 61429) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 82, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 70.3 in stage 7.0 (TID 61430) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 70, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 46.3 in stage 7.0 (TID 61431) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 46, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 88.3 in stage 7.0 (TID 61432) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 88, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 37.3 in stage 7.0 (TID 61433) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 37, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 52.3 in stage 7.0 (TID 61434) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 52, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 67.3 in stage 7.0 (TID 61435) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 67, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 64.3 in stage 7.0 (TID 61436) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 64, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 10.3 in stage 7.0 (TID 61437) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 10, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 61.3 in stage 7.0 (TID 61438) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 61, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 1.3 in stage 7.0 (TID 61439) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 1, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 7.3 in stage 7.0 (TID 61440) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 7, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 4.3 in stage 7.0 (TID 61441) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 4, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 40.3 in stage 7.0 (TID 61442) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 40, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 13.3 in stage 7.0 (TID 61443) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 13, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 28.3 in stage 7.0 (TID 61444) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 28, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 76.3 in stage 7.0 (TID 61445) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 76, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 25.3 in stage 7.0 (TID 61446) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 25, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 85.3 in stage 7.0 (TID 61447) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 85, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 79.3 in stage 7.0 (TID 61448) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 79, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 55.3 in stage 7.0 (TID 61449) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 55, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 34.3 in stage 7.0 (TID 61450) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 34, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 58.3 in stage 7.0 (TID 61451) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 58, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 91.3 in stage 7.0 (TID 61452) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 91, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:13:55 TaskSetManager: INFO: Starting task 16.3 in stage 7.0 (TID 61453) (ip-xx-xx-xx-xx.ec2.internal, executor 17, partition 16, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:46 HeartbeatReceiver: WARN: Removing executor 15 with no recent heartbeats: 151890 ms exceeds timeout 120000 ms 2022-03-03 01:14:46 HeartbeatReceiver: WARN: Removing executor 14 with no recent heartbeats: 170785 ms exceeds timeout 120000 ms 2022-03-03 01:14:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 15 2022-03-03 01:14:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 15 2022-03-03 01:14:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 14 2022-03-03 01:14:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 14 2022-03-03 01:14:46 YarnScheduler: ERROR: Lost executor 15 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 30.2 in stage 7.0 (TID 61388) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 18.2 in stage 7.0 (TID 61370) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 66.2 in stage 7.0 (TID 61379) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 57.2 in stage 7.0 (TID 61361) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 12.2 in stage 7.0 (TID 61387) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 90.2 in stage 7.0 (TID 61363) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 0.2 in stage 7.0 (TID 61381) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 48.2 in stage 7.0 (TID 61372) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 15.2 in stage 7.0 (TID 61384) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 51.2 in stage 7.0 (TID 61375) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 69.2 in stage 7.0 (TID 61366) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 27.2 in stage 7.0 (TID 61360) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 87.2 in stage 7.0 (TID 61378) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 6.2 in stage 7.0 (TID 61369) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 42.2 in stage 7.0 (TID 61359) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 63.2 in stage 7.0 (TID 61386) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 9.2 in stage 7.0 (TID 61368) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 33.2 in stage 7.0 (TID 61377) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 36.2 in stage 7.0 (TID 61389) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 24.2 in stage 7.0 (TID 61380) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 84.2 in stage 7.0 (TID 61362) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 81.2 in stage 7.0 (TID 61371) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 3.2 in stage 7.0 (TID 61365) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 45.2 in stage 7.0 (TID 61383) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 75.2 in stage 7.0 (TID 61374) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 54.2 in stage 7.0 (TID 61373) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 39.2 in stage 7.0 (TID 61382) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 60.2 in stage 7.0 (TID 61364) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 21.2 in stage 7.0 (TID 61358) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 72.2 in stage 7.0 (TID 61367) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 93.2 in stage 7.0 (TID 61376) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 78.2 in stage 7.0 (TID 61385) (ip-xx-xx-xx-xx.ec2.internal executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 151890 ms 2022-03-03 01:14:46 DAGScheduler: INFO: Executor lost: 15 (epoch 1) 2022-03-03 01:14:46 YarnScheduler: ERROR: Lost executor 14 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 15 from BlockManagerMaster. 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 86.2 in stage 7.0 (TID 61352) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 11.2 in stage 7.0 (TID 61334) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 29.2 in stage 7.0 (TID 61343) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 65.2 in stage 7.0 (TID 61355) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 53.2 in stage 7.0 (TID 61346) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 38.2 in stage 7.0 (TID 61328) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 41.2 in stage 7.0 (TID 61337) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 59.2 in stage 7.0 (TID 61354) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 35.2 in stage 7.0 (TID 61336) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(15, ip-xx-xx-xx-xx.ec2.internal, 41867, None) 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 20.2 in stage 7.0 (TID 61345) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 74.2 in stage 7.0 (TID 61327) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 BlockManagerMaster: INFO: Removed 15 successfully in removeExecutor 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 8.2 in stage 7.0 (TID 61330) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 14.2 in stage 7.0 (TID 61357) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 83.2 in stage 7.0 (TID 61339) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 56.2 in stage 7.0 (TID 61348) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 5.2 in stage 7.0 (TID 61351) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 50.2 in stage 7.0 (TID 61342) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 32.2 in stage 7.0 (TID 61333) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 17.2 in stage 7.0 (TID 61341) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 71.2 in stage 7.0 (TID 61350) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 47.2 in stage 7.0 (TID 61344) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 2.2 in stage 7.0 (TID 61353) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 23.2 in stage 7.0 (TID 61326) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 68.2 in stage 7.0 (TID 61335) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 26.2 in stage 7.0 (TID 61338) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 92.2 in stage 7.0 (TID 61347) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 89.2 in stage 7.0 (TID 61356) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 80.2 in stage 7.0 (TID 61329) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 95.2 in stage 7.0 (TID 61332) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 62.2 in stage 7.0 (TID 61349) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 44.2 in stage 7.0 (TID 61331) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 TaskSetManager: WARN: Lost task 77.2 in stage 7.0 (TID 61340) (ip-xx-xx-xx-xx.ec2.internal executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170785 ms 2022-03-03 01:14:46 DAGScheduler: INFO: Executor lost: 14 (epoch 1) 2022-03-03 01:14:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 14 from BlockManagerMaster. 2022-03-03 01:14:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(14, ip-xx-xx-xx-xx.ec2.internal, 40773, None) 2022-03-03 01:14:46 BlockManagerMaster: INFO: Removed 14 successfully in removeExecutor 2022-03-03 01:14:51 TransportChannelHandler: WARN: Exception in connection from /10.30.26.130:36746 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:14:51 TransportChannelHandler: WARN: Exception in connection from /10.30.30.83:43436 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:14:55 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.30.26.130:36884) with ID 18, ResourceProfileId 0 2022-03-03 01:14:55 ExecutorMonitor: INFO: New executor 18 has registered (new total is 3) 2022-03-03 01:14:55 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.30.30.83:43558) with ID 19, ResourceProfileId 0 2022-03-03 01:14:55 ExecutorMonitor: INFO: New executor 19 has registered (new total is 4) 2022-03-03 01:14:55 BlockManagerMasterEndpoint: INFO: Registering block manager ip-xx-xx-xx-xx.ec2.internal:45089 with 26.6 GiB RAM, BlockManagerId(18, ip-xx-xx-xx-xx.ec2.internal, 45089, None) 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 77.3 in stage 7.0 (TID 61454) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 77, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 44.3 in stage 7.0 (TID 61455) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 44, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 62.3 in stage 7.0 (TID 61456) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 62, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 95.3 in stage 7.0 (TID 61457) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 95, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 80.3 in stage 7.0 (TID 61458) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 80, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 89.3 in stage 7.0 (TID 61459) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 89, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 92.3 in stage 7.0 (TID 61460) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 92, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 26.3 in stage 7.0 (TID 61461) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 26, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 68.3 in stage 7.0 (TID 61462) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 68, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 23.3 in stage 7.0 (TID 61463) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 23, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 2.3 in stage 7.0 (TID 61464) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 2, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 47.3 in stage 7.0 (TID 61465) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 47, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 71.3 in stage 7.0 (TID 61466) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 71, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 17.3 in stage 7.0 (TID 61467) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 17, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 32.3 in stage 7.0 (TID 61468) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 32, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 50.3 in stage 7.0 (TID 61469) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 50, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 5.3 in stage 7.0 (TID 61470) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 5, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 56.3 in stage 7.0 (TID 61471) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 56, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 83.3 in stage 7.0 (TID 61472) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 83, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 14.3 in stage 7.0 (TID 61473) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 14, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 8.3 in stage 7.0 (TID 61474) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 8, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 74.3 in stage 7.0 (TID 61475) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 74, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 20.3 in stage 7.0 (TID 61476) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 20, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 35.3 in stage 7.0 (TID 61477) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 35, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 59.3 in stage 7.0 (TID 61478) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 59, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 41.3 in stage 7.0 (TID 61479) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 41, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 38.3 in stage 7.0 (TID 61480) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 38, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 53.3 in stage 7.0 (TID 61481) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 53, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 65.3 in stage 7.0 (TID 61482) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 65, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 29.3 in stage 7.0 (TID 61483) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 29, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 11.3 in stage 7.0 (TID 61484) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 11, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 86.3 in stage 7.0 (TID 61485) (ip-xx-xx-xx-xx.ec2.internal, executor 18, partition 86, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 BlockManagerMasterEndpoint: INFO: Registering block manager ip-xx-xx-xx-xx.ec2.internal:37657 with 26.6 GiB RAM, BlockManagerId(19, ip-xx-xx-xx-xx.ec2.internal, 37657, None) 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 78.3 in stage 7.0 (TID 61486) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 78, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 93.3 in stage 7.0 (TID 61487) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 93, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 72.3 in stage 7.0 (TID 61488) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 72, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 21.3 in stage 7.0 (TID 61489) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 21, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 60.3 in stage 7.0 (TID 61490) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 60, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 39.3 in stage 7.0 (TID 61491) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 39, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 54.3 in stage 7.0 (TID 61492) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 54, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 75.3 in stage 7.0 (TID 61493) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 75, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 45.3 in stage 7.0 (TID 61494) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 45, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 3.3 in stage 7.0 (TID 61495) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 3, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 81.3 in stage 7.0 (TID 61496) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 81, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 84.3 in stage 7.0 (TID 61497) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 84, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 24.3 in stage 7.0 (TID 61498) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 24, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 36.3 in stage 7.0 (TID 61499) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 36, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 33.3 in stage 7.0 (TID 61500) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 33, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 9.3 in stage 7.0 (TID 61501) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 9, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 63.3 in stage 7.0 (TID 61502) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 63, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 42.3 in stage 7.0 (TID 61503) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 42, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 6.3 in stage 7.0 (TID 61504) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 6, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 87.3 in stage 7.0 (TID 61505) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 87, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 27.3 in stage 7.0 (TID 61506) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 27, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 69.3 in stage 7.0 (TID 61507) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 69, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 51.3 in stage 7.0 (TID 61508) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 51, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 15.3 in stage 7.0 (TID 61509) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 15, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 48.3 in stage 7.0 (TID 61510) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 48, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 0.3 in stage 7.0 (TID 61511) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 0, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 90.3 in stage 7.0 (TID 61512) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 90, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 12.3 in stage 7.0 (TID 61513) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 12, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 57.3 in stage 7.0 (TID 61514) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 57, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 66.3 in stage 7.0 (TID 61515) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 66, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 18.3 in stage 7.0 (TID 61516) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 18, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 TaskSetManager: INFO: Starting task 30.3 in stage 7.0 (TID 61517) (ip-xx-xx-xx-xx.ec2.internal, executor 19, partition 30, PROCESS_LOCAL, 4679 bytes) taskResourceAssignments Map() 2022-03-03 01:14:55 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:55 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:55 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:55 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 2.4 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 2.4 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece12 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:45089 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece12 in memory on ip-xx-xx-xx-xx.ec2.internal:37657 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece3 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece10 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece24 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece2 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece35 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece16 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece27 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece33 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece26 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece5 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:56 BlockManagerInfo: INFO: Added broadcast_35_piece12 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece20 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece1 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece11 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece15 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece31 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece28 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece13 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece30 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece32 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece37 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 2.4 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece8 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece25 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece23 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece22 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece6 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.5 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece17 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece29 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece21 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece7 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece34 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece14 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece4 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece18 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece9 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece36 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:14:57 BlockManagerInfo: INFO: Added broadcast_35_piece19 in memory on ip-xx-xx-xx-xx.ec2.internal:39167 (size: 4.0 MiB, free: 26.4 GiB) 2022-03-03 01:20:19 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 19. 2022-03-03 01:20:19 DAGScheduler: INFO: Executor lost: 19 (epoch 1) 2022-03-03 01:20:19 BlockManagerMasterEndpoint: INFO: Trying to remove executor 19 from BlockManagerMaster. 2022-03-03 01:20:19 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(19, ip-xx-xx-xx-xx.ec2.internal, 37657, None) 2022-03-03 01:20:19 BlockManagerMaster: INFO: Removed 19 successfully in removeExecutor 2022-03-03 01:20:19 YarnScheduler: ERROR: Lost executor 19 on ip-xx-xx-xx-xx.ec2.internal: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 YarnSchedulerBackend$YarnSchedulerEndpoint: WARN: Requesting driver to remove executor 19 for reason Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 18.3 in stage 7.0 (TID 61516) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: ERROR: Task 18 in stage 7.0 failed 4 times; aborting job 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 69.3 in stage 7.0 (TID 61507) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 48.3 in stage 7.0 (TID 61510) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 9.3 in stage 7.0 (TID 61501) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 54.3 in stage 7.0 (TID 61492) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 78.3 in stage 7.0 (TID 61486) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 3.3 in stage 7.0 (TID 61495) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 6.3 in stage 7.0 (TID 61504) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 12.3 in stage 7.0 (TID 61513) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 24.3 in stage 7.0 (TID 61498) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 21.3 in stage 7.0 (TID 61489) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 66.3 in stage 7.0 (TID 61515) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 27.3 in stage 7.0 (TID 61506) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 72.3 in stage 7.0 (TID 61488) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 84.3 in stage 7.0 (TID 61497) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 39.3 in stage 7.0 (TID 61491) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 15.3 in stage 7.0 (TID 61509) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 33.3 in stage 7.0 (TID 61500) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 90.3 in stage 7.0 (TID 61512) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 42.3 in stage 7.0 (TID 61503) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 45.3 in stage 7.0 (TID 61494) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 0.3 in stage 7.0 (TID 61511) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 75.3 in stage 7.0 (TID 61493) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 63.3 in stage 7.0 (TID 61502) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 57.3 in stage 7.0 (TID 61514) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 81.3 in stage 7.0 (TID 61496) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 87.3 in stage 7.0 (TID 61505) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 93.3 in stage 7.0 (TID 61487) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 30.3 in stage 7.0 (TID 61517) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 60.3 in stage 7.0 (TID 61490) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 36.3 in stage 7.0 (TID 61499) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 TaskSetManager: WARN: Lost task 51.3 in stage 7.0 (TID 61508) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . 2022-03-03 01:20:19 BlockManagerMasterEndpoint: INFO: Trying to remove executor 19 from BlockManagerMaster. 2022-03-03 01:20:19 BlockManagerMaster: INFO: Removal of executor 19 requested 2022-03-03 01:20:19 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Asked to remove non-existent executor 19 2022-03-03 01:20:19 YarnScheduler: INFO: Cancelling stage 7 2022-03-03 01:20:19 YarnScheduler: INFO: Killing all running tasks in stage 7: Stage cancelled 2022-03-03 01:20:19 YarnScheduler: INFO: Stage 7 was cancelled 2022-03-03 01:20:19 DAGScheduler: INFO: ResultStage 7 (collect at ContextRDD.scala:176) failed in 1855.230 s due to Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . Driver stacktrace: 2022-03-03 01:20:19 DAGScheduler: INFO: Job 7 failed: collect at ContextRDD.scala:176, took 1855.722435 s 2022-03-03 01:20:19 root: ERROR: error while applying lowering 'LowerOrInterpretNonCompilable' 2022-03-03 01:20:19 root: INFO: RegionPool: FREE: 6.7G allocated (1.9G blocks / 4.8G chunks), regions.size = 112, 0 current java objects, thread 25: Thread-5 2022-03-03 01:20:20 root: ERROR: SparkException: Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . Driver stacktrace: From org.apache.spark.SparkException: Job aborted due to stage failure: Task 18 in stage 7.0 failed 4 times, most recent failure: Lost task 18.3 in stage 7.0 (TID 61516) (ip-xx-xx-xx-xx.ec2.internal executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container from a bad node: container_1646248373496_0004_01_000024 on host: ip-xx-xx-xx-xx.ec2.internal. Exit status: 137. Diagnostics: [2022-03-03 01:20:19.743]Container killed on request. Exit code is 137 [2022-03-03 01:20:19.743]Container exited with a non-zero exit code 137. [2022-03-03 01:20:19.744]Killed by external signal . Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2465) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2414) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2413) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2413) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1124) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1124) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1124) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2679) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2621) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2610) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:914) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2259) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2278) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303) at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) at org.apache.spark.rdd.RDD.collect(RDD.scala:1029) at is.hail.sparkextras.ContextRDD.collect(ContextRDD.scala:176) at is.hail.utils.richUtils.RichContextRDD.writePartitions(RichContextRDD.scala:106) at is.hail.io.RichContextRDDLong$.writeRows$extension(RichContextRDDRegionValue.scala:229) at is.hail.rvd.RVD.write(RVD.scala:765) at is.hail.expr.ir.TableNativeWriter.apply(TableWriter.scala:113) at is.hail.expr.ir.Interpret$.run(Interpret.scala:852) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:57) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:20) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:67) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:72) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:69) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14) at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:64) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:15) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:13) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:13) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:47) at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:381) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$2(SparkBackend.scala:417) at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$1(SparkBackend.scala:414) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeEncode(SparkBackend.scala:413) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:20:22 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.30.30.83:43608) with ID 20, ResourceProfileId 0 2022-03-03 01:20:22 ExecutorMonitor: INFO: New executor 20 has registered (new total is 4) 2022-03-03 01:20:22 BlockManagerMasterEndpoint: INFO: Registering block manager ip-xx-xx-xx-xx.ec2.internal:42745 with 26.6 GiB RAM, BlockManagerId(20, ip-xx-xx-xx-xx.ec2.internal, 42745, None) 2022-03-03 01:21:22 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 20 2022-03-03 01:21:22 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 20 2022-03-03 01:21:22 ExecutorAllocationManager: INFO: Executors 20 removed due to idle timeout. 2022-03-03 01:21:23 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 20. 2022-03-03 01:21:23 DAGScheduler: INFO: Executor lost: 20 (epoch 1) 2022-03-03 01:21:23 BlockManagerMasterEndpoint: INFO: Trying to remove executor 20 from BlockManagerMaster. 2022-03-03 01:21:23 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(20, ip-xx-xx-xx-xx.ec2.internal, 42745, None) 2022-03-03 01:21:23 BlockManagerMaster: INFO: Removed 20 successfully in removeExecutor 2022-03-03 01:21:23 YarnScheduler: INFO: Executor 20 on ip-xx-xx-xx-xx.ec2.internal killed by driver. 2022-03-03 01:21:46 HeartbeatReceiver: WARN: Removing executor 18 with no recent heartbeats: 124591 ms exceeds timeout 120000 ms 2022-03-03 01:21:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 18 2022-03-03 01:21:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 18 2022-03-03 01:21:46 YarnScheduler: ERROR: Lost executor 18 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 29.3 in stage 7.0 (TID 61483) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 47.3 in stage 7.0 (TID 61465) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 8.3 in stage 7.0 (TID 61474) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 62.3 in stage 7.0 (TID 61456) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 35.3 in stage 7.0 (TID 61477) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 32.3 in stage 7.0 (TID 61468) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 89.3 in stage 7.0 (TID 61459) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 68.3 in stage 7.0 (TID 61462) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 56.3 in stage 7.0 (TID 61471) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 38.3 in stage 7.0 (TID 61480) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 41.3 in stage 7.0 (TID 61479) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 5.3 in stage 7.0 (TID 61470) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 26.3 in stage 7.0 (TID 61461) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 65.3 in stage 7.0 (TID 61482) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 2.3 in stage 7.0 (TID 61464) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 14.3 in stage 7.0 (TID 61473) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 44.3 in stage 7.0 (TID 61455) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 17.3 in stage 7.0 (TID 61467) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 20.3 in stage 7.0 (TID 61476) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 80.3 in stage 7.0 (TID 61458) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 86.3 in stage 7.0 (TID 61485) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 11.3 in stage 7.0 (TID 61484) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 50.3 in stage 7.0 (TID 61469) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 59.3 in stage 7.0 (TID 61478) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 92.3 in stage 7.0 (TID 61460) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 53.3 in stage 7.0 (TID 61481) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 83.3 in stage 7.0 (TID 61472) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 77.3 in stage 7.0 (TID 61454) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 23.3 in stage 7.0 (TID 61463) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 71.3 in stage 7.0 (TID 61466) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 95.3 in stage 7.0 (TID 61457) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 TaskSetManager: WARN: Lost task 74.3 in stage 7.0 (TID 61475) (ip-xx-xx-xx-xx.ec2.internal executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 124591 ms 2022-03-03 01:21:46 DAGScheduler: INFO: Executor lost: 18 (epoch 1) 2022-03-03 01:21:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 18 from BlockManagerMaster. 2022-03-03 01:21:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(18, ip-xx-xx-xx-xx.ec2.internal, 45089, None) 2022-03-03 01:21:46 BlockManagerMaster: INFO: Removed 18 successfully in removeExecutor 2022-03-03 01:21:49 TransportChannelHandler: WARN: Exception in connection from /10.30.26.130:36884 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:25:46 HeartbeatReceiver: WARN: Removing executor 17 with no recent heartbeats: 167044 ms exceeds timeout 120000 ms 2022-03-03 01:25:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 17 2022-03-03 01:25:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 17 2022-03-03 01:25:46 YarnScheduler: ERROR: Lost executor 17 on ip-xx-xx-xx-xx.ec2.internal: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 85.3 in stage 7.0 (TID 61447) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 4.3 in stage 7.0 (TID 61441) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 73.3 in stage 7.0 (TID 61423) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 88.3 in stage 7.0 (TID 61432) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 34.3 in stage 7.0 (TID 61450) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 67.3 in stage 7.0 (TID 61435) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 28.3 in stage 7.0 (TID 61444) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 16.3 in stage 7.0 (TID 61453) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 22.3 in stage 7.0 (TID 61426) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 82.3 in stage 7.0 (TID 61429) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 61.3 in stage 7.0 (TID 61438) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 25.3 in stage 7.0 (TID 61446) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 31.3 in stage 7.0 (TID 61428) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 10.3 in stage 7.0 (TID 61437) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 46.3 in stage 7.0 (TID 61431) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 7.3 in stage 7.0 (TID 61440) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 55.3 in stage 7.0 (TID 61449) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 19.3 in stage 7.0 (TID 61422) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 52.3 in stage 7.0 (TID 61434) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 49.3 in stage 7.0 (TID 61425) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 13.3 in stage 7.0 (TID 61443) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 91.3 in stage 7.0 (TID 61452) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 40.3 in stage 7.0 (TID 61442) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 37.3 in stage 7.0 (TID 61433) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 94.3 in stage 7.0 (TID 61424) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 58.3 in stage 7.0 (TID 61451) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 43.3 in stage 7.0 (TID 61427) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 64.3 in stage 7.0 (TID 61436) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 76.3 in stage 7.0 (TID 61445) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 1.3 in stage 7.0 (TID 61439) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 70.3 in stage 7.0 (TID 61430) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 TaskSetManager: WARN: Lost task 79.3 in stage 7.0 (TID 61448) (ip-xx-xx-xx-xx.ec2.internal executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 167044 ms 2022-03-03 01:25:46 YarnScheduler: INFO: Removed TaskSet 7.0, whose tasks have all completed, from pool 2022-03-03 01:25:46 DAGScheduler: INFO: Executor lost: 17 (epoch 1) 2022-03-03 01:25:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 17 from BlockManagerMaster. 2022-03-03 01:25:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(17, ip-xx-xx-xx-xx.ec2.internal, 39167, None) 2022-03-03 01:25:46 BlockManagerMaster: INFO: Removed 17 successfully in removeExecutor 2022-03-03 01:25:49 TransportChannelHandler: WARN: Exception in connection from /10.30.19.149:49866 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2022-03-03 01:33:12 Hail: INFO: copying log to 's3://.../log_error.txt'... 2022-03-03 01:33:13 MultipartUploadOutputStream: INFO: uploadPart: partNum 1 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000000', 134217728 bytes in 1133 ms, md5: b8j7KfCrPjJDHPLHep0KcA== md5hex: 6fc8fb29f0ab3e32431cf2c77a9d0a70 2022-03-03 01:33:14 MultipartUploadOutputStream: INFO: uploadPart: partNum 2 of 's3://.../log_error.txt' from local file '/mnt1/s3/emrfs-2347116993970780860/0000000001', 134217728 bytes in 1198 ms, md5: JbbZ3FXGLGyKJomoCxgwew== md5hex: 25b6d9dc55c62c6c8a2689a80b18307b 2022-03-03 01:33:16 MultipartUploadOutputStream: INFO: uploadPart: partNum 4 of 's3://.../log_error.txt' from local file '/mnt1/s3/emrfs-2347116993970780860/0000000003', 134217728 bytes in 1179 ms, md5: up96iSCcZoVdPJUp25xoqg== md5hex: ba9f7a89209c66855d3c9529db9c68aa 2022-03-03 01:33:16 MultipartUploadOutputStream: INFO: uploadPart: partNum 5 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000004', 134217728 bytes in 1083 ms, md5: FVlKTjwegzadKLWP7JZGfg== md5hex: 15594a4e3c1e83369d28b58fec96467e 2022-03-03 01:33:17 MultipartUploadOutputStream: INFO: uploadPart: partNum 6 of 's3://.../log_error.txt' from local file '/mnt1/s3/emrfs-2347116993970780860/0000000005', 134217728 bytes in 1100 ms, md5: n/zKBAIqLt+g2+d515eVPg== md5hex: 9ffcca04022a2edfa0dbe779d797953e 2022-03-03 01:33:17 MultipartUploadOutputStream: INFO: uploadPart: partNum 7 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000006', 134217728 bytes in 1147 ms, md5: 41jbtwwcto5HTSY6VlM7yg== md5hex: e358dbb70c1cb68e474d263a56533bca 2022-03-03 01:33:18 MultipartUploadOutputStream: INFO: uploadPart: partNum 8 of 's3://.../log_error.txt' from local file '/mnt1/s3/emrfs-2347116993970780860/0000000007', 134217728 bytes in 1075 ms, md5: fAgnI5bAHdY/qzbO34bXxw== md5hex: 7c08272396c01dd63fab36cedf86d7c7 2022-03-03 01:33:18 MultipartUploadOutputStream: INFO: uploadPart: partNum 9 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000008', 134217728 bytes in 1115 ms, md5: a/Cd1z3fEuGYrhMQxLj/ww== md5hex: 6bf09dd73ddf12e198ae1310c4b8ffc3 2022-03-03 01:33:19 MultipartUploadOutputStream: INFO: uploadPart: partNum 10 of 's3://.../log_error.txt' from local file '/mnt1/s3/emrfs-2347116993970780860/0000000009', 134217728 bytes in 1056 ms, md5: P1n0v1Dm3PgD+sIhm0UzFg== md5hex: 3f59f4bf50e6dcf803fac2219b453316 2022-03-03 01:33:19 MultipartUploadOutputStream: INFO: uploadPart: partNum 11 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000010', 134217728 bytes in 906 ms, md5: gLosbh0ojT4u3GOPGIuQNw== md5hex: 80ba2c6e1d288d3e2edc638f188b9037 2022-03-03 01:33:20 MultipartUploadOutputStream: INFO: uploadPart: partNum 3 of 's3://.../log_error.txt' from local file '/mnt/s3/emrfs-3559139166741393986/0000000002', 134217728 bytes in 5889 ms, md5: k6QMpEeXsbgA7v1MxjAb5w== md5hex: 93a40ca44797b1b800eefd4cc6301be7