2021-07-23 15:34:25 Hail: INFO: Running Hail version 0.2.73-2ab84582d24d
2021-07-23 15:34:25 SparkContext: WARN: Using an existing SparkContext; some configuration may not take effect.
2021-07-23 15:34:27 root: INFO: RegionPool: initialized for thread 18: Thread-5
2021-07-23 15:34:27 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 50.2 KiB, free 21.7 GiB)
2021-07-23 15:34:27 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.157:45254) with ID 1,  ResourceProfileId 0
2021-07-23 15:34:27 ExecutorMonitor: INFO: New executor 1 has registered (new total is 1)
2021-07-23 15:34:27 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.141:58632) with ID 2,  ResourceProfileId 0
2021-07-23 15:34:27 ExecutorMonitor: INFO: New executor 2 has registered (new total is 2)
2021-07-23 15:34:27 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-23.c.broad-mpg-gnomad.internal:38567 with 11.2 GiB RAM, BlockManagerId(1, kc-w-23.c.broad-mpg-gnomad.internal, 38567, None)
2021-07-23 15:34:27 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 21.7 GiB)
2021-07-23 15:34:27 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:46573 (size: 3.2 KiB, free: 21.7 GiB)
2021-07-23 15:34:27 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:278
2021-07-23 15:34:27 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-3.c.broad-mpg-gnomad.internal:44197 with 11.2 GiB RAM, BlockManagerId(2, kc-w-3.c.broad-mpg-gnomad.internal, 44197, None)
2021-07-23 15:34:28 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 18: Thread-5
2021-07-23 15:34:28 root: INFO: timing SparkBackend.parse_table_ir total 1.546s self 1.546s children 0.000ms %children 0.00%
2021-07-23 15:34:28 root: INFO: RegionPool: initialized for thread 18: Thread-5
2021-07-23 15:34:28 root: INFO: JSON: JObject(List((name,JString(TableCalculateNewPartitions)), (nPartitions,JInt(5000))))
2021-07-23 15:34:28 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 18: Thread-5
2021-07-23 15:34:28 root: INFO: timing SparkBackend.parse_value_ir total 32.654ms self 32.654ms children 0.000ms %children 0.00%
2021-07-23 15:34:28 root: INFO: RegionPool: initialized for thread 18: Thread-5
2021-07-23 15:34:28 root: INFO: starting execution of query hail_query_1 of initial size 2
2021-07-23 15:34:28 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: 
(TableToValueApply
  "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":5000}"
  (TableRead None False
    (TableNativeReader gs://regional_missense_constraint/temp/simul_break_prep_no_end.ht )))
2021-07-23 15:34:29 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2:
(TableToValueApply
  "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":5000}"
  (TableRead None False
    (TableNativeReader gs://regional_missense_constraint/temp/simul_break_prep_no_end.ht )))
2021-07-23 15:34:29 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: 
(TableToValueApply
  "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":5000}"
  (TableRead None False
    (TableNativeReader gs://regional_missense_constraint/temp/simul_break_prep_no_end.ht )))
2021-07-23 15:34:29 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2:
(TableToValueApply
  "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":5000}"
  (TableRead None False
    (TableNativeReader gs://regional_missense_constraint/temp/simul_break_prep_no_end.ht )))
2021-07-23 15:34:29 root: INFO: interpreting non compilable node: TableToValueApply
2021-07-23 15:34:29 root: INFO: decoder cache miss (0 hits, 1 misses, 0.000
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C1FSContainer.<init>
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C1FSContainer.<clinit>
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C2etypeDecode.<init>
2021-07-23 15:34:29 root: INFO: instruction count: 5: __C2etypeDecode.apply
2021-07-23 15:34:29 root: INFO: instruction count: 16: __C2etypeDecode.__m3DECODE_r_struct_of_END_TO_o_struct_of_END
2021-07-23 15:34:29 root: INFO: decoder cache miss (0 hits, 2 misses, 0.000
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C4FSContainer.<init>
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C4FSContainer.<clinit>
2021-07-23 15:34:29 root: INFO: instruction count: 3: __C5etypeDecode.<init>
2021-07-23 15:34:29 root: INFO: instruction count: 5: __C5etypeDecode.apply
2021-07-23 15:34:29 root: INFO: instruction count: 179: __C5etypeDecode.__m6DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_binaryANDo_struct_of_r_binaryANDr_int32ENDANDo_int32ANDo_int32ANDo_int32END_TO_o_struct_of_o_locusANDo_stringANDo_locusANDo_int32ANDo_int32ANDo_int32END
2021-07-23 15:34:29 root: INFO: instruction count: 22: __C5etypeDecode.__m7INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus
2021-07-23 15:34:29 root: INFO: instruction count: 33: __C5etypeDecode.__m8INPLACE_DECODE_r_binary_TO_r_string
2021-07-23 15:34:29 root: INFO: instruction count: 10: __C5etypeDecode.__m9INPLACE_DECODE_r_int32_TO_r_int32
2021-07-23 15:34:29 root: INFO: instruction count: 33: __C5etypeDecode.__m10INPLACE_DECODE_o_binary_TO_o_string
2021-07-23 15:34:29 root: INFO: instruction count: 10: __C5etypeDecode.__m11INPLACE_DECODE_o_int32_TO_o_int32
2021-07-23 15:34:29 MemoryStore: INFO: Block broadcast_1 stored as values in memory (estimated size 423.9 KiB, free 21.7 GiB)
2021-07-23 15:34:29 MemoryStore: INFO: Block broadcast_1_piece0 stored as bytes in memory (estimated size 33.3 KiB, free 21.7 GiB)
2021-07-23 15:34:29 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:46573 (size: 33.3 KiB, free: 21.7 GiB)
2021-07-23 15:34:29 SparkContext: INFO: Created broadcast 1 from broadcast at SparkBackend.scala:278
2021-07-23 15:34:30 SparkContext: INFO: Starting job: runJob at ContextRDD.scala:238
2021-07-23 15:34:30 DAGScheduler: INFO: Got job 0 (runJob at ContextRDD.scala:238) with 1 output partitions
2021-07-23 15:34:30 DAGScheduler: INFO: Final stage: ResultStage 0 (runJob at ContextRDD.scala:238)
2021-07-23 15:34:30 DAGScheduler: INFO: Parents of final stage: List()
2021-07-23 15:34:30 DAGScheduler: INFO: Missing parents: List()
2021-07-23 15:34:30 DAGScheduler: INFO: Submitting ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at ContextRDD.scala:222), which has no missing parents
2021-07-23 15:34:30 MemoryStore: INFO: Block broadcast_2 stored as values in memory (estimated size 21.2 KiB, free 21.7 GiB)
2021-07-23 15:34:30 MemoryStore: INFO: Block broadcast_2_piece0 stored as bytes in memory (estimated size 10.9 KiB, free 21.7 GiB)
2021-07-23 15:34:30 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:46573 (size: 10.9 KiB, free: 21.7 GiB)
2021-07-23 15:34:30 SparkContext: INFO: Created broadcast 2 from broadcast at DAGScheduler.scala:1383
2021-07-23 15:34:30 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at ContextRDD.scala:222) (first 15 tasks are for partitions Vector(0))
2021-07-23 15:34:30 YarnScheduler: INFO: Adding task set 0.0 with 1 tasks resource profile 0
2021-07-23 15:34:30 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it
2021-07-23 15:34:30 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0.
2021-07-23 15:34:30 FairSchedulableBuilder: INFO: Added task set TaskSet_0.0 tasks to pool default
2021-07-23 15:34:30 TaskSetManager: INFO: Starting task 0.0 in stage 0.0 (TID 0) (kc-w-23.c.broad-mpg-gnomad.internal, executor 1, partition 0, PROCESS_LOCAL, 4317 bytes) taskResourceAssignments Map()
2021-07-23 15:34:30 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-w-23.c.broad-mpg-gnomad.internal:38567 (size: 10.9 KiB, free: 11.2 GiB)
2021-07-23 15:34:31 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc-w-23.c.broad-mpg-gnomad.internal:38567 (size: 33.3 KiB, free: 11.2 GiB)
2021-07-23 15:34:32 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on kc-w-23.c.broad-mpg-gnomad.internal:38567 (size: 3.2 KiB, free: 11.2 GiB)
2021-07-23 15:34:42 TaskSetManager: INFO: Finished task 0.0 in stage 0.0 (TID 0) in 12054 ms on kc-w-23.c.broad-mpg-gnomad.internal (executor 1) (1/1)
2021-07-23 15:34:42 YarnScheduler: INFO: Removed TaskSet 0.0, whose tasks have all completed, from pool default
2021-07-23 15:34:42 DAGScheduler: INFO: ResultStage 0 (runJob at ContextRDD.scala:238) finished in 12.137 s
2021-07-23 15:34:42 DAGScheduler: INFO: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2021-07-23 15:34:42 YarnScheduler: INFO: Killing all running tasks in stage 0: Stage finished
2021-07-23 15:34:42 DAGScheduler: INFO: Job 0 finished: runJob at ContextRDD.scala:238, took 12.190178 s
2021-07-23 15:34:42 root: INFO: took 13.289s
2021-07-23 15:34:42 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: 
(Literal
  Array[Interval[Struct{window_end:Locus(GRCh37)}]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1:
(Literal
  Array[Interval[Struct{window_end:Locus(GRCh37)}]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: 
(MakeTuple (0)
  (Literal
    Array[Interval[Struct{window_end:Locus(GRCh37)}]]
    <literal value>))
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1:
(Literal
  Tuple[Array[Interval[Struct{window_end:Locus(GRCh37)}]]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: 
(Literal
  Tuple[Array[Interval[Struct{window_end:Locus(GRCh37)}]]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1:
(Literal
  Tuple[Array[Interval[Struct{window_end:Locus(GRCh37)}]]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: 
(Literal
  Tuple[Array[Interval[Struct{window_end:Locus(GRCh37)}]]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1:
(Literal
  Tuple[Array[Interval[Struct{window_end:Locus(GRCh37)}]]]
  <literal value>)
2021-07-23 15:34:42 root: INFO: encoder cache miss (0 hits, 1 misses, 0.000)
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C27FSContainer.<init>
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C27FSContainer.<clinit>
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C28etypeEncode.<init>
2021-07-23 15:34:42 root: INFO: instruction count: 5: __C28etypeEncode.apply
2021-07-23 15:34:42 root: INFO: instruction count: 9: __C28etypeEncode.__m29ENCODE_o_tuple_of_r_tuple_of_r_array_of_r_interval_of_r_struct_of_o_locusENDENDEND_TO_r_struct_of_r_struct_of_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolENDENDEND
2021-07-23 15:34:42 root: INFO: instruction count: 13: __C28etypeEncode.__m30ENCODE_o_tuple_of_r_array_of_r_interval_of_r_struct_of_o_locusENDEND_TO_r_struct_of_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolENDEND
2021-07-23 15:34:42 root: INFO: instruction count: 45: __C28etypeEncode.__m31ENCODE_o_array_of_r_interval_of_r_struct_of_o_locusEND_TO_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolEND
2021-07-23 15:34:42 root: INFO: instruction count: 51: __C28etypeEncode.__m32ENCODE_o_interval_of_r_struct_of_o_locusEND_TO_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolEND
2021-07-23 15:34:42 root: INFO: instruction count: 30: __C28etypeEncode.__m33ENCODE_o_struct_of_o_locusEND_TO_r_struct_of_o_struct_of_r_binaryANDr_int32ENDEND
2021-07-23 15:34:42 root: INFO: instruction count: 39: __C28etypeEncode.__m34ENCODE_o_locus_TO_o_struct_of_r_binaryANDr_int32END
2021-07-23 15:34:42 root: INFO: instruction count: 18: __C28etypeEncode.__m35ENCODE_o_string_TO_r_binary
2021-07-23 15:34:42 root: INFO: instruction count: 6: __C28etypeEncode.__m36ENCODE_o_int32_TO_r_int32
2021-07-23 15:34:42 root: INFO: instruction count: 6: __C28etypeEncode.__m37ENCODE_o_bool_TO_r_bool
2021-07-23 15:34:42 MemoryStore: INFO: Block broadcast_3 stored as values in memory (estimated size 120.0 B, free 21.7 GiB)
2021-07-23 15:34:42 MemoryStore: INFO: Block broadcast_3_piece0 stored as bytes in memory (estimated size 74.0 B, free 21.7 GiB)
2021-07-23 15:34:42 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:46573 (size: 74.0 B, free: 21.7 GiB)
2021-07-23 15:34:42 SparkContext: INFO: Created broadcast 3 from broadcast at SparkBackend.scala:278
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C12FSContainer.<init>
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C12FSContainer.<clinit>
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C13Compiled.<init>
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C13Compiled.apply
2021-07-23 15:34:42 root: INFO: instruction count: 9: __C13Compiled.setPartitionIndex
2021-07-23 15:34:42 root: INFO: instruction count: 4: __C13Compiled.addPartitionRegion
2021-07-23 15:34:42 root: INFO: instruction count: 4: __C13Compiled.setPool
2021-07-23 15:34:42 root: INFO: instruction count: 3: __C13Compiled.addFS
2021-07-23 15:34:42 root: INFO: instruction count: 32: __C13Compiled.addLiterals
2021-07-23 15:34:42 root: INFO: instruction count: 23: __C13Compiled.__m18DECODE_r_struct_of_r_struct_of_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolENDENDEND_TO_o_tuple_of_r_tuple_of_r_array_of_r_interval_of_r_struct_of_o_locusENDENDEND
2021-07-23 15:34:42 root: INFO: instruction count: 15: __C13Compiled.__m19INPLACE_DECODE_r_struct_of_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolENDEND_TO_r_tuple_of_r_array_of_r_interval_of_r_struct_of_o_locusENDEND
2021-07-23 15:34:42 root: INFO: instruction count: 64: __C13Compiled.__m20INPLACE_DECODE_r_array_of_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolEND_TO_r_array_of_r_interval_of_r_struct_of_o_locusEND
2021-07-23 15:34:42 root: INFO: instruction count: 36: __C13Compiled.__m21INPLACE_DECODE_r_struct_of_r_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_struct_of_o_struct_of_r_binaryANDr_int32ENDENDANDr_boolANDr_boolEND_TO_r_interval_of_r_struct_of_o_locusEND
2021-07-23 15:34:42 root: INFO: instruction count: 46: __C13Compiled.__m22INPLACE_DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDEND_TO_r_struct_of_o_locusEND
2021-07-23 15:34:42 root: INFO: instruction count: 22: __C13Compiled.__m23INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus
2021-07-23 15:34:42 root: INFO: instruction count: 33: __C13Compiled.__m24INPLACE_DECODE_r_binary_TO_r_string
2021-07-23 15:34:42 root: INFO: instruction count: 10: __C13Compiled.__m25INPLACE_DECODE_r_int32_TO_r_int32
2021-07-23 15:34:42 root: INFO: instruction count: 10: __C13Compiled.__m26INPLACE_DECODE_r_bool_TO_r_bool
2021-07-23 15:34:42 root: INFO: finished execution of query hail_query_1
2021-07-23 15:34:42 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 18: Thread-5
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON total 13.981s self 94.329ms children 13.886s %children 99.33%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 223.216ms self 1.353ms children 221.863ms %children 99.39%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 1.065ms self 1.065ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 220.768ms self 151.161ms children 69.607ms %children 31.53%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 69.607ms self 2.459ms children 67.148ms %children 96.47%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 7.186ms self 7.186ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 8.703ms self 8.703ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 21.423ms self 21.423ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 19.324ms self 19.324ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 2.652ms self 2.652ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 7.859ms self 7.859ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 11.825ms self 0.023ms children 11.802ms %children 99.81%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 11.743ms self 11.743ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.046ms self 0.046ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 4.823ms self 0.023ms children 4.800ms %children 99.51%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 4.747ms self 2.901ms children 1.846ms %children 38.89%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 1.846ms self 0.103ms children 1.743ms %children 94.40%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.127ms self 0.127ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.088ms self 0.088ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.078ms self 0.078ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.568ms self 0.568ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.047ms self 0.047ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.834ms self 0.834ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.038ms self 0.038ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 13.294s self 0.025ms children 13.294s %children 100.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.028ms self 0.028ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 13.293s self 13.293s children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.287ms self 0.287ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 3.374ms self 0.018ms children 3.356ms %children 99.48%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 3.329ms self 2.148ms children 1.181ms %children 35.48%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 1.181ms self 0.124ms children 1.057ms %children 89.46%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.117ms self 0.117ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.024ms self 0.024ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.228ms self 0.228ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.609ms self 0.609ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.021ms self 0.021ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile total 335.491ms self 211.899ms children 123.592ms %children 36.84%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR total 93.795ms self 0.018ms children 93.777ms %children 99.98%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 93.745ms self 89.944ms children 3.801ms %children 4.05%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 3.801ms self 0.053ms children 3.748ms %children 98.62%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 3.070ms self 3.070ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.016ms self 0.016ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.037ms self 0.037ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.229ms self 0.229ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.039ms self 0.039ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.070ms self 0.070ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.054ms self 0.054ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.022ms self 0.022ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.127ms self 0.127ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.024ms self 0.024ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.050ms self 0.050ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR total 0.496ms self 0.011ms children 0.485ms %children 97.86%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/LoweringTransformation total 0.457ms self 0.457ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR total 1.013ms self 0.011ms children 1.002ms %children 98.88%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.987ms self 0.641ms children 0.346ms %children 35.08%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.346ms self 0.024ms children 0.322ms %children 93.08%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.057ms self 0.057ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.026ms self 0.026ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.147ms self 0.147ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.054ms self 0.054ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs total 22.341ms self 0.017ms children 22.324ms %children 99.92%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 22.032ms self 22.032ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/Verify total 0.283ms self 0.283ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 1.373ms self 0.013ms children 1.360ms %children 99.08%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 1.344ms self 0.828ms children 0.516ms %children 38.42%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.516ms self 0.027ms children 0.490ms %children 94.80%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.092ms self 0.092ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.021ms self 0.021ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.229ms self 0.229ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.035ms self 0.035ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.070ms self 0.070ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/Compile/EmitContext.analyze total 4.575ms self 4.575ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 9.588ms self 9.588ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.008ms self 0.008ms children 0.000ms %children 0.00%
2021-07-23 15:34:42 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 4.261ms self 4.261ms children 0.000ms %children 0.00%
2021-07-23 15:34:43 SparkContext: INFO: Invoking stop() from shutdown hook
2021-07-23 15:34:43 AbstractConnector: INFO: Stopped Spark@2d4d2761{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
2021-07-23 15:34:43 SparkUI: INFO: Stopped Spark web UI at http://kc-m.c.broad-mpg-gnomad.internal:33677
2021-07-23 15:34:43 YarnClientSchedulerBackend: INFO: Interrupting monitor thread
2021-07-23 15:34:43 YarnClientSchedulerBackend: INFO: Shutting down all executors
2021-07-23 15:34:43 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Asking each executor to shut down
2021-07-23 15:34:43 ApplicationMaster: INFO: Final app status: SUCCEEDED, exitCode: 0
2021-07-23 15:34:43 ApplicationMaster: INFO: Unregistering ApplicationMaster with SUCCEEDED
2021-07-23 15:34:43 ApplicationMaster: INFO: Deleting staging directory hdfs://kc-m/user/root/.sparkStaging/application_1627046056954_0007
2021-07-23 15:34:43 YarnClientSchedulerBackend: INFO: YARN client scheduler backend Stopped
2021-07-23 15:34:43 MapOutputTrackerMasterEndpoint: INFO: MapOutputTrackerMasterEndpoint stopped!
2021-07-23 15:34:43 MemoryStore: INFO: MemoryStore cleared
2021-07-23 15:34:43 BlockManager: INFO: BlockManager stopped
2021-07-23 15:34:43 BlockManagerMaster: INFO: BlockManagerMaster stopped
2021-07-23 15:34:43 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: INFO: OutputCommitCoordinator stopped!
2021-07-23 15:34:43 SparkContext: INFO: Successfully stopped SparkContext
2021-07-23 15:34:43 ShutdownHookManager: INFO: Shutdown hook called
2021-07-23 15:34:43 ShutdownHookManager: INFO: Deleting directory /hadoop/spark/tmp/spark-54eff7f2-63b1-4b34-9777-c04c9477dc7b
2021-07-23 15:34:43 ShutdownHookManager: INFO: Deleting directory /hadoop/spark/tmp/spark-54eff7f2-63b1-4b34-9777-c04c9477dc7b/pyspark-04f88de5-e873-4ca9-9395-6d6088a175c4
2021-07-23 15:34:43 ShutdownHookManager: INFO: Deleting directory /tmp/spark-ef5b77da-e1bb-48c9-af68-df89844d1d3b