2022-03-09 16:30:35 Hail: INFO: Running Hail version 0.2.89-38264124ad91 2022-03-09 16:30:36 SparkContext: WARN: Using an existing SparkContext; some configuration may not take effect. 2022-03-09 16:30:36 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:37 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.78:37152) with ID 1, ResourceProfileId 0 2022-03-09 16:30:37 ExecutorMonitor: INFO: New executor 1 has registered (new total is 1) 2022-03-09 16:30:37 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.79:48968) with ID 2, ResourceProfileId 0 2022-03-09 16:30:37 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-0.c.broad-mpg-gnomad.internal:41631 with 2.9 GiB RAM, BlockManagerId(1, kc-w-0.c.broad-mpg-gnomad.internal, 41631, None) 2022-03-09 16:30:37 ExecutorMonitor: INFO: New executor 2 has registered (new total is 2) 2022-03-09 16:30:37 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-1.c.broad-mpg-gnomad.internal:44485 with 2.9 GiB RAM, BlockManagerId(2, kc-w-1.c.broad-mpg-gnomad.internal, 44485, None) 2022-03-09 16:30:38 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:38 root: INFO: timing SparkBackend.parse_table_ir total 1.363s self 1.363s children 0.000ms %children 0.00% 2022-03-09 16:30:38 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:38 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:38 root: INFO: timing SparkBackend.parse_value_ir total 460.398ms self 460.398ms children 0.000ms %children 0.00% 2022-03-09 16:30:38 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:38 root: INFO: starting execution of query hail_query_1 of initial size 103 2022-03-09 16:30:39 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 103: (TableCount (TableMapRows (TableMapRows (TableFilter (TableKeyBy (transcript i j) True (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableFilter (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht )) (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len) (Ref row)) None (start_idx (ToArray (StreamFlatMap (ToStream False (ToArray (StreamRange 1 False (I32 0) (Ref __cse_1) (I32 500)))) (ToStream False (ToArray (StreamMap (ToStream False (ToArray (StreamRange 3 False (I32 0) (Ref __cse_1) (I32 500)))) (MakeStruct (i_start (Ref __uid_5)) (j_start (Ref __uid_6))))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx) (Ref row)) None (i (GetField i_start (Ref __uid_7))) (j (GetField j_start (Ref __uid_7))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j) (Ref row)) None (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j i_max_idx j_max_idx) (Ref row)) None (start_idx (InsertFields (Ref __cse_3) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __cse_3)) (GetField j_start (Ref __cse_3))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))) 2022-03-09 16:30:39 root: INFO: generated TableFilterIntervals node with 1 intervals: Intervals: [[ENST00000217939]-[ENST00000217939]] Predicate: (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False)) Post: (Coalesce (True) (False)) 2022-03-09 16:30:39 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_27) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_27) (I32 500)) (MakeStruct (i_start (Ref __iruid_28)) (j_start (Ref __iruid_29))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_31))) (j (GetField j_start (Ref __iruid_31)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:39 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_27) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_27) (I32 500)) (MakeStruct (i_start (Ref __iruid_28)) (j_start (Ref __iruid_29))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_31))) (j (GetField j_start (Ref __iruid_31)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:39 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_54) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_54) (I32 500)) (MakeStruct (i_start (Ref __iruid_55)) (j_start (Ref __iruid_56))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_58))) (j (GetField j_start (Ref __iruid_58)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:39 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:39 root: INFO: lowering result: TableCount 2022-03-09 16:30:39 root: INFO: compiling and evaluating result: TableCount 2022-03-09 16:30:39 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 77: (ApplyIR -1 sum () Int64 (Let (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0)) (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (MakeStruct (__iruid_63 (Ref __iruid_63))) (Let (GetField __iruid_63 (Ref __iruid_70)) (Cast Int64 (StreamLen (Let (Ref __iruid_63) (StreamFilter (Let (Ref __iruid_63) (StreamMap (StreamFlatMap (Let (Ref __iruid_63) (StreamMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_64)) (Let (Ref __iruid_65) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_54) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_54) (I32 500)) (MakeStruct (i_start (Ref __iruid_55)) (j_start (Ref __iruid_56)))))))))))) (Let (GetField start_idx (Ref __iruid_66)) (StreamMap (ToStream False (Ref __iruid_67)) (InsertFields (Ref __iruid_66) None (start_idx (Ref __iruid_68)))))) (Let (Ref __iruid_69) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_58))) (j (GetField j_start (Ref __iruid_58)))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))))))))) 2022-03-09 16:30:40 root: INFO: Prune: MakeStruct: eliminating field '__iruid_63' 2022-03-09 16:30:40 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_122)) (Let (GetField list_len (Ref __iruid_125)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_126) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_126) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_127)) (j_start (Ref __iruid_128))) (InsertFields (Ref __iruid_125) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_129)) (i (GetField i_start (Ref __iruid_129))) (j (GetField j_start (Ref __iruid_129))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_124)) (GetField i (Ref __iruid_124))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_120) (Ref __iruid_121))) 2022-03-09 16:30:40 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_122)) (Let (GetField list_len (Ref __iruid_125)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_126) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_126) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_127)) (j_start (Ref __iruid_128))) (InsertFields (Ref __iruid_125) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_129)) (i (GetField i_start (Ref __iruid_129))) (j (GetField j_start (Ref __iruid_129))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_124)) (GetField i (Ref __iruid_124))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_120) (Ref __iruid_121))) 2022-03-09 16:30:40 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_152)) (Let (GetField list_len (Ref __iruid_155)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_156) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_156) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_157)) (j_start (Ref __iruid_158))) (InsertFields (Ref __iruid_155) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_159)) (i (GetField i_start (Ref __iruid_159))) (j (GetField j_start (Ref __iruid_159))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_154)) (GetField i (Ref __iruid_154))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_150) (Ref __iruid_151))) 2022-03-09 16:30:40 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_152)) (Let (GetField list_len (Ref __iruid_155)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_156) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_156) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_157)) (j_start (Ref __iruid_158))) (InsertFields (Ref __iruid_155) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_159)) (i (GetField i_start (Ref __iruid_159))) (j (GetField j_start (Ref __iruid_159))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_154)) (GetField i (Ref __iruid_154))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_150) (Ref __iruid_151))) 2022-03-09 16:30:40 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_182)) (Let (GetField list_len (Ref __iruid_185)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_186) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_186) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_187)) (j_start (Ref __iruid_188))) (InsertFields (Ref __iruid_185) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_189)) (i (GetField i_start (Ref __iruid_189))) (j (GetField j_start (Ref __iruid_189))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_184)) (GetField i (Ref __iruid_184))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_180) (Ref __iruid_181))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_182)) (Let (GetField list_len (Ref __iruid_185)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_186) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_186) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_187)) (j_start (Ref __iruid_188))) (InsertFields (Ref __iruid_185) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_189)) (i (GetField i_start (Ref __iruid_189))) (j (GetField j_start (Ref __iruid_189))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_184)) (GetField i (Ref __iruid_184))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_180) (Ref __iruid_181)))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_212)) (Let (GetField list_len (Ref __iruid_215)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_216) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_216) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_217)) (j_start (Ref __iruid_218))) (InsertFields (Ref __iruid_215) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_219)) (i (GetField i_start (Ref __iruid_219))) (j (GetField j_start (Ref __iruid_219))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_214)) (GetField i (Ref __iruid_214))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_210) (Ref __iruid_211)))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_212)) (Let (GetField list_len (Ref __iruid_215)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_216) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_216) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_217)) (j_start (Ref __iruid_218))) (InsertFields (Ref __iruid_215) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_219)) (i (GetField i_start (Ref __iruid_219))) (j (GetField j_start (Ref __iruid_219))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_214)) (GetField i (Ref __iruid_214))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_210) (Ref __iruid_211)))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_242)) (Let (GetField list_len (Ref __iruid_245)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_246) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_246) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_247)) (j_start (Ref __iruid_248))) (InsertFields (Ref __iruid_245) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_249)) (i (GetField i_start (Ref __iruid_249))) (j (GetField j_start (Ref __iruid_249))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_244)) (GetField i (Ref __iruid_244))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_240) (Ref __iruid_241)))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_242)) (Let (GetField list_len (Ref __iruid_245)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_246) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_246) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_247)) (j_start (Ref __iruid_248))) (InsertFields (Ref __iruid_245) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_249)) (i (GetField i_start (Ref __iruid_249))) (j (GetField j_start (Ref __iruid_249))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_244)) (GetField i (Ref __iruid_244))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_240) (Ref __iruid_241)))) 2022-03-09 16:30:40 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_272)) (Let (GetField list_len (Ref __iruid_275)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_276) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_276) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_277)) (j_start (Ref __iruid_278))) (InsertFields (Ref __iruid_275) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_279)) (i (GetField i_start (Ref __iruid_279))) (j (GetField j_start (Ref __iruid_279))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_274)) (GetField i (Ref __iruid_274))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_270) (Ref __iruid_271)))) 2022-03-09 16:30:40 root: INFO: decoder cache miss (0 hits, 1 misses, 0.000 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C32HailClassLoaderContainer. 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C32HailClassLoaderContainer. 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C33FSContainer. 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C33FSContainer. 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C34etypeDecode. 2022-03-09 16:30:40 root: INFO: instruction count: 7: __C34etypeDecode.apply 2022-03-09 16:30:40 root: INFO: instruction count: 122: __C34etypeDecode.__m35DECODE_r_struct_of_o_binaryANDo_float64ANDr_int32ANDo_int32ANDo_int32ANDr_array_of_o_int64ANDr_array_of_o_float64ANDr_array_of_o_int32ANDr_int32END_TO_SBaseStructPointer 2022-03-09 16:30:40 root: INFO: instruction count: 31: __C34etypeDecode.__m36INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C34etypeDecode.__m37SKIP_o_float64 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C34etypeDecode.__m38SKIP_r_int32 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C34etypeDecode.__m39SKIP_o_int32 2022-03-09 16:30:40 root: INFO: instruction count: 51: __C34etypeDecode.__m40SKIP_r_array_of_o_int64 2022-03-09 16:30:40 root: INFO: instruction count: 3: __C34etypeDecode.__m41SKIP_o_int64 2022-03-09 16:30:40 root: INFO: instruction count: 51: __C34etypeDecode.__m42SKIP_r_array_of_o_float64 2022-03-09 16:30:40 root: INFO: instruction count: 51: __C34etypeDecode.__m43SKIP_r_array_of_o_int32 2022-03-09 16:30:40 root: INFO: instruction count: 10: __C34etypeDecode.__m44INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:40 root: INFO: decoder cache miss (0 hits, 2 misses, 0.000 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C45HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C45HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C46FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C46FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C47etypeDecode. 2022-03-09 16:30:41 root: INFO: instruction count: 7: __C47etypeDecode.apply 2022-03-09 16:30:41 root: INFO: instruction count: 36: __C47etypeDecode.__m48DECODE_o_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C47etypeDecode.__m49INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:41 root: INFO: instruction count: 58: __C47etypeDecode.__m50INPLACE_DECODE_r_array_of_r_struct_of_r_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_array_of_r_struct_of_r_struct_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 35: __C47etypeDecode.__m51INPLACE_DECODE_r_struct_of_r_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_struct_of_r_struct_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 48: __C47etypeDecode.__m52INPLACE_DECODE_r_struct_of_o_binaryEND_TO_r_struct_of_o_stringEND 2022-03-09 16:30:41 root: INFO: instruction count: 31: __C47etypeDecode.__m53INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:41 root: INFO: instruction count: 8: __C47etypeDecode.__m54INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: decoder cache miss (0 hits, 3 misses, 0.000 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C55HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C55HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C56FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C56FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C57etypeDecode. 2022-03-09 16:30:41 root: INFO: instruction count: 7: __C57etypeDecode.apply 2022-03-09 16:30:41 root: INFO: instruction count: 27: __C57etypeDecode.__m58DECODE_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 58: __C57etypeDecode.__m59INPLACE_DECODE_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 53: __C57etypeDecode.__m60INPLACE_DECODE_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C57etypeDecode.__m61INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:41 root: INFO: instruction count: 48: __C57etypeDecode.__m62INPLACE_DECODE_r_struct_of_o_binaryEND_TO_r_struct_of_o_stringEND 2022-03-09 16:30:41 root: INFO: instruction count: 31: __C57etypeDecode.__m63INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:41 root: INFO: instruction count: 8: __C57etypeDecode.__m64INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C15HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C15HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C16FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C16FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C17collect_distributed_array. 2022-03-09 16:30:41 root: INFO: instruction count: 114: __C17collect_distributed_array.apply 2022-03-09 16:30:41 root: INFO: instruction count: 17: __C17collect_distributed_array.apply 2022-03-09 16:30:41 root: INFO: instruction count: 27: __C17collect_distributed_array.__m18DECODE_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 35: __C17collect_distributed_array.__m19INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 31: __C17collect_distributed_array.__m20INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:41 root: INFO: instruction count: 44: __C17collect_distributed_array.__m21INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 26: __C17collect_distributed_array.__m22INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 17: __C17collect_distributed_array.__m23INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C17collect_distributed_array.__m24INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C17collect_distributed_array.__m25INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:41 root: INFO: instruction count: 27: __C17collect_distributed_array.__m27DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 8: __C17collect_distributed_array.__m28INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: instruction count: 280: __C17collect_distributed_array.__m30split_StreamLen 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C17collect_distributed_array.setObjects 2022-03-09 16:30:41 root: INFO: instruction count: 11: __C17collect_distributed_array.__m95ord_gteq 2022-03-09 16:30:41 root: INFO: instruction count: 14: __C17collect_distributed_array.__m96ord_gteqNonnull 2022-03-09 16:30:41 root: INFO: instruction count: 12: __C17collect_distributed_array.__m100setup_null 2022-03-09 16:30:41 root: INFO: instruction count: 13: __C17collect_distributed_array.__m102ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64END 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C17collect_distributed_array.__m103ENCODE_SInt64$_TO_r_int64 2022-03-09 16:30:41 root: INFO: instruction count: 9: __C17collect_distributed_array.setPartitionIndex 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C17collect_distributed_array.addPartitionRegion 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C17collect_distributed_array.setPool 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C17collect_distributed_array.addHailClassLoader 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C17collect_distributed_array.addFS 2022-03-09 16:30:41 root: INFO: instruction count: 366: __C17collect_distributed_array.__m30split_StreamLen_region0_2 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C106__m30split_StreamLenSpills. 2022-03-09 16:30:41 root: INFO: encoder cache miss (0 hits, 1 misses, 0.000) 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C147HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C147HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C148FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C148FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C149etypeEncode. 2022-03-09 16:30:41 root: INFO: instruction count: 7: __C149etypeEncode.apply 2022-03-09 16:30:41 root: INFO: instruction count: 33: __C149etypeEncode.__m150ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDANDr_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 1: __C149etypeEncode.__m151ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: instruction count: 35: __C149etypeEncode.__m152ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 51: __C149etypeEncode.__m153ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 16: __C149etypeEncode.__m154ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:41 root: INFO: instruction count: 41: __C149etypeEncode.__m155ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:41 root: INFO: instruction count: 21: __C149etypeEncode.__m156ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 13: __C149etypeEncode.__m157ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C149etypeEncode.__m158ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C149etypeEncode.__m159ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 336.0 B, free 12.6 GiB) 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 209.0 B, free 12.6 GiB) 2022-03-09 16:30:41 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 209.0 B, free: 12.6 GiB) 2022-03-09 16:30:41 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C1HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C1HailClassLoaderContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C2FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C2FSContainer. 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C3Compiled. 2022-03-09 16:30:41 root: INFO: instruction count: 25: __C3Compiled.apply 2022-03-09 16:30:41 root: INFO: instruction count: 326: __C3Compiled.__m4split_StreamFold 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.setBackend 2022-03-09 16:30:41 root: INFO: instruction count: 9: __C3Compiled.__m113ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 51: __C3Compiled.__m114ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 16: __C3Compiled.__m115ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:41 root: INFO: instruction count: 41: __C3Compiled.__m116ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:41 root: INFO: instruction count: 21: __C3Compiled.__m117ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 13: __C3Compiled.__m118ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.__m119ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.__m120ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:41 root: INFO: instruction count: 9: __C3Compiled.__m121ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 1: __C3Compiled.__m122ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.setObjects 2022-03-09 16:30:41 root: INFO: instruction count: 27: __C3Compiled.__m124DECODE_r_struct_of_r_int64END_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C3Compiled.__m125INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:41 root: INFO: instruction count: 9: __C3Compiled.setPartitionIndex 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.addPartitionRegion 2022-03-09 16:30:41 root: INFO: instruction count: 4: __C3Compiled.setPool 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C3Compiled.addHailClassLoader 2022-03-09 16:30:41 root: INFO: instruction count: 3: __C3Compiled.addFS 2022-03-09 16:30:41 root: INFO: instruction count: 62: __C3Compiled.addLiterals 2022-03-09 16:30:41 root: INFO: instruction count: 36: __C3Compiled.__m137DECODE_r_struct_of_r_struct_of_ENDANDr_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:41 root: INFO: instruction count: 8: __C3Compiled.__m138INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:41 root: INFO: instruction count: 58: __C3Compiled.__m139INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 35: __C3Compiled.__m140INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:41 root: INFO: instruction count: 31: __C3Compiled.__m141INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:41 root: INFO: instruction count: 44: __C3Compiled.__m142INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 26: __C3Compiled.__m143INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:41 root: INFO: instruction count: 17: __C3Compiled.__m144INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C3Compiled.__m145INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:41 root: INFO: instruction count: 10: __C3Compiled.__m146INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_1 stored as values in memory (estimated size 48.0 B, free 12.6 GiB) 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_1_piece0 stored as bytes in memory (estimated size 49.0 B, free 12.6 GiB) 2022-03-09 16:30:41 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:30:41 SparkContext: INFO: Created broadcast 1 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_2 stored as values in memory (estimated size 339.8 KiB, free 12.6 GiB) 2022-03-09 16:30:41 MemoryStore: INFO: Block broadcast_2_piece0 stored as bytes in memory (estimated size 33.4 KiB, free 12.6 GiB) 2022-03-09 16:30:41 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 33.4 KiB, free: 12.6 GiB) 2022-03-09 16:30:41 SparkContext: INFO: Created broadcast 2 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:42 SparkContext: INFO: Starting job: collect at SparkBackend.scala:288 2022-03-09 16:30:42 DAGScheduler: INFO: Got job 0 (collect at SparkBackend.scala:288) with 1 output partitions 2022-03-09 16:30:42 DAGScheduler: INFO: Final stage: ResultStage 0 (collect at SparkBackend.scala:288) 2022-03-09 16:30:42 DAGScheduler: INFO: Parents of final stage: List() 2022-03-09 16:30:42 DAGScheduler: INFO: Missing parents: List() 2022-03-09 16:30:42 DAGScheduler: INFO: Submitting ResultStage 0 (SparkBackendComputeRDD[0] at RDD at SparkBackend.scala:720), which has no missing parents 2022-03-09 16:30:42 MemoryStore: INFO: Block broadcast_3 stored as values in memory (estimated size 41.5 KiB, free 12.6 GiB) 2022-03-09 16:30:42 MemoryStore: INFO: Block broadcast_3_piece0 stored as bytes in memory (estimated size 21.1 KiB, free 12.6 GiB) 2022-03-09 16:30:42 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 21.1 KiB, free: 12.6 GiB) 2022-03-09 16:30:42 SparkContext: INFO: Created broadcast 3 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:30:42 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 0 (SparkBackendComputeRDD[0] at RDD at SparkBackend.scala:720) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:30:42 YarnScheduler: INFO: Adding task set 0.0 with 1 tasks resource profile 0 2022-03-09 16:30:42 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:30:42 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0. 2022-03-09 16:30:42 FairSchedulableBuilder: INFO: Added task set TaskSet_0.0 tasks to pool default 2022-03-09 16:30:42 TaskSetManager: INFO: Starting task 0.0 in stage 0.0 (TID 0) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, PROCESS_LOCAL, 4563 bytes) taskResourceAssignments Map() 2022-03-09 16:30:42 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 21.1 KiB, free: 2.9 GiB) 2022-03-09 16:30:43 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 33.4 KiB, free: 2.9 GiB) 2022-03-09 16:30:44 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:30:46 TaskSetManager: INFO: Finished task 0.0 in stage 0.0 (TID 0) in 3707 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/1) 2022-03-09 16:30:46 YarnScheduler: INFO: Removed TaskSet 0.0, whose tasks have all completed, from pool default 2022-03-09 16:30:46 DAGScheduler: INFO: ResultStage 0 (collect at SparkBackend.scala:288) finished in 3.806 s 2022-03-09 16:30:46 DAGScheduler: INFO: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:30:46 YarnScheduler: INFO: Killing all running tasks in stage 0: Stage finished 2022-03-09 16:30:46 DAGScheduler: INFO: Job 0 finished: collect at SparkBackend.scala:288, took 3.907450 s 2022-03-09 16:30:46 root: INFO: took 6.738s 2022-03-09 16:30:46 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:46 root: INFO: lowering result: I64 2022-03-09 16:30:46 root: INFO: compiling and evaluating result: I64 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (I64 120)) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1: (Literal Tuple[Int64] ) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: (Literal Tuple[Int64] ) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1: (Literal Tuple[Int64] ) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: (Literal Tuple[Int64] ) 2022-03-09 16:30:46 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1: (Literal Tuple[Int64] ) 2022-03-09 16:30:46 root: INFO: encoder cache miss (0 hits, 2 misses, 0.000) 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C170HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C170HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C171FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C171FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C172etypeEncode. 2022-03-09 16:30:46 root: INFO: instruction count: 7: __C172etypeEncode.apply 2022-03-09 16:30:46 root: INFO: instruction count: 9: __C172etypeEncode.__m173ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_int64ENDEND 2022-03-09 16:30:46 root: INFO: instruction count: 13: __C172etypeEncode.__m174ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64END 2022-03-09 16:30:46 root: INFO: instruction count: 4: __C172etypeEncode.__m175ENCODE_SInt64$_TO_r_int64 2022-03-09 16:30:46 MemoryStore: INFO: Block broadcast_4 stored as values in memory (estimated size 80.0 B, free 12.6 GiB) 2022-03-09 16:30:46 MemoryStore: INFO: Block broadcast_4_piece0 stored as bytes in memory (estimated size 65.0 B, free 12.6 GiB) 2022-03-09 16:30:46 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 65.0 B, free: 12.6 GiB) 2022-03-09 16:30:46 SparkContext: INFO: Created broadcast 4 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C160HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C160HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C161FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C161FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C162Compiled. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C162Compiled.apply 2022-03-09 16:30:46 root: INFO: instruction count: 9: __C162Compiled.setPartitionIndex 2022-03-09 16:30:46 root: INFO: instruction count: 4: __C162Compiled.addPartitionRegion 2022-03-09 16:30:46 root: INFO: instruction count: 4: __C162Compiled.setPool 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C162Compiled.addHailClassLoader 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C162Compiled.addFS 2022-03-09 16:30:46 root: INFO: instruction count: 35: __C162Compiled.addLiterals 2022-03-09 16:30:46 root: INFO: instruction count: 27: __C162Compiled.__m167DECODE_r_struct_of_r_struct_of_r_int64ENDEND_TO_SBaseStructPointer 2022-03-09 16:30:46 root: INFO: instruction count: 17: __C162Compiled.__m168INPLACE_DECODE_r_struct_of_r_int64END_TO_r_tuple_of_r_int64END 2022-03-09 16:30:46 root: INFO: instruction count: 10: __C162Compiled.__m169INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:46 root: INFO: took 39.350ms 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I64 120) 2022-03-09 16:30:46 root: INFO: encoder cache miss (0 hits, 3 misses, 0.000) 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C176HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C176HailClassLoaderContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C177FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C177FSContainer. 2022-03-09 16:30:46 root: INFO: instruction count: 3: __C178etypeEncode. 2022-03-09 16:30:46 root: INFO: instruction count: 11: __C178etypeEncode.apply 2022-03-09 16:30:46 root: INFO: instruction count: 4: __C178etypeEncode.__m179ENCODE_SInt64$_TO_o_int64 2022-03-09 16:30:46 root: INFO: finished execution of query hail_query_1 2022-03-09 16:30:46 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode total 7.403s self 100.427ms children 7.302s %children 98.64% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 425.161ms self 1.175ms children 423.986ms %children 99.72% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 1.218ms self 1.218ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 422.599ms self 206.678ms children 215.921ms %children 51.09% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 215.921ms self 2.735ms children 213.186ms %children 98.73% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 12.969ms self 12.969ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 24.694ms self 24.694ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 96.258ms self 96.258ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 9.946ms self 9.946ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 3.403ms self 3.403ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 34.163ms self 34.163ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.845ms self 0.845ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.775ms self 0.775ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.749ms self 0.749ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 7.157ms self 7.157ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.463ms self 0.463ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 5.226ms self 5.226ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.772ms self 0.772ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.672ms self 0.672ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.531ms self 0.531ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 6.091ms self 6.091ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.404ms self 0.404ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 8.067ms self 8.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 15.100ms self 0.026ms children 15.074ms %children 99.83% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.120ms self 0.120ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 14.765ms self 14.765ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.189ms self 0.189ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 73.324ms self 0.030ms children 73.294ms %children 99.96% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.135ms self 0.135ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 72.979ms self 11.591ms children 61.388ms %children 84.12% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 61.388ms self 0.243ms children 61.145ms %children 99.60% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.482ms self 0.482ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.008ms self 1.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.975ms self 0.975ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 8.608ms self 8.608ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.354ms self 0.354ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 5.890ms self 5.890ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.518ms self 0.518ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.711ms self 0.711ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 8.971ms self 8.971ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 8.700ms self 8.700ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.593ms self 0.593ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 5.580ms self 5.580ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.627ms self 0.627ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.651ms self 0.651ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.545ms self 0.545ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 8.478ms self 8.478ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.578ms self 0.578ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 7.876ms self 7.876ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 4.393ms self 0.029ms children 4.364ms %children 99.34% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.218ms self 0.218ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 3.948ms self 3.948ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.198ms self 0.198ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 6.783s self 0.029ms children 6.783s %children 100.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.160ms self 0.160ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 6.782s self 485.762ms children 6.297s %children 92.84% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 145.369ms self 0.028ms children 145.341ms %children 99.98% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.413ms self 0.413ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 144.751ms self 55.034ms children 89.717ms %children 61.98% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 89.717ms self 0.276ms children 89.441ms %children 99.69% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.604ms self 1.604ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.052ms self 1.052ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 5.329ms self 5.329ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 20.284ms self 20.284ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 4.086ms self 4.086ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 9.556ms self 9.556ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.906ms self 0.906ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.456ms self 0.456ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 10.934ms self 10.934ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 5.449ms self 5.449ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.465ms self 0.465ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.128ms self 4.128ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 13.506ms self 13.506ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.662ms self 0.662ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.506ms self 0.506ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 6.746ms self 6.746ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.340ms self 0.340ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.431ms self 3.431ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.362ms self 0.017ms children 0.345ms %children 95.33% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.213ms self 0.213ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 48.895ms self 0.023ms children 48.872ms %children 99.95% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 48.771ms self 11.480ms children 37.291ms %children 76.46% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 37.291ms self 0.183ms children 37.108ms %children 99.51% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.627ms self 0.627ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.271ms self 0.271ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.547ms self 0.547ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 10.207ms self 10.207ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.351ms self 0.351ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 3.869ms self 3.869ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.612ms self 0.612ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.212ms self 0.212ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.719ms self 0.719ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 6.635ms self 6.635ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.283ms self 0.283ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 3.769ms self 3.769ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.612ms self 0.612ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.237ms self 0.237ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.484ms self 0.484ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 3.814ms self 3.814ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.306ms self 0.306ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 3.553ms self 3.553ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.274ms self 0.015ms children 0.259ms %children 94.64% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.138ms self 0.138ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 2.148ms self 0.033ms children 2.114ms %children 98.45% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.256ms self 0.256ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.187ms self 0.187ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 1.672ms self 1.672ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 52.641ms self 0.029ms children 52.612ms %children 99.95% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 52.499ms self 16.956ms children 35.543ms %children 67.70% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 35.543ms self 0.308ms children 35.235ms %children 99.13% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.541ms self 0.541ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.268ms self 0.268ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.496ms self 0.496ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 3.699ms self 3.699ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.236ms self 0.236ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 3.944ms self 3.944ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.584ms self 0.584ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.385ms self 0.385ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.501ms self 0.501ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 6.773ms self 6.773ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.342ms self 0.342ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 6.372ms self 6.372ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.625ms self 0.625ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.259ms self 0.259ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.477ms self 0.477ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 4.673ms self 4.673ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.281ms self 0.281ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 4.777ms self 4.777ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 1.650s self 1.406s children 243.974ms %children 14.79% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 132.954ms self 0.029ms children 132.925ms %children 99.98% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.142ms self 0.142ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 132.735ms self 95.842ms children 36.893ms %children 27.79% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 36.893ms self 0.272ms children 36.621ms %children 99.26% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.348ms self 1.348ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.313ms self 0.313ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.895ms self 0.895ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.830ms self 3.830ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.294ms self 0.294ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 7.976ms self 7.976ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.566ms self 0.566ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.259ms self 0.259ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.684ms self 0.684ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 6.236ms self 6.236ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.329ms self 0.329ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.086ms self 3.086ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.597ms self 0.597ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.236ms self 0.236ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.471ms self 0.471ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 5.628ms self 5.628ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.356ms self 0.356ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.518ms self 3.518ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.884ms self 0.018ms children 0.865ms %children 97.93% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.173ms self 0.173ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.576ms self 0.576ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.116ms self 0.116ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 39.990ms self 0.021ms children 39.969ms %children 99.95% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 39.909ms self 9.666ms children 30.243ms %children 75.78% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 30.243ms self 0.182ms children 30.061ms %children 99.40% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.282ms self 0.282ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.234ms self 0.234ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.524ms self 0.524ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 3.000ms self 3.000ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.835ms self 2.835ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.508ms self 0.508ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.202ms self 0.202ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.476ms self 0.476ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 3.117ms self 3.117ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.455ms self 0.455ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.389ms self 2.389ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.358ms self 0.358ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.187ms self 0.187ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.468ms self 0.468ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 11.777ms self 11.777ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.299ms self 0.299ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.769ms self 2.769ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 19.831ms self 0.030ms children 19.802ms %children 99.85% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.166ms self 0.166ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 18.513ms self 18.513ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 1.122ms self 1.122ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 35.302ms self 0.025ms children 35.277ms %children 99.93% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 35.217ms self 9.885ms children 25.332ms %children 71.93% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 25.332ms self 0.205ms children 25.126ms %children 99.19% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.377ms self 0.377ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.254ms self 0.254ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.557ms self 0.557ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 2.888ms self 2.888ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.164ms self 0.164ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 2.267ms self 2.267ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.596ms self 0.596ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.194ms self 0.194ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.507ms self 0.507ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 3.858ms self 3.858ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.167ms self 0.167ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 2.283ms self 2.283ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.356ms self 0.356ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.598ms self 0.598ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 5.866ms self 5.866ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.423ms self 0.423ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 3.591ms self 3.591ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 15.013ms self 15.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 20.111ms self 20.111ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 4.339s self 4.339s children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 1.069ms self 0.015ms children 1.054ms %children 98.59% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 1.035ms self 0.411ms children 0.625ms %children 60.32% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.625ms self 0.025ms children 0.600ms %children 96.04% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.264ms self 0.264ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.172ms self 0.172ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.045ms self 0.008ms children 0.037ms %children 82.87% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.509ms self 0.009ms children 0.500ms %children 98.31% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.473ms self 0.184ms children 0.289ms %children 61.11% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.289ms self 0.019ms children 0.270ms %children 93.40% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.109ms self 0.109ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.027ms self 0.005ms children 0.022ms %children 81.01% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.026ms self 0.006ms children 0.019ms %children 75.18% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.450ms self 0.009ms children 0.442ms %children 98.05% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.437ms self 0.195ms children 0.242ms %children 55.30% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.242ms self 0.020ms children 0.222ms %children 91.88% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.109ms self 0.109ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 32.128ms self 27.735ms children 4.392ms %children 13.67% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 2.929ms self 0.013ms children 2.917ms %children 99.57% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 2.904ms self 0.490ms children 2.415ms %children 83.14% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 2.415ms self 0.050ms children 2.364ms %children 97.91% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.041ms self 1.041ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.106ms self 0.106ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.946ms self 0.946ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.033ms self 0.006ms children 0.027ms %children 83.01% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 0.458ms self 0.008ms children 0.450ms %children 98.16% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.445ms self 0.238ms children 0.206ms %children 46.41% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.206ms self 0.018ms children 0.188ms %children 91.29% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.288ms self 0.009ms children 0.279ms %children 96.79% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.263ms self 0.263ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.541ms self 0.008ms children 0.532ms %children 98.49% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.527ms self 0.258ms children 0.269ms %children 51.06% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.269ms self 0.021ms children 0.248ms %children 92.19% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 3.765ms self 3.765ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.985ms self 0.014ms children 0.971ms %children 98.60% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.962ms self 0.403ms children 0.559ms %children 58.12% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.559ms self 0.033ms children 0.526ms %children 94.08% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.207ms self 0.207ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.106ms self 0.106ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/Compile total 0.398ms self 0.398ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:46 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:46 root: INFO: timing SparkBackend.parse_value_ir total 32.150ms self 32.150ms children 0.000ms %children 0.00% 2022-03-09 16:30:46 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:46 root: INFO: starting execution of query hail_query_2 of initial size 103 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 103: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":null}" (TableMapRows (TableMapRows (TableFilter (TableKeyBy (transcript i j) True (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableFilter (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht )) (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len) (Ref row)) None (start_idx (ToArray (StreamFlatMap (ToStream False (ToArray (StreamRange 1 False (I32 0) (Ref __cse_1) (I32 500)))) (ToStream False (ToArray (StreamMap (ToStream False (ToArray (StreamRange 3 False (I32 0) (Ref __cse_1) (I32 500)))) (MakeStruct (i_start (Ref __uid_5)) (j_start (Ref __uid_6))))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx) (Ref row)) None (i (GetField i_start (Ref __uid_7))) (j (GetField j_start (Ref __uid_7))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j) (Ref row)) None (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j i_max_idx j_max_idx) (Ref row)) None (start_idx (InsertFields (Ref __cse_3) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __cse_3)) (GetField j_start (Ref __cse_3))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))) 2022-03-09 16:30:46 root: INFO: generated TableFilterIntervals node with 1 intervals: Intervals: [[ENST00000217939]-[ENST00000217939]] Predicate: (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False)) Post: (Coalesce (True) (False)) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 83: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":null}" (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_312) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_312) (I32 500)) (MakeStruct (i_start (Ref __iruid_313)) (j_start (Ref __iruid_314))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_316))) (j (GetField j_start (Ref __iruid_316)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_317) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_317) (I32 1)))) (start_idx (InsertFields (Ref __iruid_318) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_318)) (GetField j_start (Ref __iruid_318))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row)))))))))))) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 83: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":null}" (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_312) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_312) (I32 500)) (MakeStruct (i_start (Ref __iruid_313)) (j_start (Ref __iruid_314))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_316))) (j (GetField j_start (Ref __iruid_316)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_317) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_317) (I32 1)))) (start_idx (InsertFields (Ref __iruid_318) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_318)) (GetField j_start (Ref __iruid_318))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row)))))))))))) 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 83: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":null}" (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_345) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_345) (I32 500)) (MakeStruct (i_start (Ref __iruid_346)) (j_start (Ref __iruid_347))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_349))) (j (GetField j_start (Ref __iruid_349)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_350) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_350) (I32 1)))) (start_idx (InsertFields (Ref __iruid_351) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_351)) (GetField j_start (Ref __iruid_351))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row)))))))))))) 2022-03-09 16:30:46 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:46 root: INFO: lowering result: TableWrite 2022-03-09 16:30:46 root: INFO: compiling and evaluating result: TableWrite 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 159: (Let (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0)) (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_363)) (writeCtx (Ref __iruid_364)))) (MakeStruct (__iruid_355 (Ref __iruid_355))) (Let (GetField __iruid_355 (Ref __iruid_367)) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (Let (GetField oldCtx (Ref __iruid_365)) (Let (Ref __iruid_366) (Let (Ref __iruid_355) (StreamMap (Let (Ref __iruid_355) (StreamFilter (Let (Ref __iruid_355) (StreamMap (StreamFlatMap (Let (Ref __iruid_355) (StreamMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_356)) (Let (Ref __iruid_357) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_345) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_345) (I32 500)) (MakeStruct (i_start (Ref __iruid_346)) (j_start (Ref __iruid_347)))))))))))) (Let (GetField start_idx (Ref __iruid_358)) (StreamMap (ToStream False (Ref __iruid_359)) (InsertFields (Ref __iruid_358) None (start_idx (Ref __iruid_360)))))) (Let (Ref __iruid_361) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_349))) (j (GetField j_start (Ref __iruid_349)))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (Ref __iruid_362) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_350) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_350) (I32 1)))) (start_idx (InsertFields (Ref __iruid_351) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_351)) (GetField j_start (Ref __iruid_351))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_365)) (UUID4 __iruid_368))))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (Ref __iruid_355)) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_369)) (GetField filePath (Ref __iruid_370))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_369)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_371))))))))) 2022-03-09 16:30:46 root: INFO: Prune: MakeStruct: eliminating field '__iruid_355' 2022-03-09 16:30:46 root: INFO: Prune: MakeStruct: eliminating field '__iruid_355' 2022-03-09 16:30:46 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 124: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_439)) (writeCtx (Ref __iruid_440)))) (If (IsNA (Literal Struct{} )) (NA Struct{}) (Literal Struct{} )) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_437))) (Let (GetField list_len (Ref __iruid_444)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_445) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_445) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_446)) (j_start (Ref __iruid_447))) (InsertFields (Ref __iruid_444) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_448)) (i (GetField i_start (Ref __iruid_448))) (j (GetField j_start (Ref __iruid_448))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_443)) (GetField i (Ref __iruid_443))) (False))) (Let (GetField list_len (Ref __iruid_442)) (Let (GetField start_idx (Ref __iruid_442)) (InsertFields (Ref __iruid_442) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_442)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_449) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_442)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_449) (I32 1)))) (start_idx (InsertFields (Ref __iruid_450) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_450)) (GetField j_start (Ref __iruid_450))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_442))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_442))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_437)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_436)) (GetField filePath (Ref __iruid_451))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_436)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_452)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 124: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_439)) (writeCtx (Ref __iruid_440)))) (If (IsNA (Literal Struct{} )) (NA Struct{}) (Literal Struct{} )) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_437))) (Let (GetField list_len (Ref __iruid_444)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_445) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_445) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_446)) (j_start (Ref __iruid_447))) (InsertFields (Ref __iruid_444) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_448)) (i (GetField i_start (Ref __iruid_448))) (j (GetField j_start (Ref __iruid_448))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_443)) (GetField i (Ref __iruid_443))) (False))) (Let (GetField list_len (Ref __iruid_442)) (Let (GetField start_idx (Ref __iruid_442)) (InsertFields (Ref __iruid_442) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_442)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_449) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_442)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_449) (I32 1)))) (start_idx (InsertFields (Ref __iruid_450) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_450)) (GetField j_start (Ref __iruid_450))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_442))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_442))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_437)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_436)) (GetField filePath (Ref __iruid_451))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_436)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_452)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_488)) (writeCtx (Ref __iruid_489)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_486))) (Let (GetField list_len (Ref __iruid_492)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_493) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_493) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_494)) (j_start (Ref __iruid_495))) (InsertFields (Ref __iruid_492) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_496)) (i (GetField i_start (Ref __iruid_496))) (j (GetField j_start (Ref __iruid_496))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_491)) (GetField i (Ref __iruid_491))) (False))) (Let (GetField list_len (Ref __iruid_490)) (Let (GetField start_idx (Ref __iruid_490)) (InsertFields (Ref __iruid_490) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_490)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_497) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_490)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_497) (I32 1)))) (start_idx (InsertFields (Ref __iruid_498) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_498)) (GetField j_start (Ref __iruid_498))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_490))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_490))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_486)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_485)) (GetField filePath (Ref __iruid_499))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_485)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_500)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_488)) (writeCtx (Ref __iruid_489)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_486))) (Let (GetField list_len (Ref __iruid_492)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_493) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_493) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_494)) (j_start (Ref __iruid_495))) (InsertFields (Ref __iruid_492) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_496)) (i (GetField i_start (Ref __iruid_496))) (j (GetField j_start (Ref __iruid_496))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_491)) (GetField i (Ref __iruid_491))) (False))) (Let (GetField list_len (Ref __iruid_490)) (Let (GetField start_idx (Ref __iruid_490)) (InsertFields (Ref __iruid_490) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_490)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_497) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_490)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_497) (I32 1)))) (start_idx (InsertFields (Ref __iruid_498) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_498)) (GetField j_start (Ref __iruid_498))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_490))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_490))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_486)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_485)) (GetField filePath (Ref __iruid_499))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_485)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_500)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_536)) (writeCtx (Ref __iruid_537)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_534))) (Let (GetField list_len (Ref __iruid_540)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_541) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_541) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_542)) (j_start (Ref __iruid_543))) (InsertFields (Ref __iruid_540) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_544)) (i (GetField i_start (Ref __iruid_544))) (j (GetField j_start (Ref __iruid_544))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_539)) (GetField i (Ref __iruid_539))) (False))) (Let (GetField list_len (Ref __iruid_538)) (Let (GetField start_idx (Ref __iruid_538)) (InsertFields (Ref __iruid_538) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_538)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_545) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_538)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_545) (I32 1)))) (start_idx (InsertFields (Ref __iruid_546) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_546)) (GetField j_start (Ref __iruid_546))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_538))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_538))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_534)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_533)) (GetField filePath (Ref __iruid_547))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_533)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_548)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_536)) (writeCtx (Ref __iruid_537)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_534))) (Let (GetField list_len (Ref __iruid_540)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_541) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_541) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_542)) (j_start (Ref __iruid_543))) (InsertFields (Ref __iruid_540) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_544)) (i (GetField i_start (Ref __iruid_544))) (j (GetField j_start (Ref __iruid_544))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_539)) (GetField i (Ref __iruid_539))) (False))) (Let (GetField list_len (Ref __iruid_538)) (Let (GetField start_idx (Ref __iruid_538)) (InsertFields (Ref __iruid_538) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_538)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_545) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_538)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_545) (I32 1)))) (start_idx (InsertFields (Ref __iruid_546) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_546)) (GetField j_start (Ref __iruid_546))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_538))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_538))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_534)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_533)) (GetField filePath (Ref __iruid_547))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_533)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_548)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_584)) (writeCtx (Ref __iruid_585)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_582))) (Let (GetField list_len (Ref __iruid_588)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_589) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_589) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_590)) (j_start (Ref __iruid_591))) (InsertFields (Ref __iruid_588) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_592)) (i (GetField i_start (Ref __iruid_592))) (j (GetField j_start (Ref __iruid_592))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_587)) (GetField i (Ref __iruid_587))) (False))) (Let (GetField list_len (Ref __iruid_586)) (Let (GetField start_idx (Ref __iruid_586)) (InsertFields (Ref __iruid_586) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_586)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_593) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_586)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_593) (I32 1)))) (start_idx (InsertFields (Ref __iruid_594) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_594)) (GetField j_start (Ref __iruid_594))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_586))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_586))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_582)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_581)) (GetField filePath (Ref __iruid_595))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_581)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_596)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_584)) (writeCtx (Ref __iruid_585)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_582))) (Let (GetField list_len (Ref __iruid_588)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_589) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_589) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_590)) (j_start (Ref __iruid_591))) (InsertFields (Ref __iruid_588) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_592)) (i (GetField i_start (Ref __iruid_592))) (j (GetField j_start (Ref __iruid_592))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_587)) (GetField i (Ref __iruid_587))) (False))) (Let (GetField list_len (Ref __iruid_586)) (Let (GetField start_idx (Ref __iruid_586)) (InsertFields (Ref __iruid_586) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_586)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_593) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_586)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_593) (I32 1)))) (start_idx (InsertFields (Ref __iruid_594) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_594)) (GetField j_start (Ref __iruid_594))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_586))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_586))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_582)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_581)) (GetField filePath (Ref __iruid_595))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_581)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_596)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_632)) (writeCtx (Ref __iruid_633)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_630))) (Let (GetField list_len (Ref __iruid_636)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_637) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_637) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_638)) (j_start (Ref __iruid_639))) (InsertFields (Ref __iruid_636) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_640)) (i (GetField i_start (Ref __iruid_640))) (j (GetField j_start (Ref __iruid_640))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_635)) (GetField i (Ref __iruid_635))) (False))) (Let (GetField list_len (Ref __iruid_634)) (Let (GetField start_idx (Ref __iruid_634)) (InsertFields (Ref __iruid_634) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_634)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_641) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_634)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_641) (I32 1)))) (start_idx (InsertFields (Ref __iruid_642) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_642)) (GetField j_start (Ref __iruid_642))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_634))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_634))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_630)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_629)) (GetField filePath (Ref __iruid_643))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_629)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_644)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_632)) (writeCtx (Ref __iruid_633)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_630))) (Let (GetField list_len (Ref __iruid_636)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_637) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_637) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_638)) (j_start (Ref __iruid_639))) (InsertFields (Ref __iruid_636) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_640)) (i (GetField i_start (Ref __iruid_640))) (j (GetField j_start (Ref __iruid_640))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_635)) (GetField i (Ref __iruid_635))) (False))) (Let (GetField list_len (Ref __iruid_634)) (Let (GetField start_idx (Ref __iruid_634)) (InsertFields (Ref __iruid_634) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_634)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_641) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_634)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_641) (I32 1)))) (start_idx (InsertFields (Ref __iruid_642) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_642)) (GetField j_start (Ref __iruid_642))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_634))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_634))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_630)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_629)) (GetField filePath (Ref __iruid_643))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_629)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_644)))))))) 2022-03-09 16:30:47 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 120: (WriteMetadata "{\"name\":\"RelationalWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"overwrite\":true,\"maybeRefs\":{\"references\":[]}}" (Let (CollectDistributedArray (StreamZip -1 AssertSameLength (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (ToStream False (Literal Array[String] )) (MakeStruct (oldCtx (Ref __iruid_680)) (writeCtx (Ref __iruid_681)))) (Literal Struct{} ) (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[\"transcript\",\"i\",\"j\"],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows/parts/\",\"index\":{\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/index/\":\"+PCStruct{transcript:PCString,i:+PInt32,j:+PInt32}\"}}" (StreamMap (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (GetField oldCtx (Ref __iruid_678))) (Let (GetField list_len (Ref __iruid_684)) (StreamFlatMap (StreamRange 1 False (I32 0) (Ref __iruid_685) (I32 500)) (StreamMap (StreamRange 3 False (I32 0) (Ref __iruid_685) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_686)) (j_start (Ref __iruid_687))) (InsertFields (Ref __iruid_684) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_688)) (i (GetField i_start (Ref __iruid_688))) (j (GetField j_start (Ref __iruid_688))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_683)) (GetField i (Ref __iruid_683))) (False))) (Let (GetField list_len (Ref __iruid_682)) (Let (GetField start_idx (Ref __iruid_682)) (InsertFields (Ref __iruid_682) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 4 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref __iruid_682)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_689) (I32 1)))) (j_max_idx (ApplySpecial 5 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref __iruid_682)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_689) (I32 1)))) (start_idx (InsertFields (Ref __iruid_690) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_690)) (GetField j_start (Ref __iruid_690))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref __iruid_682))) (I32 1)) (GetField j_start (GetField start_idx (Ref __iruid_682))))))))))) (Apply -1 concat () String (GetField writeCtx (Ref __iruid_678)) (UUID4 __iruid_368)))) (Begin (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"key\":[],\"bounds\":[{\"start\":{},\"end\":{},\"includeStart\":true,\"includeEnd\":true}],\"indexSpec\":null,\"attrs\":{}}}" (MakeArray Array[String] (GetField filePath (WritePartition "{\"name\":\"PartitionNativeWriter\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"keyFields\":[],\"partPrefix\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/globals/parts/\"}" (MakeStream Stream[Struct{min_window_size:Int32}] False (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0))) (Str "part-0"))))) (WriteMetadata "{\"name\":\"RVDSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht/rows\",\"spec\":{\"name\":\"RVDSpecMaker\",\"codecSpec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32,start_idx:+EBaseStruct{i_start:+EInt32,j_start:+EInt32},i:+EInt32,j:+EInt32,i_max_idx:+EInt32,j_max_idx:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBl..." (ToArray (StreamMap (ToStream False (Ref __iruid_677)) (GetField filePath (Ref __iruid_691))))) (WriteMetadata "{\"name\":\"TableSpecWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht\",\"typ\":{\"rowType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32}\",\"key\":[\"transcript\",\"i\",\"j\"],\"globalType\":\"Struct{min_window_size:Int32}\"},\"rowRelPath\":\"rows\",\"globalRelPath\":\"globals\",\"refRelPath\":\"references\",\"log\":true}" (ToArray (StreamMap (ToStream False (Ref __iruid_677)) (SelectFields (partitionCounts distinctlyKeyed firstKey lastKey) (Ref __iruid_692)))))))) 2022-03-09 16:30:47 root: INFO: decoder cache miss (0 hits, 4 misses, 0.000 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C230HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C230HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C231FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C231FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C232etypeDecode. 2022-03-09 16:30:47 root: INFO: instruction count: 7: __C232etypeDecode.apply 2022-03-09 16:30:47 root: INFO: instruction count: 184: __C232etypeDecode.__m233DECODE_r_struct_of_o_binaryANDo_float64ANDr_int32ANDo_int32ANDo_int32ANDr_array_of_o_int64ANDr_array_of_o_float64ANDr_array_of_o_int32ANDr_int32END_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 31: __C232etypeDecode.__m234INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C232etypeDecode.__m235INPLACE_DECODE_o_float64_TO_o_float64 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C232etypeDecode.__m236INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C232etypeDecode.__m237INPLACE_DECODE_o_int32_TO_o_int32 2022-03-09 16:30:47 root: INFO: instruction count: 114: __C232etypeDecode.__m238INPLACE_DECODE_r_array_of_o_int64_TO_r_array_of_o_int64 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C232etypeDecode.__m239INPLACE_DECODE_o_int64_TO_o_int64 2022-03-09 16:30:47 root: INFO: instruction count: 114: __C232etypeDecode.__m240INPLACE_DECODE_r_array_of_o_float64_TO_r_array_of_o_float64 2022-03-09 16:30:47 root: INFO: instruction count: 114: __C232etypeDecode.__m241INPLACE_DECODE_r_array_of_o_int32_TO_r_array_of_o_int32 2022-03-09 16:30:47 root: INFO: decoder cache hit 2022-03-09 16:30:47 root: INFO: decoder cache hit 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C209HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C209HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C210FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C210FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C211collect_distributed_array. 2022-03-09 16:30:47 root: INFO: instruction count: 416: __C211collect_distributed_array.apply 2022-03-09 16:30:47 root: INFO: instruction count: 17: __C211collect_distributed_array.apply 2022-03-09 16:30:47 root: INFO: instruction count: 27: __C211collect_distributed_array.__m212DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryENDEND_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 26: __C211collect_distributed_array.__m213INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryEND_TO_r_struct_of_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDENDANDr_stringEND 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C211collect_distributed_array.__m214INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 31: __C211collect_distributed_array.__m215INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:47 root: INFO: instruction count: 44: __C211collect_distributed_array.__m216INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 26: __C211collect_distributed_array.__m217INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 17: __C211collect_distributed_array.__m218INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C211collect_distributed_array.__m219INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C211collect_distributed_array.__m220INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:47 root: INFO: instruction count: 27: __C211collect_distributed_array.__m222DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 8: __C211collect_distributed_array.__m223INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:47 root: INFO: instruction count: 49: __C211collect_distributed_array.__m227concat 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.setObjects 2022-03-09 16:30:47 root: INFO: instruction count: 11: __C211collect_distributed_array.__m272ord_gteq 2022-03-09 16:30:47 root: INFO: instruction count: 14: __C211collect_distributed_array.__m273ord_gteqNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 116: __C211collect_distributed_array.__m282split_InsertFields 2022-03-09 16:30:47 root: INFO: instruction count: 11: __C211collect_distributed_array.__m284ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 14: __C211collect_distributed_array.__m285ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 413: __C211collect_distributed_array.__m302writeInternalNode 2022-03-09 16:30:47 root: INFO: instruction count: 25: __C211collect_distributed_array.__m303ENCODE_SBaseStructPointer_TO_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C211collect_distributed_array.__m304ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 53: __C211collect_distributed_array.__m305ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m306ENCODE_SInt64$_TO_r_int64 2022-03-09 16:30:47 root: INFO: instruction count: 60: __C211collect_distributed_array.__m307ENCODE_SBaseStructPointer_TO_r_struct_of_o_binaryANDr_int32ANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 16: __C211collect_distributed_array.__m308ENCODE_SStringPointer_TO_o_binary 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m309ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 1: __C211collect_distributed_array.__m310ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:47 root: INFO: instruction count: 324: __C211collect_distributed_array.__m312writeLeafNode 2022-03-09 16:30:47 root: INFO: instruction count: 37: __C211collect_distributed_array.__m313ENCODE_SBaseStructPointer_TO_o_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C211collect_distributed_array.__m314ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 29: __C211collect_distributed_array.__m315ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 52: __C211collect_distributed_array.__m316flush 2022-03-09 16:30:47 root: INFO: instruction count: 12: __C211collect_distributed_array.__m319setup_null 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C211collect_distributed_array.__m320ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 145: __C211collect_distributed_array.__m321ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 30: __C211collect_distributed_array.__m322ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C211collect_distributed_array.__m323ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C211collect_distributed_array.__m324ord_compareNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 89: __C211collect_distributed_array.__m325ord_compareNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 292: __C211collect_distributed_array.__m326ENCODE_SInsertFieldsStruct_TO_r_struct_of_o_binaryANDo_float64ANDr_int32ANDo_int32ANDo_int32ANDr_array_of_o_int64ANDr_array_of_o_float64ANDr_array_of_o_int32ANDr_int32ANDr_struct_of_r_int32ANDr_int32ENDANDr_int32ANDr_int32ANDr_int32ANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m327ENCODE_SFloat64$_TO_o_float64 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m328ENCODE_SInt32$_TO_o_int32 2022-03-09 16:30:47 root: INFO: instruction count: 115: __C211collect_distributed_array.__m329ENCODE_SIndexablePointer_TO_r_array_of_o_int64 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m330ENCODE_SInt64$_TO_o_int64 2022-03-09 16:30:47 root: INFO: instruction count: 115: __C211collect_distributed_array.__m331ENCODE_SIndexablePointer_TO_r_array_of_o_float64 2022-03-09 16:30:47 root: INFO: instruction count: 115: __C211collect_distributed_array.__m332ENCODE_SIndexablePointer_TO_r_array_of_o_int32 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C211collect_distributed_array.__m333ENCODE_SInsertFieldsStruct_TO_r_struct_of_r_int32ANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C211collect_distributed_array.__m334ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_int64ANDr_boolANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 87: __C211collect_distributed_array.__m335ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_int64ANDr_boolANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 16: __C211collect_distributed_array.__m336ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.__m337ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:47 root: INFO: instruction count: 60: __C211collect_distributed_array.__m338ENCODE_SBaseStructPointer_TO_o_struct_of_o_binaryANDr_int32ANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C211collect_distributed_array.setPartitionIndex 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.addPartitionRegion 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C211collect_distributed_array.setPool 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C211collect_distributed_array.addHailClassLoader 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C211collect_distributed_array.addFS 2022-03-09 16:30:47 root: INFO: instruction count: 566: __C211collect_distributed_array.apply_region0_0 2022-03-09 16:30:47 root: INFO: instruction count: 195: __C211collect_distributed_array.apply_region14_20 2022-03-09 16:30:47 root: INFO: instruction count: 659: __C211collect_distributed_array.apply_region5_63 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C341applySpills. 2022-03-09 16:30:47 root: INFO: encoder cache miss (0 hits, 4 misses, 0.000) 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C458HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C458HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C459FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C459FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C460etypeEncode. 2022-03-09 16:30:47 root: INFO: instruction count: 7: __C460etypeEncode.apply 2022-03-09 16:30:47 root: INFO: instruction count: 81: __C460etypeEncode.__m461ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_struct_of_ENDANDr_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryANDr_array_of_r_binaryEND 2022-03-09 16:30:47 root: INFO: instruction count: 16: __C460etypeEncode.__m462ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:47 root: INFO: instruction count: 1: __C460etypeEncode.__m463ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C460etypeEncode.__m464ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 51: __C460etypeEncode.__m465ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 41: __C460etypeEncode.__m466ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C460etypeEncode.__m467ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 13: __C460etypeEncode.__m468ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C460etypeEncode.__m469ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C460etypeEncode.__m470ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:47 root: INFO: instruction count: 39: __C460etypeEncode.__m471ENCODE_SIndexablePointer_TO_r_array_of_r_binary 2022-03-09 16:30:47 MemoryStore: INFO: Block broadcast_5 stored as values in memory (estimated size 416.0 B, free 12.6 GiB) 2022-03-09 16:30:47 MemoryStore: INFO: Block broadcast_5_piece0 stored as bytes in memory (estimated size 246.0 B, free 12.6 GiB) 2022-03-09 16:30:47 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 246.0 B, free: 12.6 GiB) 2022-03-09 16:30:47 SparkContext: INFO: Created broadcast 5 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C180HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C180HailClassLoaderContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C181FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C181FSContainer. 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C182Compiled. 2022-03-09 16:30:47 root: INFO: instruction count: 26: __C182Compiled.apply 2022-03-09 16:30:47 root: INFO: instruction count: 419: __C182Compiled.__m183split_Let 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.setBackend 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C182Compiled.__m373ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C182Compiled.__m374ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryEND 2022-03-09 16:30:47 root: INFO: instruction count: 51: __C182Compiled.__m375ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 16: __C182Compiled.__m376ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:47 root: INFO: instruction count: 41: __C182Compiled.__m377ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C182Compiled.__m378ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 13: __C182Compiled.__m379ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.__m380ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.__m381ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C182Compiled.__m382ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 1: __C182Compiled.__m383ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.setObjects 2022-03-09 16:30:47 root: INFO: instruction count: 27: __C182Compiled.__m385DECODE_r_struct_of_r_struct_of_r_binaryANDr_int64ANDr_boolANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 102: __C182Compiled.__m386INPLACE_DECODE_r_struct_of_r_binaryANDr_int64ANDr_boolANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDANDo_struct_of_o_binaryANDr_int32ANDr_int32ENDEND_TO_r_struct_of_r_stringANDr_int64ANDr_boolANDo_struct_of_o_stringANDr_int32ANDr_int32ENDANDo_struct_of_o_stringANDr_int32ANDr_int32ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 31: __C182Compiled.__m387INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C182Compiled.__m388INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C182Compiled.__m389INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:47 root: INFO: instruction count: 66: __C182Compiled.__m390INPLACE_DECODE_o_struct_of_o_binaryANDr_int32ANDr_int32END_TO_o_struct_of_o_stringANDr_int32ANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 31: __C182Compiled.__m391INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:47 root: INFO: instruction count: 10: __C182Compiled.__m392INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:47 root: INFO: instruction count: 432: __C182Compiled.__m396begin_group_0 2022-03-09 16:30:47 root: INFO: instruction count: 264: __C182Compiled.__m399split_ToArray 2022-03-09 16:30:47 root: INFO: instruction count: 27: __C182Compiled.__m405DECODE_r_struct_of_r_int32END_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 12: __C182Compiled.__m408setup_jab 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C182Compiled.__m413arrayref_bounds_check 2022-03-09 16:30:47 root: INFO: instruction count: 13: __C182Compiled.__m416ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 143: __C182Compiled.__m417split_ToArray 2022-03-09 16:30:47 root: INFO: instruction count: 229: __C182Compiled.__m428split_ToArray 2022-03-09 16:30:47 root: INFO: instruction count: 18: __C182Compiled.__m439ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 176: __C182Compiled.__m440ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 30: __C182Compiled.__m441ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 21: __C182Compiled.__m442ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C182Compiled.__m443ord_compareNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 89: __C182Compiled.__m444ord_compareNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 11: __C182Compiled.__m445ord_equiv 2022-03-09 16:30:47 root: INFO: instruction count: 14: __C182Compiled.__m446ord_equivNonnull 2022-03-09 16:30:47 root: INFO: instruction count: 9: __C182Compiled.setPartitionIndex 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.addPartitionRegion 2022-03-09 16:30:47 root: INFO: instruction count: 4: __C182Compiled.setPool 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C182Compiled.addHailClassLoader 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C182Compiled.addFS 2022-03-09 16:30:47 root: INFO: instruction count: 111: __C182Compiled.addLiterals 2022-03-09 16:30:47 root: INFO: instruction count: 63: __C182Compiled.__m450DECODE_r_struct_of_r_binaryANDr_struct_of_ENDANDr_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDANDr_binaryANDr_array_of_r_binaryEND_TO_SBaseStructPointer 2022-03-09 16:30:47 root: INFO: instruction count: 8: __C182Compiled.__m451INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:47 root: INFO: instruction count: 58: __C182Compiled.__m452INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 35: __C182Compiled.__m453INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:47 root: INFO: instruction count: 44: __C182Compiled.__m454INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 26: __C182Compiled.__m455INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:47 root: INFO: instruction count: 17: __C182Compiled.__m456INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:47 root: INFO: instruction count: 58: __C182Compiled.__m457INPLACE_DECODE_r_array_of_r_binary_TO_r_array_of_r_string 2022-03-09 16:30:47 root: INFO: instruction count: 178: __C182Compiled.__m396begin_group_0_region21_43 2022-03-09 16:30:47 root: INFO: instruction count: 3: __C472__m396begin_group_0Spills. 2022-03-09 16:30:48 MemoryStore: INFO: Block broadcast_6 stored as values in memory (estimated size 48.0 B, free 12.6 GiB) 2022-03-09 16:30:48 MemoryStore: INFO: Block broadcast_6_piece0 stored as bytes in memory (estimated size 49.0 B, free 12.6 GiB) 2022-03-09 16:30:48 BlockManagerInfo: INFO: Added broadcast_6_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:30:48 SparkContext: INFO: Created broadcast 6 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:48 SparkContext: INFO: Starting job: collect at SparkBackend.scala:288 2022-03-09 16:30:48 DAGScheduler: INFO: Got job 1 (collect at SparkBackend.scala:288) with 1 output partitions 2022-03-09 16:30:48 DAGScheduler: INFO: Final stage: ResultStage 1 (collect at SparkBackend.scala:288) 2022-03-09 16:30:48 DAGScheduler: INFO: Parents of final stage: List() 2022-03-09 16:30:48 DAGScheduler: INFO: Missing parents: List() 2022-03-09 16:30:48 DAGScheduler: INFO: Submitting ResultStage 1 (SparkBackendComputeRDD[1] at RDD at SparkBackend.scala:720), which has no missing parents 2022-03-09 16:30:48 MemoryStore: INFO: Block broadcast_7 stored as values in memory (estimated size 68.4 KiB, free 12.6 GiB) 2022-03-09 16:30:48 MemoryStore: INFO: Block broadcast_7_piece0 stored as bytes in memory (estimated size 36.3 KiB, free 12.6 GiB) 2022-03-09 16:30:48 BlockManagerInfo: INFO: Added broadcast_7_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 36.3 KiB, free: 12.6 GiB) 2022-03-09 16:30:48 SparkContext: INFO: Created broadcast 7 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:30:48 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 1 (SparkBackendComputeRDD[1] at RDD at SparkBackend.scala:720) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:30:48 YarnScheduler: INFO: Adding task set 1.0 with 1 tasks resource profile 0 2022-03-09 16:30:48 FairSchedulableBuilder: INFO: Added task set TaskSet_1.0 tasks to pool default 2022-03-09 16:30:48 TaskSetManager: INFO: Starting task 0.0 in stage 1.0 (TID 1) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 0, PROCESS_LOCAL, 4574 bytes) taskResourceAssignments Map() 2022-03-09 16:30:48 BlockManagerInfo: INFO: Added broadcast_7_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:41631 (size: 36.3 KiB, free: 2.9 GiB) 2022-03-09 16:30:49 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:41631 (size: 33.4 KiB, free: 2.9 GiB) 2022-03-09 16:30:49 BlockManagerInfo: INFO: Added broadcast_6_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:41631 (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:30:52 TaskSetManager: INFO: Finished task 0.0 in stage 1.0 (TID 1) in 4325 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (1/1) 2022-03-09 16:30:52 YarnScheduler: INFO: Removed TaskSet 1.0, whose tasks have all completed, from pool default 2022-03-09 16:30:52 DAGScheduler: INFO: ResultStage 1 (collect at SparkBackend.scala:288) finished in 4.337 s 2022-03-09 16:30:52 DAGScheduler: INFO: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:30:52 YarnScheduler: INFO: Killing all running tasks in stage 1: Stage finished 2022-03-09 16:30:52 DAGScheduler: INFO: Job 1 finished: collect at SparkBackend.scala:288, took 4.341623 s 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_6_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_6_piece0 on kc-w-0.c.broad-mpg-gnomad.internal:41631 in memory (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_7_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 36.3 KiB, free: 12.6 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_7_piece0 on kc-w-0.c.broad-mpg-gnomad.internal:41631 in memory (size: 36.3 KiB, free: 2.9 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 21.1 KiB, free: 12.6 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 21.1 KiB, free: 2.9 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_1_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:30:52 BlockManagerInfo: INFO: Removed broadcast_1_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:30:53 GoogleCloudStorageImpl: INFO: Ignoring exception of type GoogleJsonResponseException; verified object already exists with desired state. 2022-03-09 16:30:53 Hail: INFO: wrote table with 120 rows in 1 partition to gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht 2022-03-09 16:30:53 root: INFO: took 7.265s 2022-03-09 16:30:53 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1: (Begin) 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C481HailClassLoaderContainer. 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C481HailClassLoaderContainer. 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C482FSContainer. 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C482FSContainer. 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C483Compiled. 2022-03-09 16:30:53 root: INFO: instruction count: 1: __C483Compiled.apply 2022-03-09 16:30:53 root: INFO: instruction count: 9: __C483Compiled.setPartitionIndex 2022-03-09 16:30:53 root: INFO: instruction count: 4: __C483Compiled.addPartitionRegion 2022-03-09 16:30:53 root: INFO: instruction count: 4: __C483Compiled.setPool 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C483Compiled.addHailClassLoader 2022-03-09 16:30:53 root: INFO: instruction count: 3: __C483Compiled.addFS 2022-03-09 16:30:53 root: INFO: finished execution of query hail_query_2 2022-03-09 16:30:53 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode total 7.451s self 12.383ms children 7.438s %children 99.83% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 93.596ms self 0.022ms children 93.574ms %children 99.98% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 93.467ms self 15.248ms children 78.219ms %children 83.69% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 78.219ms self 0.287ms children 77.932ms %children 99.63% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.835ms self 0.835ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 3.675ms self 3.675ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 12.184ms self 12.184ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 6.495ms self 6.495ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.499ms self 0.499ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 13.263ms self 13.263ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.127ms self 1.127ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.852ms self 1.852ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.565ms self 1.565ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 9.343ms self 9.343ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.569ms self 0.569ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 8.961ms self 8.961ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.847ms self 0.847ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.585ms self 1.585ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.003ms self 1.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 5.655ms self 5.655ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.329ms self 0.329ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 8.144ms self 8.144ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.628ms self 0.017ms children 0.611ms %children 97.25% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.463ms self 0.463ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.097ms self 0.097ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 70.300ms self 0.022ms children 70.278ms %children 99.97% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 70.165ms self 6.775ms children 63.390ms %children 90.34% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 63.390ms self 0.442ms children 62.948ms %children 99.30% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.568ms self 0.568ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.611ms self 1.611ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.537ms self 1.537ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 6.781ms self 6.781ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 1.398ms self 1.398ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 7.952ms self 7.952ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.619ms self 0.619ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.275ms self 1.275ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.394ms self 1.394ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 7.746ms self 7.746ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.600ms self 0.600ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 9.365ms self 9.365ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.714ms self 0.714ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.875ms self 1.875ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.191ms self 1.191ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 8.581ms self 8.581ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.381ms self 0.381ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 9.359ms self 9.359ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.447ms self 0.040ms children 0.407ms %children 90.95% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.110ms self 0.110ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.240ms self 0.240ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 7.266s self 0.045ms children 7.266s %children 100.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 7.265s self 254.729ms children 7.011s %children 96.49% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 148.874ms self 0.058ms children 148.817ms %children 99.96% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.089ms self 0.089ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 148.643ms self 70.970ms children 77.673ms %children 52.25% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 77.673ms self 0.127ms children 77.546ms %children 99.84% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.374ms self 1.374ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 2.099ms self 2.099ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.138ms self 2.138ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 13.502ms self 13.502ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.522ms self 0.522ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 20.918ms self 20.918ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.968ms self 1.968ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.560ms self 0.560ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 5.607ms self 5.607ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 5.770ms self 5.770ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.421ms self 0.421ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 6.439ms self 6.439ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.471ms self 1.471ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.466ms self 0.466ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.152ms self 1.152ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 7.660ms self 7.660ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.665ms self 0.665ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.815ms self 4.815ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.520ms self 0.009ms children 0.512ms %children 98.32% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.078ms self 0.078ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.367ms self 0.367ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 64.536ms self 0.012ms children 64.524ms %children 99.98% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 64.405ms self 21.928ms children 42.477ms %children 65.95% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 42.477ms self 0.138ms children 42.338ms %children 99.67% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.787ms self 0.787ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.495ms self 0.495ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.142ms self 1.142ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 4.356ms self 4.356ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.387ms self 0.387ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.648ms self 4.648ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 1.087ms self 1.087ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 2.551ms self 2.551ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 2.682ms self 2.682ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 6.891ms self 6.891ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.381ms self 0.381ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.295ms self 4.295ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 1.151ms self 1.151ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.421ms self 0.421ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.926ms self 0.926ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 4.966ms self 4.966ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.296ms self 0.296ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.879ms self 4.879ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.393ms self 0.007ms children 0.385ms %children 98.18% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.270ms self 0.270ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.469ms self 0.006ms children 0.463ms %children 98.73% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.269ms self 0.269ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.141ms self 0.141ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 55.420ms self 0.012ms children 55.408ms %children 99.98% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 55.289ms self 18.631ms children 36.658ms %children 66.30% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 36.658ms self 0.096ms children 36.562ms %children 99.74% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.417ms self 0.417ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.454ms self 0.454ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.944ms self 0.944ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 4.305ms self 4.305ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.255ms self 0.255ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 4.722ms self 4.722ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.848ms self 0.848ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.506ms self 0.506ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.886ms self 0.886ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 6.076ms self 6.076ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.347ms self 0.347ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 4.740ms self 4.740ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.980ms self 0.980ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.445ms self 0.445ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 1.217ms self 1.217ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 4.333ms self 4.333ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.259ms self 0.259ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 4.828ms self 4.828ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 678.206ms self 466.467ms children 211.739ms %children 31.22% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 61.244ms self 0.015ms children 61.229ms %children 99.98% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 61.094ms self 18.767ms children 42.327ms %children 69.28% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 42.327ms self 0.145ms children 42.182ms %children 99.66% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.312ms self 0.312ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.524ms self 0.524ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.969ms self 0.969ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 6.285ms self 6.285ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.319ms self 0.319ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 5.787ms self 5.787ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.145ms self 1.145ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.413ms self 0.413ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.061ms self 1.061ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 4.553ms self 4.553ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.569ms self 0.569ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 6.257ms self 6.257ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.756ms self 1.756ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.737ms self 0.737ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.594ms self 1.594ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 4.022ms self 4.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.251ms self 0.251ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 5.631ms self 5.631ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.814ms self 0.013ms children 0.802ms %children 98.44% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.219ms self 0.219ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.250ms self 0.250ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.332ms self 0.332ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 60.335ms self 0.015ms children 60.320ms %children 99.97% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 60.175ms self 19.503ms children 40.672ms %children 67.59% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 40.672ms self 0.117ms children 40.555ms %children 99.71% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.463ms self 0.463ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.706ms self 0.706ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.686ms self 1.686ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 5.714ms self 5.714ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.456ms self 0.456ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 4.733ms self 4.733ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 1.332ms self 1.332ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.547ms self 0.547ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.978ms self 0.978ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 4.230ms self 4.230ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.281ms self 0.281ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 5.807ms self 5.807ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 1.276ms self 1.276ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.483ms self 0.483ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.989ms self 0.989ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 4.005ms self 4.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.257ms self 0.257ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 6.611ms self 6.611ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 24.056ms self 0.017ms children 24.039ms %children 99.93% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.245ms self 0.245ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 23.514ms self 23.514ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.280ms self 0.280ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 57.985ms self 0.015ms children 57.970ms %children 99.97% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 57.827ms self 17.352ms children 40.475ms %children 69.99% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 40.475ms self 0.112ms children 40.363ms %children 99.72% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.349ms self 0.349ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.556ms self 0.556ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.308ms self 1.308ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 5.082ms self 5.082ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.414ms self 0.414ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 5.317ms self 5.317ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.920ms self 0.920ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.562ms self 0.562ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.380ms self 1.380ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 5.654ms self 5.654ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.377ms self 0.377ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 5.174ms self 5.174ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.023ms self 1.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.524ms self 0.524ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.085ms self 1.085ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 5.931ms self 5.931ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.273ms self 0.273ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 4.433ms self 4.433ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 7.305ms self 7.305ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 5.312ms self 5.312ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledVoidFunction total 6.057s self 6.057s children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.887ms self 0.008ms children 0.879ms %children 99.09% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.872ms self 0.308ms children 0.564ms %children 64.71% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.564ms self 0.038ms children 0.526ms %children 93.23% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.279ms self 0.279ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile total 5.019ms self 3.799ms children 1.221ms %children 24.32% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR total 0.367ms self 0.004ms children 0.363ms %children 98.79% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 0.358ms self 0.131ms children 0.227ms %children 63.45% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.227ms self 0.013ms children 0.215ms %children 94.36% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.091ms self 0.091ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR total 0.038ms self 0.003ms children 0.036ms %children 92.99% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/LoweringTransformation total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR total 0.342ms self 0.004ms children 0.338ms %children 98.89% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.334ms self 0.123ms children 0.211ms %children 63.24% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.211ms self 0.012ms children 0.199ms %children 94.31% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs total 0.098ms self 0.003ms children 0.095ms %children 96.79% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.085ms self 0.085ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.305ms self 0.004ms children 0.301ms %children 98.65% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.297ms self 0.119ms children 0.178ms %children 59.79% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.178ms self 0.012ms children 0.165ms %children 93.14% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/Compile/EmitContext.analyze total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 1.687ms self 1.687ms children 0.000ms %children 0.00% 2022-03-09 16:30:53 root: INFO: timing SparkBackend.executeEncode/RunCompiledVoidFunction total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:54 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:54 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:54 root: INFO: timing SparkBackend.parse_table_ir total 295.777ms self 295.777ms children 0.000ms %children 0.00% 2022-03-09 16:30:54 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:54 root: INFO: JSON: JObject(List((name,JString(TableCalculateNewPartitions)), (nPartitions,JInt(120)))) 2022-03-09 16:30:54 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:54 root: INFO: timing SparkBackend.parse_value_ir total 41.570ms self 41.570ms children 0.000ms %children 0.00% 2022-03-09 16:30:54 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:54 root: INFO: starting execution of query hail_query_3 of initial size 2 2022-03-09 16:30:54 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":120}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht ))) 2022-03-09 16:30:54 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":120}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht ))) 2022-03-09 16:30:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":120}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht ))) 2022-03-09 16:30:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":120}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht ))) 2022-03-09 16:30:54 root: INFO: LowerOrInterpretNonCompilable: cannot efficiently lower query: TableToValueApply: no lowering for is.hail.expr.ir.functions.TableCalculateNewPartitions 2022-03-09 16:30:54 root: INFO: interpreting non-compilable result: TableToValueApply 2022-03-09 16:30:54 root: INFO: decoder cache miss (2 hits, 5 misses, 0.286 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C487HailClassLoaderContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C487HailClassLoaderContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C488FSContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C488FSContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C489etypeDecode. 2022-03-09 16:30:54 root: INFO: instruction count: 7: __C489etypeDecode.apply 2022-03-09 16:30:54 root: INFO: instruction count: 27: __C489etypeDecode.__m490DECODE_r_struct_of_r_int32END_TO_SBaseStructPointer 2022-03-09 16:30:54 root: INFO: instruction count: 10: __C489etypeDecode.__m491INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:54 root: INFO: decoder cache miss (2 hits, 6 misses, 0.250 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C492HailClassLoaderContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C492HailClassLoaderContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C493FSContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C493FSContainer. 2022-03-09 16:30:54 root: INFO: instruction count: 3: __C494etypeDecode. 2022-03-09 16:30:54 root: INFO: instruction count: 7: __C494etypeDecode.apply 2022-03-09 16:30:54 root: INFO: instruction count: 229: __C494etypeDecode.__m495DECODE_r_struct_of_o_binaryANDo_float64ANDr_int32ANDo_int32ANDo_int32ANDr_array_of_o_int64ANDr_array_of_o_float64ANDr_array_of_o_int32ANDr_int32ANDr_struct_of_r_int32ANDr_int32ENDANDr_int32ANDr_int32ANDr_int32ANDr_int32END_TO_SBaseStructPointer 2022-03-09 16:30:54 root: INFO: instruction count: 31: __C494etypeDecode.__m496INPLACE_DECODE_o_binary_TO_o_string 2022-03-09 16:30:54 root: INFO: instruction count: 10: __C494etypeDecode.__m497INPLACE_DECODE_o_float64_TO_o_float64 2022-03-09 16:30:54 root: INFO: instruction count: 10: __C494etypeDecode.__m498INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:54 root: INFO: instruction count: 10: __C494etypeDecode.__m499INPLACE_DECODE_o_int32_TO_o_int32 2022-03-09 16:30:54 root: INFO: instruction count: 114: __C494etypeDecode.__m500INPLACE_DECODE_r_array_of_o_int64_TO_r_array_of_o_int64 2022-03-09 16:30:54 root: INFO: instruction count: 10: __C494etypeDecode.__m501INPLACE_DECODE_o_int64_TO_o_int64 2022-03-09 16:30:54 root: INFO: instruction count: 114: __C494etypeDecode.__m502INPLACE_DECODE_r_array_of_o_float64_TO_r_array_of_o_float64 2022-03-09 16:30:54 root: INFO: instruction count: 114: __C494etypeDecode.__m503INPLACE_DECODE_r_array_of_o_int32_TO_r_array_of_o_int32 2022-03-09 16:30:54 root: INFO: instruction count: 26: __C494etypeDecode.__m504INPLACE_DECODE_r_struct_of_r_int32ANDr_int32END_TO_r_struct_of_r_int32ANDr_int32END 2022-03-09 16:30:54 SparkContext: INFO: Starting job: runJob at ContextRDD.scala:238 2022-03-09 16:30:54 DAGScheduler: INFO: Got job 2 (runJob at ContextRDD.scala:238) with 1 output partitions 2022-03-09 16:30:54 DAGScheduler: INFO: Final stage: ResultStage 2 (runJob at ContextRDD.scala:238) 2022-03-09 16:30:54 DAGScheduler: INFO: Parents of final stage: List() 2022-03-09 16:30:54 DAGScheduler: INFO: Missing parents: List() 2022-03-09 16:30:54 DAGScheduler: INFO: Submitting ResultStage 2 (MapPartitionsRDD[5] at mapPartitions at ContextRDD.scala:222), which has no missing parents 2022-03-09 16:30:54 MemoryStore: INFO: Block broadcast_8 stored as values in memory (estimated size 23.1 KiB, free 12.6 GiB) 2022-03-09 16:30:54 MemoryStore: INFO: Block broadcast_8_piece0 stored as bytes in memory (estimated size 11.6 KiB, free 12.6 GiB) 2022-03-09 16:30:54 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 11.6 KiB, free: 12.6 GiB) 2022-03-09 16:30:54 SparkContext: INFO: Created broadcast 8 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:30:54 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[5] at mapPartitions at ContextRDD.scala:222) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:30:54 YarnScheduler: INFO: Adding task set 2.0 with 1 tasks resource profile 0 2022-03-09 16:30:54 FairSchedulableBuilder: INFO: Added task set TaskSet_2.0 tasks to pool default 2022-03-09 16:30:54 TaskSetManager: INFO: Starting task 0.0 in stage 2.0 (TID 2) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 0, PROCESS_LOCAL, 4310 bytes) taskResourceAssignments Map() 2022-03-09 16:30:54 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:41631 (size: 11.6 KiB, free: 2.9 GiB) 2022-03-09 16:30:55 TaskSetManager: INFO: Finished task 0.0 in stage 2.0 (TID 2) in 436 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (1/1) 2022-03-09 16:30:55 YarnScheduler: INFO: Removed TaskSet 2.0, whose tasks have all completed, from pool default 2022-03-09 16:30:55 DAGScheduler: INFO: ResultStage 2 (runJob at ContextRDD.scala:238) finished in 0.449 s 2022-03-09 16:30:55 DAGScheduler: INFO: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:30:55 YarnScheduler: INFO: Killing all running tasks in stage 2: Stage finished 2022-03-09 16:30:55 DAGScheduler: INFO: Job 2 finished: runJob at ContextRDD.scala:238, took 0.454564 s 2022-03-09 16:30:55 root: INFO: took 840.779ms 2022-03-09 16:30:55 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:55 root: INFO: lowering result: Literal 2022-03-09 16:30:55 root: INFO: compiling and evaluating result: Literal 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (Literal Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]] )) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1: (Literal Tuple[Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: (Literal Tuple[Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1: (Literal Tuple[Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: (Literal Tuple[Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]] ) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1: (Literal Tuple[Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]] ) 2022-03-09 16:30:55 root: INFO: encoder cache miss (0 hits, 5 misses, 0.000) 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C520HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C520HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C521FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C521FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C522etypeEncode. 2022-03-09 16:30:55 root: INFO: instruction count: 7: __C522etypeEncode.apply 2022-03-09 16:30:55 root: INFO: instruction count: 9: __C522etypeEncode.__m523ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolENDENDEND 2022-03-09 16:30:55 root: INFO: instruction count: 25: __C522etypeEncode.__m524ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:55 root: INFO: instruction count: 53: __C522etypeEncode.__m525ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 41: __C522etypeEncode.__m526ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 37: __C522etypeEncode.__m527ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 16: __C522etypeEncode.__m528ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C522etypeEncode.__m529ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C522etypeEncode.__m530ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:55 MemoryStore: INFO: Block broadcast_9 stored as values in memory (estimated size 1200.0 B, free 12.6 GiB) 2022-03-09 16:30:55 MemoryStore: INFO: Block broadcast_9_piece0 stored as bytes in memory (estimated size 384.0 B, free 12.6 GiB) 2022-03-09 16:30:55 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 384.0 B, free: 12.6 GiB) 2022-03-09 16:30:55 SparkContext: INFO: Created broadcast 9 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C505HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C505HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C506FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C506FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C507Compiled. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C507Compiled.apply 2022-03-09 16:30:55 root: INFO: instruction count: 9: __C507Compiled.setPartitionIndex 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C507Compiled.addPartitionRegion 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C507Compiled.setPool 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C507Compiled.addHailClassLoader 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C507Compiled.addFS 2022-03-09 16:30:55 root: INFO: instruction count: 35: __C507Compiled.addLiterals 2022-03-09 16:30:55 root: INFO: instruction count: 27: __C507Compiled.__m512DECODE_r_struct_of_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:55 root: INFO: instruction count: 17: __C507Compiled.__m513INPLACE_DECODE_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_tuple_of_r_array_of_r_interval_of_r_struct_of_r_stringANDr_int32ANDr_int32ENDEND 2022-03-09 16:30:55 root: INFO: instruction count: 58: __C507Compiled.__m514INPLACE_DECODE_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND_TO_r_array_of_r_interval_of_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 44: __C507Compiled.__m515INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 35: __C507Compiled.__m516INPLACE_DECODE_r_struct_of_r_binaryANDr_int32ANDr_int32END_TO_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 31: __C507Compiled.__m517INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:55 root: INFO: instruction count: 10: __C507Compiled.__m518INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:55 root: INFO: instruction count: 10: __C507Compiled.__m519INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:55 root: INFO: encoder cache miss (0 hits, 6 misses, 0.000) 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C531HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C531HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C532FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C532FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C533etypeEncode. 2022-03-09 16:30:55 root: INFO: instruction count: 19: __C533etypeEncode.apply 2022-03-09 16:30:55 root: INFO: instruction count: 53: __C533etypeEncode.__m534ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 41: __C533etypeEncode.__m535ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 37: __C533etypeEncode.__m536ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 16: __C533etypeEncode.__m537ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C533etypeEncode.__m538ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C533etypeEncode.__m539ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:55 root: INFO: took 70.116ms 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]) 2022-03-09 16:30:55 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]]) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{transcript:String,i:Int32,j:Int32}]])) 2022-03-09 16:30:55 root: INFO: encoder cache miss (0 hits, 7 misses, 0.000) 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C557HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C557HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C558FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C558FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C559etypeEncode. 2022-03-09 16:30:55 root: INFO: instruction count: 7: __C559etypeEncode.apply 2022-03-09 16:30:55 root: INFO: instruction count: 1: __C559etypeEncode.__m560ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:55 MemoryStore: INFO: Block broadcast_10 stored as values in memory (estimated size 1240.0 B, free 12.6 GiB) 2022-03-09 16:30:55 MemoryStore: INFO: Block broadcast_10_piece0 stored as bytes in memory (estimated size 391.0 B, free 12.6 GiB) 2022-03-09 16:30:55 BlockManagerInfo: INFO: Added broadcast_10_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 391.0 B, free: 12.6 GiB) 2022-03-09 16:30:55 SparkContext: INFO: Created broadcast 10 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C540HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C540HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C541FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C541FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C542Compiled. 2022-03-09 16:30:55 root: INFO: instruction count: 26: __C542Compiled.apply 2022-03-09 16:30:55 root: INFO: instruction count: 9: __C542Compiled.setPartitionIndex 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C542Compiled.addPartitionRegion 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C542Compiled.setPool 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C542Compiled.addHailClassLoader 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C542Compiled.addFS 2022-03-09 16:30:55 root: INFO: instruction count: 67: __C542Compiled.addLiterals 2022-03-09 16:30:55 root: INFO: instruction count: 18: __C542Compiled.__m549DECODE_r_struct_of_END_TO_SBaseStructPointer 2022-03-09 16:30:55 root: INFO: instruction count: 62: __C542Compiled.__m550DECODE_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND_TO_SIndexablePointer 2022-03-09 16:30:55 root: INFO: instruction count: 44: __C542Compiled.__m552INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 35: __C542Compiled.__m553INPLACE_DECODE_r_struct_of_r_binaryANDr_int32ANDr_int32END_TO_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 31: __C542Compiled.__m554INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:55 root: INFO: instruction count: 10: __C542Compiled.__m555INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:55 root: INFO: instruction count: 10: __C542Compiled.__m556INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C551Tuple3. 2022-03-09 16:30:55 root: INFO: instruction count: 12: __C551Tuple3. 2022-03-09 16:30:55 root: INFO: encoder cache miss (0 hits, 8 misses, 0.000) 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C561HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C561HailClassLoaderContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C562FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C562FSContainer. 2022-03-09 16:30:55 root: INFO: instruction count: 3: __C563etypeEncode. 2022-03-09 16:30:55 root: INFO: instruction count: 19: __C563etypeEncode.apply 2022-03-09 16:30:55 root: INFO: instruction count: 92: __C563etypeEncode.__m564ENCODE_SIndexablePointer_TO_o_array_of_o_struct_of_o_struct_of_o_binaryANDo_int32ANDo_int32ENDANDo_struct_of_o_binaryANDo_int32ANDo_int32ENDANDo_boolANDo_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 61: __C563etypeEncode.__m565ENCODE_SIntervalPointer_TO_o_struct_of_o_struct_of_o_binaryANDo_int32ANDo_int32ENDANDo_struct_of_o_binaryANDo_int32ANDo_int32ENDANDo_boolANDo_boolEND 2022-03-09 16:30:55 root: INFO: instruction count: 53: __C563etypeEncode.__m566ENCODE_SBaseStructPointer_TO_o_struct_of_o_binaryANDo_int32ANDo_int32END 2022-03-09 16:30:55 root: INFO: instruction count: 16: __C563etypeEncode.__m567ENCODE_SStringPointer_TO_o_binary 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C563etypeEncode.__m568ENCODE_SInt32$_TO_o_int32 2022-03-09 16:30:55 root: INFO: instruction count: 4: __C563etypeEncode.__m569ENCODE_SBoolean$_TO_o_bool 2022-03-09 16:30:55 root: INFO: finished execution of query hail_query_3 2022-03-09 16:30:55 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode total 963.737ms self 12.221ms children 951.516ms %children 98.73% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 2.655ms self 0.011ms children 2.644ms %children 99.58% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 2.626ms self 1.139ms children 1.487ms %children 56.62% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 1.487ms self 0.074ms children 1.413ms %children 95.03% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.319ms self 0.319ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.098ms self 0.098ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.301ms self 0.301ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.565ms self 0.565ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.042ms self 0.004ms children 0.037ms %children 89.39% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 1.230ms self 0.007ms children 1.223ms %children 99.40% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 1.214ms self 0.529ms children 0.685ms %children 56.41% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.685ms self 0.030ms children 0.655ms %children 95.63% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.188ms self 0.188ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.300ms self 0.300ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.069ms self 0.004ms children 0.064ms %children 93.99% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 911.326ms self 0.012ms children 911.314ms %children 100.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 911.280ms self 859.950ms children 51.330ms %children 5.63% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 0.834ms self 0.012ms children 0.822ms %children 98.58% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 0.806ms self 0.369ms children 0.437ms %children 54.22% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.437ms self 0.025ms children 0.412ms %children 94.37% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.135ms self 0.135ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.081ms self 0.081ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.043ms self 0.005ms children 0.039ms %children 88.71% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.589ms self 0.019ms children 0.570ms %children 96.70% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.563ms self 0.272ms children 0.291ms %children 51.72% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.291ms self 0.048ms children 0.243ms %children 83.47% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.033ms self 0.004ms children 0.029ms %children 87.05% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.047ms self 0.004ms children 0.043ms %children 91.08% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.617ms self 0.007ms children 0.610ms %children 98.86% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.603ms self 0.335ms children 0.268ms %children 44.45% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.268ms self 0.023ms children 0.245ms %children 91.59% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 45.783ms self 42.282ms children 3.501ms %children 7.65% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 1.255ms self 0.009ms children 1.246ms %children 99.29% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 1.234ms self 0.527ms children 0.707ms %children 57.27% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.707ms self 0.040ms children 0.667ms %children 94.37% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.263ms self 0.263ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.033ms self 0.005ms children 0.028ms %children 84.80% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 0.472ms self 0.007ms children 0.465ms %children 98.50% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.459ms self 0.249ms children 0.210ms %children 45.81% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.210ms self 0.014ms children 0.197ms %children 93.53% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.606ms self 0.006ms children 0.599ms %children 98.94% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.582ms self 0.582ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.669ms self 0.007ms children 0.662ms %children 98.89% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.654ms self 0.316ms children 0.338ms %children 51.65% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.338ms self 0.016ms children 0.322ms %children 95.14% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.104ms self 0.104ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.112ms self 0.112ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.466ms self 0.466ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 3.381ms self 3.381ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.812ms self 0.033ms children 0.779ms %children 95.97% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.769ms self 0.358ms children 0.411ms %children 53.48% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.411ms self 0.043ms children 0.368ms %children 89.51% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.129ms self 0.129ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile total 31.128ms self 26.397ms children 4.731ms %children 15.20% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR total 1.214ms self 0.008ms children 1.206ms %children 99.31% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 1.193ms self 0.432ms children 0.762ms %children 63.83% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.762ms self 0.033ms children 0.729ms %children 95.69% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.108ms self 0.108ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.106ms self 0.106ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.227ms self 0.227ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.222ms self 0.222ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR total 0.032ms self 0.004ms children 0.028ms %children 88.06% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/LoweringTransformation total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR total 0.867ms self 0.005ms children 0.862ms %children 99.37% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.855ms self 0.426ms children 0.429ms %children 50.13% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.429ms self 0.016ms children 0.413ms %children 96.34% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs total 1.219ms self 0.005ms children 1.214ms %children 99.59% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 1.196ms self 1.196ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.770ms self 0.006ms children 0.765ms %children 99.27% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.757ms self 0.332ms children 0.424ms %children 56.06% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.424ms self 0.016ms children 0.409ms %children 96.31% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.174ms self 0.174ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.097ms self 0.097ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/Compile/EmitContext.analyze total 0.629ms self 0.629ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 4.251ms self 4.251ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:55 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: timing SparkBackend.parse_table_ir total 552.376ms self 552.376ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: JSON: JObject(List((name,JString(NPartitionsTable)))) 2022-03-09 16:30:56 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: timing SparkBackend.parse_value_ir total 3.365ms self 3.365ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: starting execution of query hail_query_4 of initial size 2 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht (IntervalRead: 20 intervals, filter=false)))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht (IntervalRead: 20 intervals, filter=false)))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht (IntervalRead: 20 intervals, filter=false)))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_rep_on_read_test.ht (IntervalRead: 20 intervals, filter=false)))) 2022-03-09 16:30:56 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:56 root: INFO: lowering result: TableToValueApply 2022-03-09 16:30:56 root: INFO: compiling and evaluating result: TableToValueApply 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 9: (Let (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0)) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] ))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] ))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] ))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] ))) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] ))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String,i:Int32,j:Int32},Int32]]}] )))) 2022-03-09 16:30:56 root: INFO: encoder cache miss (0 hits, 9 misses, 0.000) 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C596HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C596HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C597FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C597FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C598etypeEncode. 2022-03-09 16:30:56 root: INFO: instruction count: 7: __C598etypeEncode.apply 2022-03-09 16:30:56 root: INFO: instruction count: 25: __C598etypeEncode.__m599ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 35: __C598etypeEncode.__m600ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 51: __C598etypeEncode.__m601ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 16: __C598etypeEncode.__m602ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:56 root: INFO: instruction count: 41: __C598etypeEncode.__m603ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:56 root: INFO: instruction count: 21: __C598etypeEncode.__m604ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 37: __C598etypeEncode.__m605ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_int32ANDr_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C598etypeEncode.__m606ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C598etypeEncode.__m607ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:56 MemoryStore: INFO: Block broadcast_11 stored as values in memory (estimated size 5.5 KiB, free 12.6 GiB) 2022-03-09 16:30:56 MemoryStore: INFO: Block broadcast_11_piece0 stored as bytes in memory (estimated size 515.0 B, free 12.6 GiB) 2022-03-09 16:30:56 BlockManagerInfo: INFO: Added broadcast_11_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 515.0 B, free: 12.6 GiB) 2022-03-09 16:30:56 SparkContext: INFO: Created broadcast 11 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C570HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C570HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C571FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C571FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C572Compiled. 2022-03-09 16:30:56 root: INFO: instruction count: 25: __C572Compiled.apply 2022-03-09 16:30:56 root: INFO: instruction count: 22: __C572Compiled.__m573split_StreamLen 2022-03-09 16:30:56 root: INFO: instruction count: 9: __C572Compiled.setPartitionIndex 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C572Compiled.addPartitionRegion 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C572Compiled.setPool 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C572Compiled.addHailClassLoader 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C572Compiled.addFS 2022-03-09 16:30:56 root: INFO: instruction count: 55: __C572Compiled.addLiterals 2022-03-09 16:30:56 root: INFO: instruction count: 27: __C572Compiled.__m587DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:56 root: INFO: instruction count: 58: __C572Compiled.__m588INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringANDr_int32ANDr_int32ENDANDr_int32ENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 35: __C572Compiled.__m589INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringANDr_int32ANDr_int32ENDANDr_int32ENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 31: __C572Compiled.__m590INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:56 root: INFO: instruction count: 44: __C572Compiled.__m591INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringANDr_int32ANDr_int32ENDANDr_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 26: __C572Compiled.__m592INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryANDr_int32ANDr_int32ENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringANDr_int32ANDr_int32ENDANDr_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 35: __C572Compiled.__m593INPLACE_DECODE_r_struct_of_r_binaryANDr_int32ANDr_int32END_TO_r_struct_of_r_stringANDr_int32ANDr_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 10: __C572Compiled.__m594INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:56 root: INFO: instruction count: 10: __C572Compiled.__m595INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:56 root: INFO: took 172.581ms 2022-03-09 16:30:56 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:56 root: INFO: lowering result: I32 2022-03-09 16:30:56 root: INFO: compiling and evaluating result: I32 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (I32 20)) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:30:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:30:56 root: INFO: encoder cache miss (0 hits, 10 misses, 0.000) 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C618HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C618HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C619FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C619FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C620etypeEncode. 2022-03-09 16:30:56 root: INFO: instruction count: 7: __C620etypeEncode.apply 2022-03-09 16:30:56 root: INFO: instruction count: 9: __C620etypeEncode.__m621ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_int32ENDEND 2022-03-09 16:30:56 root: INFO: instruction count: 13: __C620etypeEncode.__m622ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C620etypeEncode.__m623ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:56 MemoryStore: INFO: Block broadcast_12 stored as values in memory (estimated size 80.0 B, free 12.6 GiB) 2022-03-09 16:30:56 MemoryStore: INFO: Block broadcast_12_piece0 stored as bytes in memory (estimated size 61.0 B, free 12.6 GiB) 2022-03-09 16:30:56 BlockManagerInfo: INFO: Added broadcast_12_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 61.0 B, free: 12.6 GiB) 2022-03-09 16:30:56 SparkContext: INFO: Created broadcast 12 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C608HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C608HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C609FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C609FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C610Compiled. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C610Compiled.apply 2022-03-09 16:30:56 root: INFO: instruction count: 9: __C610Compiled.setPartitionIndex 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C610Compiled.addPartitionRegion 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C610Compiled.setPool 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C610Compiled.addHailClassLoader 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C610Compiled.addFS 2022-03-09 16:30:56 root: INFO: instruction count: 35: __C610Compiled.addLiterals 2022-03-09 16:30:56 root: INFO: instruction count: 27: __C610Compiled.__m615DECODE_r_struct_of_r_struct_of_r_int32ENDEND_TO_SBaseStructPointer 2022-03-09 16:30:56 root: INFO: instruction count: 17: __C610Compiled.__m616INPLACE_DECODE_r_struct_of_r_int32END_TO_r_tuple_of_r_int32END 2022-03-09 16:30:56 root: INFO: instruction count: 10: __C610Compiled.__m617INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:56 root: INFO: took 24.345ms 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I32 20) 2022-03-09 16:30:56 root: INFO: encoder cache miss (0 hits, 11 misses, 0.000) 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C624HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C624HailClassLoaderContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C625FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C625FSContainer. 2022-03-09 16:30:56 root: INFO: instruction count: 3: __C626etypeEncode. 2022-03-09 16:30:56 root: INFO: instruction count: 11: __C626etypeEncode.apply 2022-03-09 16:30:56 root: INFO: instruction count: 4: __C626etypeEncode.__m627ENCODE_SInt32$_TO_o_int32 2022-03-09 16:30:56 root: INFO: finished execution of query hail_query_4 2022-03-09 16:30:56 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode total 205.969ms self 4.819ms children 201.150ms %children 97.66% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 1.832ms self 0.012ms children 1.820ms %children 99.34% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 1.804ms self 0.850ms children 0.954ms %children 52.89% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.954ms self 0.040ms children 0.914ms %children 95.80% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.098ms self 0.098ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.223ms self 0.223ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.410ms self 0.410ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.040ms self 0.003ms children 0.037ms %children 91.72% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 1.342ms self 0.006ms children 1.337ms %children 99.58% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 1.330ms self 0.602ms children 0.728ms %children 54.75% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.728ms self 0.028ms children 0.700ms %children 96.21% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.369ms self 0.369ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.066ms self 0.003ms children 0.063ms %children 95.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 197.186ms self 0.008ms children 197.178ms %children 100.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 197.158ms self 110.111ms children 87.047ms %children 44.15% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 3.541ms self 0.013ms children 3.528ms %children 99.64% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.163ms self 0.163ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 3.355ms self 1.499ms children 1.856ms %children 55.32% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 1.856ms self 0.059ms children 1.797ms %children 96.81% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.182ms self 0.182ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.656ms self 0.656ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.196ms self 0.196ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.091ms self 0.006ms children 0.085ms %children 93.02% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.923ms self 0.007ms children 0.916ms %children 99.27% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.906ms self 0.430ms children 0.476ms %children 52.57% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.476ms self 0.016ms children 0.461ms %children 96.74% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.182ms self 0.182ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.098ms self 0.098ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.047ms self 0.005ms children 0.042ms %children 89.68% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.042ms self 0.004ms children 0.038ms %children 90.42% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.931ms self 0.006ms children 0.925ms %children 99.32% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.916ms self 0.391ms children 0.525ms %children 57.32% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.525ms self 0.017ms children 0.508ms %children 96.71% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.202ms self 0.202ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.129ms self 0.129ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 54.501ms self 46.948ms children 7.553ms %children 13.86% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 1.542ms self 0.009ms children 1.533ms %children 99.41% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 1.517ms self 0.643ms children 0.873ms %children 57.58% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.873ms self 0.058ms children 0.815ms %children 93.34% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.106ms self 0.106ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.301ms self 0.301ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.248ms self 0.248ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.051ms self 0.006ms children 0.045ms %children 88.55% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 1.415ms self 0.007ms children 1.407ms %children 99.48% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.395ms self 0.592ms children 0.803ms %children 57.59% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.803ms self 0.022ms children 0.782ms %children 97.31% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.375ms self 0.375ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.187ms self 0.187ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.991ms self 0.007ms children 0.984ms %children 99.29% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.955ms self 0.955ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 2.809ms self 0.008ms children 2.801ms %children 99.71% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 2.785ms self 0.756ms children 2.029ms %children 72.84% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 2.029ms self 0.026ms children 2.003ms %children 98.74% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.120ms self 0.120ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.407ms self 1.407ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.277ms self 0.277ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.745ms self 0.745ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 2.982ms self 2.982ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 0.434ms self 0.007ms children 0.427ms %children 98.38% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 0.417ms self 0.161ms children 0.256ms %children 61.40% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.256ms self 0.011ms children 0.245ms %children 95.65% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.029ms self 0.003ms children 0.026ms %children 91.28% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.208ms self 0.004ms children 0.204ms %children 98.12% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.201ms self 0.091ms children 0.111ms %children 54.99% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.111ms self 0.007ms children 0.104ms %children 94.01% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.019ms self 0.003ms children 0.016ms %children 86.13% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.016ms self 0.002ms children 0.014ms %children 87.08% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.199ms self 0.004ms children 0.195ms %children 98.14% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.193ms self 0.091ms children 0.102ms %children 52.99% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.102ms self 0.006ms children 0.096ms %children 93.95% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 20.820ms self 19.404ms children 1.416ms %children 6.80% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 0.689ms self 0.004ms children 0.685ms %children 99.35% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 0.679ms self 0.244ms children 0.435ms %children 64.08% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.435ms self 0.015ms children 0.421ms %children 96.63% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.228ms self 0.228ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.018ms self 0.003ms children 0.015ms %children 85.04% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 0.237ms self 0.004ms children 0.233ms %children 98.31% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.230ms self 0.114ms children 0.116ms %children 50.43% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.116ms self 0.008ms children 0.108ms %children 93.47% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.108ms self 0.003ms children 0.105ms %children 97.25% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.271ms self 0.004ms children 0.267ms %children 98.52% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.264ms self 0.137ms children 0.127ms %children 48.06% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.127ms self 0.008ms children 0.119ms %children 93.96% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 2.253ms self 2.253ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.477ms self 0.007ms children 0.470ms %children 98.57% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.464ms self 0.194ms children 0.270ms %children 58.19% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.270ms self 0.012ms children 0.258ms %children 95.70% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/Compile total 0.167ms self 0.167ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:30:56 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:57 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:57 root: INFO: timing SparkBackend.parse_table_ir total 315.740ms self 315.740ms children 0.000ms %children 0.00% 2022-03-09 16:30:57 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:57 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:30:57 root: INFO: timing SparkBackend.parse_value_ir total 6.232ms self 6.232ms children 0.000ms %children 0.00% 2022-03-09 16:30:57 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:30:57 root: INFO: starting execution of query hail_query_5 of initial size 103 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 103: (TableCount (TableMapRows (TableMapRows (TableFilter (TableKeyBy (transcript i j) True (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableFilter (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht )) (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len) (Ref row)) None (start_idx (ToArray (StreamFlatMap (ToStream False (ToArray (StreamRange 6 False (I32 0) (Ref __cse_1) (I32 500)))) (ToStream False (ToArray (StreamMap (ToStream False (ToArray (StreamRange 8 False (I32 0) (Ref __cse_1) (I32 500)))) (MakeStruct (i_start (Ref __uid_10)) (j_start (Ref __uid_11))))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx) (Ref row)) None (i (GetField i_start (Ref __uid_12))) (j (GetField j_start (Ref __uid_12))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j) (Ref row)) None (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j i_max_idx j_max_idx) (Ref row)) None (start_idx (InsertFields (Ref __cse_3) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __cse_3)) (GetField j_start (Ref __cse_3))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))) 2022-03-09 16:30:57 root: INFO: generated TableFilterIntervals node with 1 intervals: Intervals: [[ENST00000217939]-[ENST00000217939]] Predicate: (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False)) Post: (Coalesce (True) (False)) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_726) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_726) (I32 500)) (MakeStruct (i_start (Ref __iruid_727)) (j_start (Ref __iruid_728))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_730))) (j (GetField j_start (Ref __iruid_730)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_726) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_726) (I32 500)) (MakeStruct (i_start (Ref __iruid_727)) (j_start (Ref __iruid_728))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_730))) (j (GetField j_start (Ref __iruid_730)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 41: (TableCount (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead Table{global:Struct{},key:[transcript],row:Struct{transcript:String,list_len:Int32}} False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_753) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_753) (I32 500)) (MakeStruct (i_start (Ref __iruid_754)) (j_start (Ref __iruid_755))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_757))) (j (GetField j_start (Ref __iruid_757)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) 2022-03-09 16:30:57 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:30:57 root: INFO: lowering result: TableCount 2022-03-09 16:30:57 root: INFO: compiling and evaluating result: TableCount 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 77: (ApplyIR -1 sum () Int64 (Let (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0)) (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (MakeStruct (__iruid_762 (Ref __iruid_762))) (Let (GetField __iruid_762 (Ref __iruid_769)) (Cast Int64 (StreamLen (Let (Ref __iruid_762) (StreamFilter (Let (Ref __iruid_762) (StreamMap (StreamFlatMap (Let (Ref __iruid_762) (StreamMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_763)) (Let (Ref __iruid_764) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_753) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_753) (I32 500)) (MakeStruct (i_start (Ref __iruid_754)) (j_start (Ref __iruid_755)))))))))))) (Let (GetField start_idx (Ref __iruid_765)) (StreamMap (ToStream False (Ref __iruid_766)) (InsertFields (Ref __iruid_765) None (start_idx (Ref __iruid_767)))))) (Let (Ref __iruid_768) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_757))) (j (GetField j_start (Ref __iruid_757)))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))))))))) 2022-03-09 16:30:57 root: INFO: Prune: MakeStruct: eliminating field '__iruid_762' 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_821)) (Let (GetField list_len (Ref __iruid_824)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_825) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_825) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_826)) (j_start (Ref __iruid_827))) (InsertFields (Ref __iruid_824) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_828)) (i (GetField i_start (Ref __iruid_828))) (j (GetField j_start (Ref __iruid_828))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_823)) (GetField i (Ref __iruid_823))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_819) (Ref __iruid_820))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_821)) (Let (GetField list_len (Ref __iruid_824)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_825) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_825) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_826)) (j_start (Ref __iruid_827))) (InsertFields (Ref __iruid_824) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_828)) (i (GetField i_start (Ref __iruid_828))) (j (GetField j_start (Ref __iruid_828))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_823)) (GetField i (Ref __iruid_823))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_819) (Ref __iruid_820))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_851)) (Let (GetField list_len (Ref __iruid_854)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_855) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_855) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_856)) (j_start (Ref __iruid_857))) (InsertFields (Ref __iruid_854) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_858)) (i (GetField i_start (Ref __iruid_858))) (j (GetField j_start (Ref __iruid_858))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_853)) (GetField i (Ref __iruid_853))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_849) (Ref __iruid_850))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_851)) (Let (GetField list_len (Ref __iruid_854)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_855) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_855) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_856)) (j_start (Ref __iruid_857))) (InsertFields (Ref __iruid_854) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_858)) (i (GetField i_start (Ref __iruid_858))) (j (GetField j_start (Ref __iruid_858))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_853)) (GetField i (Ref __iruid_853))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_849) (Ref __iruid_850))) 2022-03-09 16:30:57 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 47: (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_881)) (Let (GetField list_len (Ref __iruid_884)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_885) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_885) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_886)) (j_start (Ref __iruid_887))) (InsertFields (Ref __iruid_884) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_888)) (i (GetField i_start (Ref __iruid_888))) (j (GetField j_start (Ref __iruid_888))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_883)) (GetField i (Ref __iruid_883))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_879) (Ref __iruid_880))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_881)) (Let (GetField list_len (Ref __iruid_884)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_885) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_885) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_886)) (j_start (Ref __iruid_887))) (InsertFields (Ref __iruid_884) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_888)) (i (GetField i_start (Ref __iruid_888))) (j (GetField j_start (Ref __iruid_888))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_883)) (GetField i (Ref __iruid_883))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_879) (Ref __iruid_880)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_911)) (Let (GetField list_len (Ref __iruid_914)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_915) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_915) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_916)) (j_start (Ref __iruid_917))) (InsertFields (Ref __iruid_914) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_918)) (i (GetField i_start (Ref __iruid_918))) (j (GetField j_start (Ref __iruid_918))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_913)) (GetField i (Ref __iruid_913))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_909) (Ref __iruid_910)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_911)) (Let (GetField list_len (Ref __iruid_914)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_915) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_915) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_916)) (j_start (Ref __iruid_917))) (InsertFields (Ref __iruid_914) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_918)) (i (GetField i_start (Ref __iruid_918))) (j (GetField j_start (Ref __iruid_918))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_913)) (GetField i (Ref __iruid_913))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_909) (Ref __iruid_910)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_941)) (Let (GetField list_len (Ref __iruid_944)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_945) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_945) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_946)) (j_start (Ref __iruid_947))) (InsertFields (Ref __iruid_944) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_948)) (i (GetField i_start (Ref __iruid_948))) (j (GetField j_start (Ref __iruid_948))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_943)) (GetField i (Ref __iruid_943))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_939) (Ref __iruid_940)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_941)) (Let (GetField list_len (Ref __iruid_944)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_945) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_945) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_946)) (j_start (Ref __iruid_947))) (InsertFields (Ref __iruid_944) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_948)) (i (GetField i_start (Ref __iruid_948))) (j (GetField j_start (Ref __iruid_948))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_943)) (GetField i (Ref __iruid_943))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_939) (Ref __iruid_940)))) 2022-03-09 16:30:57 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 48: (MakeTuple (0) (StreamFold (ToStream False (CollectDistributedArray (ToStream False (Literal Array[Struct{partitionPath:String,indexPath:String,interval:Interval[Tuple[Struct{transcript:String},Int32]]}] )) (Literal Struct{} ) (Cast Int64 (StreamLen (StreamFilter (StreamFlatMap (ReadPartition Struct{transcript:String,list_len:Int32} "{\"name\":\"PartitionNativeReaderIndexed\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{transcript:EBinary,total_oe:EFloat64,max_idx:+EInt32,transcript_start:EInt32,transcript_end:EInt32,cum_obs:+EArray[EInt64],cum_exp:+EArray[EFloat64],positions:+EArray[EInt32],list_len:+EInt32}\",\"_vType\":\"Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"indexSpec\":{\"_relPath\":\"../index\",\"_leafCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{first_idx:+EInt64,keys:+EArray[+EBaseStruct{key:+EBaseStruct{transcript:EBinary},offset:+EInt64,annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{first_idx:Int64,keys:Array[Struct{key:Struct{transcript:String},offset:Int64,annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_internalNodeCodec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"EBaseStruct{children:+EArray[+EBaseStruct{index_file_offset:+EInt64,first_idx:+EInt64,first_key:+EBaseStruct{transcript:EBinary},first_record_offset:+EInt64,first_annotation:+EBaseStruct{}}]}\",\"_vType\":\"Struct{children:Array[Struct{index_file_offset:Int64,first_idx:Int64,first_key:Struct{transcript:String},first_record_offset:Int64,first_annotation:Struct{}}]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4HCBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}},\"_keyType\":\"Struct{transcript:String}\",\"_annotationType\":\"Struct{}\"},\"key\":[\"transcript\"]}" (Ref __iruid_971)) (Let (GetField list_len (Ref __iruid_974)) (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_975) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_975) (I32 500)) (Let (MakeStruct (i_start (Ref __iruid_976)) (j_start (Ref __iruid_977))) (InsertFields (Ref __iruid_974) ("transcript" "list_len" "start_idx" "i" "j") (start_idx (Ref __iruid_978)) (i (GetField i_start (Ref __iruid_978))) (j (GetField j_start (Ref __iruid_978))))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref __iruid_973)) (GetField i (Ref __iruid_973))) (False))))))) (I64 0) (ApplyBinaryPrimOp Add (Ref __iruid_969) (Ref __iruid_970)))) 2022-03-09 16:30:57 root: INFO: decoder cache hit 2022-03-09 16:30:57 root: INFO: decoder cache hit 2022-03-09 16:30:57 root: INFO: decoder cache hit 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C642HailClassLoaderContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C642HailClassLoaderContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C643FSContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C643FSContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C644collect_distributed_array. 2022-03-09 16:30:57 root: INFO: instruction count: 114: __C644collect_distributed_array.apply 2022-03-09 16:30:57 root: INFO: instruction count: 17: __C644collect_distributed_array.apply 2022-03-09 16:30:57 root: INFO: instruction count: 27: __C644collect_distributed_array.__m645DECODE_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:57 root: INFO: instruction count: 35: __C644collect_distributed_array.__m646INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 31: __C644collect_distributed_array.__m647INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:57 root: INFO: instruction count: 44: __C644collect_distributed_array.__m648INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:57 root: INFO: instruction count: 26: __C644collect_distributed_array.__m649INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:57 root: INFO: instruction count: 17: __C644collect_distributed_array.__m650INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:57 root: INFO: instruction count: 10: __C644collect_distributed_array.__m651INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:57 root: INFO: instruction count: 10: __C644collect_distributed_array.__m652INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:57 root: INFO: instruction count: 27: __C644collect_distributed_array.__m654DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-03-09 16:30:57 root: INFO: instruction count: 8: __C644collect_distributed_array.__m655INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:57 root: INFO: instruction count: 280: __C644collect_distributed_array.__m657split_StreamLen 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C644collect_distributed_array.setObjects 2022-03-09 16:30:57 root: INFO: instruction count: 11: __C644collect_distributed_array.__m689ord_gteq 2022-03-09 16:30:57 root: INFO: instruction count: 14: __C644collect_distributed_array.__m690ord_gteqNonnull 2022-03-09 16:30:57 root: INFO: instruction count: 12: __C644collect_distributed_array.__m694setup_null 2022-03-09 16:30:57 root: INFO: instruction count: 13: __C644collect_distributed_array.__m696ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64END 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C644collect_distributed_array.__m697ENCODE_SInt64$_TO_r_int64 2022-03-09 16:30:57 root: INFO: instruction count: 9: __C644collect_distributed_array.setPartitionIndex 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C644collect_distributed_array.addPartitionRegion 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C644collect_distributed_array.setPool 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C644collect_distributed_array.addHailClassLoader 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C644collect_distributed_array.addFS 2022-03-09 16:30:57 root: INFO: instruction count: 366: __C644collect_distributed_array.__m657split_StreamLen_region0_2 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C700__m657split_StreamLenSpills. 2022-03-09 16:30:57 root: INFO: encoder cache hit 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_13 stored as values in memory (estimated size 336.0 B, free 12.6 GiB) 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_13_piece0 stored as bytes in memory (estimated size 209.0 B, free 12.6 GiB) 2022-03-09 16:30:57 BlockManagerInfo: INFO: Added broadcast_13_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 209.0 B, free: 12.6 GiB) 2022-03-09 16:30:57 SparkContext: INFO: Created broadcast 13 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C628HailClassLoaderContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C628HailClassLoaderContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C629FSContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C629FSContainer. 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C630Compiled. 2022-03-09 16:30:57 root: INFO: instruction count: 25: __C630Compiled.apply 2022-03-09 16:30:57 root: INFO: instruction count: 326: __C630Compiled.__m631split_StreamFold 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.setBackend 2022-03-09 16:30:57 root: INFO: instruction count: 9: __C630Compiled.__m707ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 51: __C630Compiled.__m708ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 16: __C630Compiled.__m709ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:30:57 root: INFO: instruction count: 41: __C630Compiled.__m710ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND 2022-03-09 16:30:57 root: INFO: instruction count: 21: __C630Compiled.__m711ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDANDr_int32END 2022-03-09 16:30:57 root: INFO: instruction count: 13: __C630Compiled.__m712ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.__m713ENCODE_SInt32$_TO_r_int32 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.__m714ENCODE_SBoolean$_TO_r_bool 2022-03-09 16:30:57 root: INFO: instruction count: 9: __C630Compiled.__m715ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 1: __C630Compiled.__m716ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.setObjects 2022-03-09 16:30:57 root: INFO: instruction count: 27: __C630Compiled.__m718DECODE_r_struct_of_r_int64END_TO_SBaseStructPointer 2022-03-09 16:30:57 root: INFO: instruction count: 10: __C630Compiled.__m719INPLACE_DECODE_r_int64_TO_r_int64 2022-03-09 16:30:57 root: INFO: instruction count: 9: __C630Compiled.setPartitionIndex 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.addPartitionRegion 2022-03-09 16:30:57 root: INFO: instruction count: 4: __C630Compiled.setPool 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C630Compiled.addHailClassLoader 2022-03-09 16:30:57 root: INFO: instruction count: 3: __C630Compiled.addFS 2022-03-09 16:30:57 root: INFO: instruction count: 62: __C630Compiled.addLiterals 2022-03-09 16:30:57 root: INFO: instruction count: 36: __C630Compiled.__m731DECODE_r_struct_of_r_struct_of_ENDANDr_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDENDEND_TO_SBaseStructPointer 2022-03-09 16:30:57 root: INFO: instruction count: 8: __C630Compiled.__m732INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-03-09 16:30:57 root: INFO: instruction count: 58: __C630Compiled.__m733INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 35: __C630Compiled.__m734INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_struct_of_r_stringANDr_stringANDr_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32ENDEND 2022-03-09 16:30:57 root: INFO: instruction count: 31: __C630Compiled.__m735INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:30:57 root: INFO: instruction count: 44: __C630Compiled.__m736INPLACE_DECODE_r_struct_of_r_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_struct_of_r_struct_of_r_binaryENDANDr_int32ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:57 root: INFO: instruction count: 26: __C630Compiled.__m737INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryENDANDr_int32END_TO_r_tuple_of_r_struct_of_r_stringENDANDr_int32END 2022-03-09 16:30:57 root: INFO: instruction count: 17: __C630Compiled.__m738INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:30:57 root: INFO: instruction count: 10: __C630Compiled.__m739INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:30:57 root: INFO: instruction count: 10: __C630Compiled.__m740INPLACE_DECODE_r_bool_TO_r_bool 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_14 stored as values in memory (estimated size 48.0 B, free 12.6 GiB) 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_14_piece0 stored as bytes in memory (estimated size 49.0 B, free 12.6 GiB) 2022-03-09 16:30:57 BlockManagerInfo: INFO: Added broadcast_14_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:30:57 SparkContext: INFO: Created broadcast 14 from broadcast at SparkBackend.scala:280 2022-03-09 16:30:57 SparkContext: INFO: Starting job: collect at SparkBackend.scala:288 2022-03-09 16:30:57 DAGScheduler: INFO: Got job 3 (collect at SparkBackend.scala:288) with 1 output partitions 2022-03-09 16:30:57 DAGScheduler: INFO: Final stage: ResultStage 3 (collect at SparkBackend.scala:288) 2022-03-09 16:30:57 DAGScheduler: INFO: Parents of final stage: List() 2022-03-09 16:30:57 DAGScheduler: INFO: Missing parents: List() 2022-03-09 16:30:57 DAGScheduler: INFO: Submitting ResultStage 3 (SparkBackendComputeRDD[6] at RDD at SparkBackend.scala:720), which has no missing parents 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_15 stored as values in memory (estimated size 41.7 KiB, free 12.6 GiB) 2022-03-09 16:30:57 MemoryStore: INFO: Block broadcast_15_piece0 stored as bytes in memory (estimated size 21.1 KiB, free 12.6 GiB) 2022-03-09 16:30:57 BlockManagerInfo: INFO: Added broadcast_15_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 21.1 KiB, free: 12.6 GiB) 2022-03-09 16:30:57 SparkContext: INFO: Created broadcast 15 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:30:57 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 3 (SparkBackendComputeRDD[6] at RDD at SparkBackend.scala:720) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:30:57 YarnScheduler: INFO: Adding task set 3.0 with 1 tasks resource profile 0 2022-03-09 16:30:57 FairSchedulableBuilder: INFO: Added task set TaskSet_3.0 tasks to pool default 2022-03-09 16:30:57 TaskSetManager: INFO: Starting task 0.0 in stage 3.0 (TID 3) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, PROCESS_LOCAL, 4563 bytes) taskResourceAssignments Map() 2022-03-09 16:30:57 BlockManagerInfo: INFO: Added broadcast_15_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 21.1 KiB, free: 2.9 GiB) 2022-03-09 16:30:57 BlockManagerInfo: INFO: Added broadcast_14_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:31:00 TaskSetManager: INFO: Finished task 0.0 in stage 3.0 (TID 3) in 3068 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/1) 2022-03-09 16:31:00 YarnScheduler: INFO: Removed TaskSet 3.0, whose tasks have all completed, from pool default 2022-03-09 16:31:00 DAGScheduler: INFO: ResultStage 3 (collect at SparkBackend.scala:288) finished in 3.078 s 2022-03-09 16:31:00 DAGScheduler: INFO: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:31:00 YarnScheduler: INFO: Killing all running tasks in stage 3: Stage finished 2022-03-09 16:31:00 DAGScheduler: INFO: Job 3 finished: collect at SparkBackend.scala:288, took 3.082537 s 2022-03-09 16:31:00 root: INFO: took 3.493s 2022-03-09 16:31:00 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:31:00 root: INFO: lowering result: I64 2022-03-09 16:31:00 root: INFO: compiling and evaluating result: I64 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: took 2.099ms 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I64 120) 2022-03-09 16:31:00 root: INFO: encoder cache hit 2022-03-09 16:31:00 root: INFO: finished execution of query hail_query_5 2022-03-09 16:31:00 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode total 3.541s self 4.357ms children 3.537s %children 99.88% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 26.367ms self 0.011ms children 26.356ms %children 99.96% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 26.291ms self 3.220ms children 23.071ms %children 87.75% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 23.071ms self 0.104ms children 22.967ms %children 99.55% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.257ms self 0.257ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.600ms self 1.600ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 5.882ms self 5.882ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.385ms self 1.385ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.123ms self 0.123ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.714ms self 2.714ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.386ms self 0.386ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.518ms self 0.518ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.700ms self 0.700ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.951ms self 1.951ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.137ms self 0.137ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.930ms self 1.930ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.190ms self 0.190ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.359ms self 0.359ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.435ms self 0.435ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.886ms self 1.886ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.137ms self 0.137ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.375ms self 2.375ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.172ms self 0.006ms children 0.166ms %children 96.67% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.130ms self 0.130ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 14.683ms self 0.007ms children 14.676ms %children 99.95% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 14.638ms self 1.879ms children 12.760ms %children 87.17% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 12.760ms self 0.080ms children 12.680ms %children 99.38% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.151ms self 0.151ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.445ms self 0.445ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.414ms self 0.414ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.395ms self 1.395ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.150ms self 0.150ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 2.143ms self 2.143ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.182ms self 0.182ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.341ms self 0.341ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.427ms self 0.427ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.241ms self 1.241ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.138ms self 0.138ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 1.814ms self 1.814ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.159ms self 0.159ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.300ms self 0.300ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.398ms self 0.398ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.198ms self 1.198ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.104ms self 0.104ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 1.681ms self 1.681ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.172ms self 0.005ms children 0.167ms %children 97.14% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.110ms self 0.110ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 3.495s self 0.009ms children 3.495s %children 100.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 3.495s self 198.250ms children 3.297s %children 94.33% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 30.103ms self 0.017ms children 30.086ms %children 99.94% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.149ms self 0.149ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 29.908ms self 7.244ms children 22.663ms %children 75.78% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 22.663ms self 0.075ms children 22.589ms %children 99.67% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.403ms self 0.403ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.320ms self 0.320ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.671ms self 2.671ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.147ms self 3.147ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.340ms self 0.340ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.965ms self 2.965ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.367ms self 0.367ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.233ms self 0.233ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 4.594ms self 4.594ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.841ms self 1.841ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.534ms self 1.534ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.303ms self 0.303ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.208ms self 0.208ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.551ms self 0.551ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.235ms self 1.235ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.130ms self 0.130ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.602ms self 1.602ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.302ms self 0.008ms children 0.294ms %children 97.39% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.232ms self 0.232ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 18.714ms self 0.012ms children 18.702ms %children 99.94% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 18.637ms self 4.232ms children 14.405ms %children 77.29% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 14.405ms self 0.073ms children 14.332ms %children 99.50% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.238ms self 0.238ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.288ms self 0.288ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.669ms self 0.669ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.827ms self 1.827ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.124ms self 0.124ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 1.342ms self 1.342ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.209ms self 0.209ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.163ms self 0.163ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.527ms self 0.527ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.846ms self 1.846ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.214ms self 0.214ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 2.001ms self 2.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.391ms self 0.391ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.257ms self 0.257ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.366ms self 0.366ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.419ms self 1.419ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.183ms self 0.183ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 2.267ms self 2.267ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.247ms self 0.008ms children 0.239ms %children 96.67% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.170ms self 0.170ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.309ms self 0.008ms children 0.300ms %children 97.33% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.184ms self 0.184ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 21.081ms self 0.009ms children 21.072ms %children 99.96% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 21.015ms self 4.044ms children 16.971ms %children 80.76% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 16.971ms self 0.084ms children 16.886ms %children 99.50% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.241ms self 0.241ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.286ms self 0.286ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.627ms self 0.627ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 1.948ms self 1.948ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.182ms self 0.182ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 2.371ms self 2.371ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.336ms self 0.336ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.264ms self 0.264ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.605ms self 0.605ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 1.899ms self 1.899ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.195ms self 0.195ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 2.138ms self 2.138ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.340ms self 0.340ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.288ms self 0.288ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.594ms self 0.594ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 2.674ms self 2.674ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.195ms self 0.195ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 1.703ms self 1.703ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 124.666ms self 61.184ms children 63.482ms %children 50.92% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 19.783ms self 0.014ms children 19.769ms %children 99.93% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 19.705ms self 4.256ms children 15.449ms %children 78.40% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 15.449ms self 0.085ms children 15.364ms %children 99.45% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.465ms self 0.465ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.254ms self 0.254ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.864ms self 0.864ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.772ms self 1.772ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.253ms self 0.253ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.139ms self 2.139ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.356ms self 0.356ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.230ms self 0.230ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.462ms self 0.462ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.702ms self 1.702ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.171ms self 0.171ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.587ms self 1.587ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.387ms self 0.387ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.244ms self 0.244ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.481ms self 0.481ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.811ms self 1.811ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.172ms self 0.172ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.015ms self 2.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.177ms self 0.007ms children 0.170ms %children 96.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.078ms self 0.078ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 18.320ms self 0.009ms children 18.311ms %children 99.95% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 18.264ms self 3.614ms children 14.651ms %children 80.21% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 14.651ms self 0.081ms children 14.570ms %children 99.45% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.222ms self 0.222ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.391ms self 0.391ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.474ms self 1.474ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.220ms self 0.220ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.832ms self 1.832ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.417ms self 0.417ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.222ms self 0.222ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.470ms self 0.470ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.907ms self 1.907ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.205ms self 0.205ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.126ms self 2.126ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.353ms self 0.353ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.226ms self 0.226ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.410ms self 0.410ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 2.435ms self 2.435ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.130ms self 0.130ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.362ms self 1.362ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 3.026ms self 0.010ms children 3.016ms %children 99.68% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 2.884ms self 2.884ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 20.165ms self 0.011ms children 20.155ms %children 99.95% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 20.081ms self 4.166ms children 15.915ms %children 79.25% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 15.915ms self 0.093ms children 15.822ms %children 99.42% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.360ms self 0.360ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.348ms self 0.348ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.602ms self 0.602ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.999ms self 1.999ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.172ms self 0.172ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 2.551ms self 2.551ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.292ms self 0.292ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.277ms self 0.277ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.432ms self 0.432ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 3.351ms self 3.351ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.338ms self 0.338ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.660ms self 1.660ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.259ms self 0.259ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.164ms self 0.164ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.424ms self 0.424ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.181ms self 1.181ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.318ms self 1.318ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 2.011ms self 2.011ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 2.812ms self 2.812ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 3.097s self 3.097s children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 0.503ms self 0.015ms children 0.488ms %children 96.94% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 0.476ms self 0.180ms children 0.296ms %children 62.18% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.296ms self 0.013ms children 0.283ms %children 95.53% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.105ms self 0.105ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.135ms self 0.004ms children 0.131ms %children 96.80% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.278ms self 0.005ms children 0.272ms %children 98.14% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.269ms self 0.145ms children 0.124ms %children 46.11% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.124ms self 0.009ms children 0.115ms %children 92.70% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.076ms self 0.005ms children 0.071ms %children 93.32% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.096ms self 0.009ms children 0.087ms %children 90.50% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.345ms self 0.008ms children 0.338ms %children 97.81% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.332ms self 0.154ms children 0.178ms %children 53.65% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.178ms self 0.014ms children 0.165ms %children 92.26% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 0.252ms self 0.252ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.384ms self 0.030ms children 0.354ms %children 92.30% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.348ms self 0.169ms children 0.178ms %children 51.31% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.178ms self 0.016ms children 0.163ms %children 91.17% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/Compile total 0.140ms self 0.140ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:31:00 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:00 root: INFO: timing SparkBackend.parse_value_ir total 7.689ms self 7.689ms children 0.000ms %children 0.00% 2022-03-09 16:31:00 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:31:00 root: INFO: starting execution of query hail_query_6 of initial size 104 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 104: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_repart_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":\"{\\n \\\"name\\\": \\\"LEB128BufferSpec\\\",\\n \\\"child\\\": {\\n \\\"name\\\": \\\"BlockingBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"LZ4FastBlockBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"StreamBlockBufferSpec\\\"\\n }\\n }\\n }\\n}\"}" (TableRepartition 120 0 (TableMapRows (TableMapRows (TableFilter (TableKeyBy (transcript i j) True (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableFilter (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht )) (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len) (Ref row)) None (start_idx (ToArray (StreamFlatMap (ToStream False (ToArray (StreamRange 6 False (I32 0) (Ref __cse_1) (I32 500)))) (ToStream False (ToArray (StreamMap (ToStream False (ToArray (StreamRange 8 False (I32 0) (Ref __cse_1) (I32 500)))) (MakeStruct (i_start (Ref __uid_10)) (j_start (Ref __uid_11))))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx) (Ref row)) None (i (GetField i_start (Ref __uid_12))) (j (GetField j_start (Ref __uid_12))))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False))) (Let (GetField list_len (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j) (Ref row)) None (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __cse_2) (I32 1))))))) (Let (GetField start_idx (Ref row)) (InsertFields (SelectFields (transcript total_oe max_idx transcript_start transcript_end cum_obs cum_exp positions list_len start_idx i j i_max_idx j_max_idx) (Ref row)) None (start_idx (InsertFields (Ref __cse_3) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __cse_3)) (GetField j_start (Ref __cse_3))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row)))))))))))) 2022-03-09 16:31:00 root: INFO: generated TableFilterIntervals node with 1 intervals: Intervals: [[ENST00000217939]-[ENST00000217939]] Predicate: (Coalesce (ApplyComparisonOp EQ (GetField transcript (Ref row)) (Str "ENST000002...")) (False)) Post: (Coalesce (True) (False)) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 84: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_repart_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":\"{\\n \\\"name\\\": \\\"LEB128BufferSpec\\\",\\n \\\"child\\\": {\\n \\\"name\\\": \\\"BlockingBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"LZ4FastBlockBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"StreamBlockBufferSpec\\\"\\n }\\n }\\n }\\n}\"}" (TableRepartition 120 0 (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1011) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1011) (I32 500)) (MakeStruct (i_start (Ref __iruid_1012)) (j_start (Ref __iruid_1013))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_1015))) (j (GetField j_start (Ref __iruid_1015)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1016) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1016) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1017) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1017)) (GetField j_start (Ref __iruid_1017))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))))) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 84: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_repart_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":\"{\\n \\\"name\\\": \\\"LEB128BufferSpec\\\",\\n \\\"child\\\": {\\n \\\"name\\\": \\\"BlockingBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"LZ4FastBlockBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"StreamBlockBufferSpec\\\"\\n }\\n }\\n }\\n}\"}" (TableRepartition 120 0 (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1011) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1011) (I32 500)) (MakeStruct (i_start (Ref __iruid_1012)) (j_start (Ref __iruid_1013))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_1015))) (j (GetField j_start (Ref __iruid_1015)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1016) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1016) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1017) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1017)) (GetField j_start (Ref __iruid_1017))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))))) 2022-03-09 16:31:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 84: (TableWrite "{\"name\":\"TableNativeWriter\",\"path\":\"gs://gnomad-tmp/kc/mxra5_repart_test.ht\",\"overwrite\":true,\"stageLocally\":false,\"codecSpecJSONStr\":\"{\\n \\\"name\\\": \\\"LEB128BufferSpec\\\",\\n \\\"child\\\": {\\n \\\"name\\\": \\\"BlockingBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"LZ4FastBlockBufferSpec\\\",\\n \\\"blockSize\\\": 32768,\\n \\\"child\\\": {\\n \\\"name\\\": \\\"StreamBlockBufferSpec\\\"\\n }\\n }\\n }\\n}\"}" (TableRepartition 120 0 (TableMapRows (TableKeyBy (transcript i j) True (TableFilter (TableMapRows (TableExplode ("start_idx") (TableMapRows (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/test_100_over5k.ht (IntervalRead: 1 intervals, filter=true))) (Let (GetField list_len (Ref row)) (InsertFields (Ref row) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1044) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1044) (I32 500)) (MakeStruct (i_start (Ref __iruid_1045)) (j_start (Ref __iruid_1046))))))))))) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) None (i (GetField i_start (Ref __iruid_1048))) (j (GetField j_start (Ref __iruid_1048)))))) (Coalesce (ApplyComparisonOp GTEQ (GetField j (Ref row)) (GetField i (Ref row))) (False)))) (Let (GetField list_len (Ref row)) (Let (GetField start_idx (Ref row)) (InsertFields (Ref row) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1049) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (Ref row)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1049) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1050) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1050)) (GetField j_start (Ref __iruid_1050))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (Ref row))) (I32 1)) (GetField j_start (GetField start_idx (Ref row))))))))))))) 2022-03-09 16:31:00 root: INFO: LowerOrInterpretNonCompilable: cannot efficiently lower query: TableRepartition has no lowered implementation 2022-03-09 16:31:00 root: INFO: interpreting non-compilable result: TableWrite 2022-03-09 16:31:01 root: INFO: decoder cache hit 2022-03-09 16:31:01 root: INFO: decoder cache hit 2022-03-09 16:31:01 root: INFO: decoder cache hit 2022-03-09 16:31:01 root: INFO: decoder cache hit 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1044) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1044) (I32 500)) (MakeStruct (i_start (Ref __iruid_1045)) (j_start (Ref __iruid_1046))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1061) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1061) (I32 500)) (MakeStruct (i_start (Ref __iruid_1062)) (j_start (Ref __iruid_1063))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1061) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1061) (I32 500)) (MakeStruct (i_start (Ref __iruid_1062)) (j_start (Ref __iruid_1063))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1070) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1070) (I32 500)) (MakeStruct (i_start (Ref __iruid_1071)) (j_start (Ref __iruid_1072))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1070) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1070) (I32 500)) (MakeStruct (i_start (Ref __iruid_1071)) (j_start (Ref __iruid_1072))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 22: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32})) 1) None (start_idx (ToArray (StreamFlatMap (StreamRange 6 False (I32 0) (Ref __iruid_1079) (I32 500)) (StreamMap (StreamRange 8 False (I32 0) (Ref __iruid_1079) (I32 500)) (MakeStruct (i_start (Ref __iruid_1080)) (j_start (Ref __iruid_1081))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Array[Struct{i_start:Int32,j_start:Int32}]} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: encoder cache miss (2 hits, 12 misses, 0.143) 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C777HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C777HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C778FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C778FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C779etypeEncode. 2022-03-09 16:31:01 root: INFO: instruction count: 7: __C779etypeEncode.apply 2022-03-09 16:31:01 root: INFO: instruction count: 13: __C779etypeEncode.__m780ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:31:01 root: INFO: instruction count: 16: __C779etypeEncode.__m781ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_16 stored as values in memory (estimated size 128.0 B, free 12.6 GiB) 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_16_piece0 stored as bytes in memory (estimated size 113.0 B, free 12.6 GiB) 2022-03-09 16:31:01 BlockManagerInfo: INFO: Added broadcast_16_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 113.0 B, free: 12.6 GiB) 2022-03-09 16:31:01 SparkContext: INFO: Created broadcast 16 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C741HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C741HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C742FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C742FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C743Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 328: __C743Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 284: __C743Compiled.__m749split_ToArray 2022-03-09 16:31:01 root: INFO: instruction count: 12: __C743Compiled.__m766setup_jab 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C743Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C743Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C743Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C743Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C743Compiled.addFS 2022-03-09 16:31:01 root: INFO: instruction count: 39: __C743Compiled.addLiterals 2022-03-09 16:31:01 root: INFO: instruction count: 27: __C743Compiled.__m775DECODE_r_struct_of_r_binaryEND_TO_SBaseStructPointer 2022-03-09 16:31:01 root: INFO: instruction count: 31: __C743Compiled.__m776INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 6: (Coalesce (ArrayLen (CastToArray (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)))) (I32 0)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 5: (Coalesce (ArrayLen (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (I32 0)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 5: (Coalesce (ArrayLen (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (I32 0)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 5: (Coalesce (ArrayLen (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (I32 0)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 5: (Coalesce (ArrayLen (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (I32 0)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 5: (Coalesce (ArrayLen (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (I32 0)) 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C782HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C782HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C783FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C783FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C784Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 32: __C784Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C784Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C784Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C784Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C784Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C784Compiled.addFS 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 7: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (CastToArray (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0))) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 6: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 6: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 6: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 6: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 6: (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0) None (start_idx (ArrayRef -1 (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCArray[+PCStruct{i_start:+PInt32,j_start:+PInt32}]})) 0)) (In SingleCodeEmitParamType(true, Int32SingleCodeType) 1)))) 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C790HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C790HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C791FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C791FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C792Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 342: __C792Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 35: __C792Compiled.__m796arrayref_bounds_check 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C792Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C792Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C792Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C792Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C792Compiled.addFS 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1048))) (j (GetField j_start (Ref __iruid_1048))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1085))) (j (GetField j_start (Ref __iruid_1085))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1085))) (j (GetField j_start (Ref __iruid_1085))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1088))) (j (GetField j_start (Ref __iruid_1088))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1088))) (j (GetField j_start (Ref __iruid_1088))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 12: (Coalesce (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32}})) 1) None (i (GetField i_start (Ref __iruid_1091))) (j (GetField j_start (Ref __iruid_1091))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: encoder cache hit 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_17 stored as values in memory (estimated size 128.0 B, free 12.6 GiB) 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_17_piece0 stored as bytes in memory (estimated size 113.0 B, free 12.6 GiB) 2022-03-09 16:31:01 BlockManagerInfo: INFO: Added broadcast_17_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 113.0 B, free: 12.6 GiB) 2022-03-09 16:31:01 SparkContext: INFO: Created broadcast 17 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C800HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C800HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C801FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C801FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C802Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 349: __C802Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C802Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C802Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C802Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C802Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C802Compiled.addFS 2022-03-09 16:31:01 root: INFO: instruction count: 39: __C802Compiled.addLiterals 2022-03-09 16:31:01 root: INFO: instruction count: 27: __C802Compiled.__m812DECODE_r_struct_of_r_binaryEND_TO_SBaseStructPointer 2022-03-09 16:31:01 root: INFO: instruction count: 31: __C802Compiled.__m813INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 9: (Coalesce (Coalesce (ApplyComparisonOp GTEQ (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0)) (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 0))) (False)) (False)) 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C814HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C814HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C815FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C815FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C816Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 55: __C816Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 11: __C816Compiled.__m821ord_gteq 2022-03-09 16:31:01 root: INFO: instruction count: 14: __C816Compiled.__m822ord_gteqNonnull 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C816Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C816Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C816Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C816Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C816Compiled.addFS 2022-03-09 16:31:01 root: INFO: encoder cache miss (3 hits, 13 misses, 0.188) 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C826HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C826HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C827FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C827FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C828etypeEncode. 2022-03-09 16:31:01 root: INFO: instruction count: 7: __C828etypeEncode.apply 2022-03-09 16:31:01 root: INFO: instruction count: 13: __C828etypeEncode.__m829ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32END 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C828etypeEncode.__m830ENCODE_SInt32$_TO_r_int32 2022-03-09 16:31:01 root: INFO: BroadcastRegionValue.broadcast: broadcasting 1 byte arrays of total size 17 (17.00 B 2022-03-09 16:31:01 root: INFO: decoder cache hit 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_18 stored as values in memory (estimated size 3.1 KiB, free 12.6 GiB) 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_18_piece0 stored as bytes in memory (estimated size 1725.0 B, free 12.6 GiB) 2022-03-09 16:31:01 BlockManagerInfo: INFO: Added broadcast_18_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 1725.0 B, free: 12.6 GiB) 2022-03-09 16:31:01 SparkContext: INFO: Created broadcast 18 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_19 stored as values in memory (estimated size 920.0 B, free 12.6 GiB) 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_19_piece0 stored as bytes in memory (estimated size 578.0 B, free 12.6 GiB) 2022-03-09 16:31:01 BlockManagerInfo: INFO: Added broadcast_19_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 578.0 B, free: 12.6 GiB) 2022-03-09 16:31:01 SparkContext: INFO: Created broadcast 19 from broadcast at RVDPartitioner.scala:93 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1049) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1049) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1050) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1050)) (GetField j_start (Ref __iruid_1050))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1097) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1097) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1098) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1098)) (GetField j_start (Ref __iruid_1098))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1097) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1097) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1098) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1098)) (GetField j_start (Ref __iruid_1098))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1103) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1103) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1104) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1104)) (GetField j_start (Ref __iruid_1104))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1103) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1103) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1104) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1104)) (GetField j_start (Ref __iruid_1104))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 43: (Coalesce (Let (GetField list_len (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (Let (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (InsertFields (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1) ("transcript" "total_oe" "max_idx" "transcript_start" "transcript_end" "cum_obs" "cum_exp" "positions" "list_len" "start_idx" "i" "j" "i_max_idx" "j_max_idx") (i_max_idx (ApplySpecial 9 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField i (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1109) (I32 1)))) (j_max_idx (ApplySpecial 10 min_ignore_missing () Int32 (ApplyBinaryPrimOp Add (GetField j (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)) (I32 500)) (ApplyBinaryPrimOp Subtract (Ref __iruid_1109) (I32 1)))) (start_idx (InsertFields (Ref __iruid_1110) None (j_start (If (ApplyComparisonOp EQ (GetField i_start (Ref __iruid_1110)) (GetField j_start (Ref __iruid_1110))) (ApplyBinaryPrimOp Add (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1))) (I32 1)) (GetField j_start (GetField start_idx (In SingleCodeEmitParamType(true, PTypeReferenceSingleCodeType(+PCStruct{transcript:PCString,total_oe:PFloat64,max_idx:+PInt32,transcript_start:PInt32,transcript_end:PInt32,cum_obs:+PCArray[PInt64],cum_exp:+PCArray[PFloat64],positions:+PCArray[PInt32],list_len:+PInt32,start_idx:+PCStruct{i_start:+PInt32,j_start:+PInt32},i:+PInt32,j:+PInt32})) 1)))))))))) (Die Struct{transcript:String,total_oe:Float64,max_idx:Int32,transcript_start:Int32,transcript_end:Int32,cum_obs:Array[Int64],cum_exp:Array[Float64],positions:Array[Int32],list_len:Int32,start_idx:Struct{i_start:Int32,j_start:Int32},i:Int32,j:Int32,i_max_idx:Int32,j_max_idx:Int32} -1 (Str "Internal e..."))) 2022-03-09 16:31:01 root: INFO: encoder cache hit 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_20 stored as values in memory (estimated size 128.0 B, free 12.6 GiB) 2022-03-09 16:31:01 MemoryStore: INFO: Block broadcast_20_piece0 stored as bytes in memory (estimated size 113.0 B, free 12.6 GiB) 2022-03-09 16:31:01 BlockManagerInfo: INFO: Added broadcast_20_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 113.0 B, free: 12.6 GiB) 2022-03-09 16:31:01 SparkContext: INFO: Created broadcast 20 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C831HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C831HailClassLoaderContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C832FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C832FSContainer. 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C833Compiled. 2022-03-09 16:31:01 root: INFO: instruction count: 391: __C833Compiled.apply 2022-03-09 16:31:01 root: INFO: instruction count: 158: __C833Compiled.__m840split_InsertFields 2022-03-09 16:31:01 root: INFO: instruction count: 11: __C833Compiled.__m842ord_equiv 2022-03-09 16:31:01 root: INFO: instruction count: 14: __C833Compiled.__m843ord_equivNonnull 2022-03-09 16:31:01 root: INFO: instruction count: 9: __C833Compiled.setPartitionIndex 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C833Compiled.addPartitionRegion 2022-03-09 16:31:01 root: INFO: instruction count: 4: __C833Compiled.setPool 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C833Compiled.addHailClassLoader 2022-03-09 16:31:01 root: INFO: instruction count: 3: __C833Compiled.addFS 2022-03-09 16:31:01 root: INFO: instruction count: 39: __C833Compiled.addLiterals 2022-03-09 16:31:01 root: INFO: instruction count: 27: __C833Compiled.__m853DECODE_r_struct_of_r_binaryEND_TO_SBaseStructPointer 2022-03-09 16:31:01 root: INFO: instruction count: 31: __C833Compiled.__m854INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:31:02 root: INFO: encoder cache miss (4 hits, 14 misses, 0.222) 2022-03-09 16:31:02 root: INFO: instruction count: 3: __C855HailClassLoaderContainer. 2022-03-09 16:31:02 root: INFO: instruction count: 3: __C855HailClassLoaderContainer. 2022-03-09 16:31:02 root: INFO: instruction count: 3: __C856FSContainer. 2022-03-09 16:31:02 root: INFO: instruction count: 3: __C856FSContainer. 2022-03-09 16:31:02 root: INFO: instruction count: 3: __C857etypeEncode. 2022-03-09 16:31:02 root: INFO: instruction count: 7: __C857etypeEncode.apply 2022-03-09 16:31:02 root: INFO: instruction count: 290: __C857etypeEncode.__m858ENCODE_SBaseStructPointer_TO_r_struct_of_o_binaryANDo_float64ANDr_int32ANDo_int32ANDo_int32ANDr_array_of_o_int64ANDr_array_of_o_float64ANDr_array_of_o_int32ANDr_int32ANDr_struct_of_r_int32ANDr_int32ENDANDr_int32ANDr_int32ANDr_int32ANDr_int32END 2022-03-09 16:31:02 root: INFO: instruction count: 16: __C857etypeEncode.__m859ENCODE_SStringPointer_TO_o_binary 2022-03-09 16:31:02 root: INFO: instruction count: 4: __C857etypeEncode.__m860ENCODE_SFloat64$_TO_o_float64 2022-03-09 16:31:02 root: INFO: instruction count: 4: __C857etypeEncode.__m861ENCODE_SInt32$_TO_r_int32 2022-03-09 16:31:02 root: INFO: instruction count: 4: __C857etypeEncode.__m862ENCODE_SInt32$_TO_o_int32 2022-03-09 16:31:02 root: INFO: instruction count: 115: __C857etypeEncode.__m863ENCODE_SIndexablePointer_TO_r_array_of_o_int64 2022-03-09 16:31:02 root: INFO: instruction count: 4: __C857etypeEncode.__m864ENCODE_SInt64$_TO_o_int64 2022-03-09 16:31:02 root: INFO: instruction count: 115: __C857etypeEncode.__m865ENCODE_SIndexablePointer_TO_r_array_of_o_float64 2022-03-09 16:31:02 root: INFO: instruction count: 115: __C857etypeEncode.__m866ENCODE_SIndexablePointer_TO_r_array_of_o_int32 2022-03-09 16:31:02 root: INFO: instruction count: 25: __C857etypeEncode.__m867ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32ANDr_int32END 2022-03-09 16:31:02 root: INFO: decoder cache hit 2022-03-09 16:31:02 SparkContext: INFO: Starting job: runJob at ContextRDD.scala:238 2022-03-09 16:31:02 DAGScheduler: INFO: Registering RDD 20 (coalesce at RVD.scala:315) as input to shuffle 0 2022-03-09 16:31:02 DAGScheduler: INFO: Got job 4 (runJob at ContextRDD.scala:238) with 120 output partitions 2022-03-09 16:31:02 DAGScheduler: INFO: Final stage: ResultStage 5 (runJob at ContextRDD.scala:238) 2022-03-09 16:31:02 DAGScheduler: INFO: Parents of final stage: List(ShuffleMapStage 4) 2022-03-09 16:31:02 DAGScheduler: INFO: Missing parents: List(ShuffleMapStage 4) 2022-03-09 16:31:02 DAGScheduler: INFO: Submitting ShuffleMapStage 4 (MapPartitionsRDD[20] at coalesce at RVD.scala:315), which has no missing parents 2022-03-09 16:31:02 MemoryStore: INFO: Block broadcast_21 stored as values in memory (estimated size 84.9 KiB, free 12.6 GiB) 2022-03-09 16:31:02 MemoryStore: INFO: Block broadcast_21_piece0 stored as bytes in memory (estimated size 39.7 KiB, free 12.6 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_21_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 39.7 KiB, free: 12.6 GiB) 2022-03-09 16:31:02 SparkContext: INFO: Created broadcast 21 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:31:02 DAGScheduler: INFO: Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[20] at coalesce at RVD.scala:315) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:31:02 YarnScheduler: INFO: Adding task set 4.0 with 1 tasks resource profile 0 2022-03-09 16:31:02 FairSchedulableBuilder: INFO: Added task set TaskSet_4.0 tasks to pool default 2022-03-09 16:31:02 TaskSetManager: INFO: Starting task 0.0 in stage 4.0 (TID 4) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, PROCESS_LOCAL, 4695 bytes) taskResourceAssignments Map() 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_21_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 39.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_20_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 113.0 B, free: 2.9 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_18_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 1725.0 B, free: 2.9 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_17_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 113.0 B, free: 2.9 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_16_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 113.0 B, free: 2.9 GiB) 2022-03-09 16:31:02 BlockManagerInfo: INFO: Added broadcast_19_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 578.0 B, free: 2.9 GiB) 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 0.0 in stage 4.0 (TID 4) in 1341 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/1) 2022-03-09 16:31:03 YarnScheduler: INFO: Removed TaskSet 4.0, whose tasks have all completed, from pool default 2022-03-09 16:31:03 DAGScheduler: INFO: ShuffleMapStage 4 (coalesce at RVD.scala:315) finished in 1.368 s 2022-03-09 16:31:03 DAGScheduler: INFO: looking for newly runnable stages 2022-03-09 16:31:03 DAGScheduler: INFO: running: Set() 2022-03-09 16:31:03 DAGScheduler: INFO: waiting: Set(ResultStage 5) 2022-03-09 16:31:03 DAGScheduler: INFO: failed: Set() 2022-03-09 16:31:03 DAGScheduler: INFO: Submitting ResultStage 5 (MapPartitionsRDD[26] at mapPartitions at ContextRDD.scala:222), which has no missing parents 2022-03-09 16:31:03 MemoryStore: INFO: Block broadcast_22 stored as values in memory (estimated size 24.4 KiB, free 12.6 GiB) 2022-03-09 16:31:03 MemoryStore: INFO: Block broadcast_22_piece0 stored as bytes in memory (estimated size 12.4 KiB, free 12.6 GiB) 2022-03-09 16:31:03 BlockManagerInfo: INFO: Added broadcast_22_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 12.4 KiB, free: 12.6 GiB) 2022-03-09 16:31:03 SparkContext: INFO: Created broadcast 22 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:31:03 DAGScheduler: INFO: Submitting 120 missing tasks from ResultStage 5 (MapPartitionsRDD[26] at mapPartitions at ContextRDD.scala:222) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2022-03-09 16:31:03 YarnScheduler: INFO: Adding task set 5.0 with 120 tasks resource profile 0 2022-03-09 16:31:03 FairSchedulableBuilder: INFO: Added task set TaskSet_5.0 tasks to pool default 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 0.0 in stage 5.0 (TID 5) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 1.0 in stage 5.0 (TID 6) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 1, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 2.0 in stage 5.0 (TID 7) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 2, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 3.0 in stage 5.0 (TID 8) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 3, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 BlockManagerInfo: INFO: Added broadcast_22_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 12.4 KiB, free: 2.9 GiB) 2022-03-09 16:31:03 MapOutputTrackerMasterEndpoint: INFO: Asked to send map output locations for shuffle 0 to 10.128.0.79:48968 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 4.0 in stage 5.0 (TID 9) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 4, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 0.0 in stage 5.0 (TID 5) in 399 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 5.0 in stage 5.0 (TID 10) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 5, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 6.0 in stage 5.0 (TID 11) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 6, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 1.0 in stage 5.0 (TID 6) in 397 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (2/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 2.0 in stage 5.0 (TID 7) in 396 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (3/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 7.0 in stage 5.0 (TID 12) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 7, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 3.0 in stage 5.0 (TID 8) in 398 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (4/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 8.0 in stage 5.0 (TID 13) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 8, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 4.0 in stage 5.0 (TID 9) in 41 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (5/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 9.0 in stage 5.0 (TID 14) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 9, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 6.0 in stage 5.0 (TID 11) in 38 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (6/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 10.0 in stage 5.0 (TID 15) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 10, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 7.0 in stage 5.0 (TID 12) in 39 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (7/120) 2022-03-09 16:31:03 TaskSetManager: INFO: Starting task 11.0 in stage 5.0 (TID 16) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 11, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:03 TaskSetManager: INFO: Finished task 5.0 in stage 5.0 (TID 10) in 43 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (8/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 12.0 in stage 5.0 (TID 17) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 12, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 10.0 in stage 5.0 (TID 15) in 65 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (9/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 13.0 in stage 5.0 (TID 18) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 13, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 14.0 in stage 5.0 (TID 19) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 14, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 11.0 in stage 5.0 (TID 16) in 65 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (10/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 8.0 in stage 5.0 (TID 13) in 72 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (11/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 15.0 in stage 5.0 (TID 20) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 15, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 9.0 in stage 5.0 (TID 14) in 72 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (12/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 16.0 in stage 5.0 (TID 21) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 16, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 17.0 in stage 5.0 (TID 22) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 17, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 12.0 in stage 5.0 (TID 17) in 47 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (13/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 18.0 in stage 5.0 (TID 23) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 18, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 13.0 in stage 5.0 (TID 18) in 46 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (14/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 19.0 in stage 5.0 (TID 24) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 19, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 14.0 in stage 5.0 (TID 19) in 47 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (15/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 15.0 in stage 5.0 (TID 20) in 46 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (16/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 20.0 in stage 5.0 (TID 25) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 20, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 21.0 in stage 5.0 (TID 26) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 21, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 19.0 in stage 5.0 (TID 24) in 35 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (17/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 22.0 in stage 5.0 (TID 27) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 22, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 16.0 in stage 5.0 (TID 21) in 40 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (18/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 23.0 in stage 5.0 (TID 28) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 23, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 17.0 in stage 5.0 (TID 22) in 42 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (19/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 18.0 in stage 5.0 (TID 23) in 61 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (20/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 24.0 in stage 5.0 (TID 29) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 24, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 22.0 in stage 5.0 (TID 27) in 36 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (21/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 25.0 in stage 5.0 (TID 30) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 25, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 20.0 in stage 5.0 (TID 25) in 40 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (22/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 26.0 in stage 5.0 (TID 31) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 26, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 21.0 in stage 5.0 (TID 26) in 41 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (23/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 27.0 in stage 5.0 (TID 32) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 27, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 23.0 in stage 5.0 (TID 28) in 40 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (24/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 28.0 in stage 5.0 (TID 33) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 28, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 24.0 in stage 5.0 (TID 29) in 28 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (25/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 29.0 in stage 5.0 (TID 34) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 29, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 25.0 in stage 5.0 (TID 30) in 29 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (26/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 30.0 in stage 5.0 (TID 35) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 30, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 26.0 in stage 5.0 (TID 31) in 29 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (27/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 31.0 in stage 5.0 (TID 36) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 31, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 27.0 in stage 5.0 (TID 32) in 30 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (28/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 32.0 in stage 5.0 (TID 37) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 32, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 28.0 in stage 5.0 (TID 33) in 30 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (29/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 33.0 in stage 5.0 (TID 38) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 33, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 30.0 in stage 5.0 (TID 35) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (30/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 34.0 in stage 5.0 (TID 39) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 34, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 29.0 in stage 5.0 (TID 34) in 31 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (31/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 35.0 in stage 5.0 (TID 40) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 35, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 31.0 in stage 5.0 (TID 36) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (32/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 36.0 in stage 5.0 (TID 41) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 36, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 35.0 in stage 5.0 (TID 40) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (33/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 37.0 in stage 5.0 (TID 42) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 37, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 33.0 in stage 5.0 (TID 38) in 30 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (34/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 38.0 in stage 5.0 (TID 43) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 38, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 34.0 in stage 5.0 (TID 39) in 29 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (35/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 32.0 in stage 5.0 (TID 37) in 34 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (36/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 39.0 in stage 5.0 (TID 44) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 39, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 40.0 in stage 5.0 (TID 45) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 40, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 36.0 in stage 5.0 (TID 41) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (37/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 41.0 in stage 5.0 (TID 46) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 41, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 42.0 in stage 5.0 (TID 47) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 42, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 38.0 in stage 5.0 (TID 43) in 25 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (38/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 37.0 in stage 5.0 (TID 42) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (39/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 43.0 in stage 5.0 (TID 48) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 43, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 39.0 in stage 5.0 (TID 44) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (40/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 44.0 in stage 5.0 (TID 49) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 44, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 45.0 in stage 5.0 (TID 50) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 45, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 41.0 in stage 5.0 (TID 46) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (41/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 46.0 in stage 5.0 (TID 51) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 46, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 42.0 in stage 5.0 (TID 47) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (42/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 40.0 in stage 5.0 (TID 45) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (43/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 47.0 in stage 5.0 (TID 52) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 47, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 43.0 in stage 5.0 (TID 48) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (44/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 48.0 in stage 5.0 (TID 53) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 48, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 44.0 in stage 5.0 (TID 49) in 25 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (45/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 49.0 in stage 5.0 (TID 54) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 49, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 46.0 in stage 5.0 (TID 51) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (46/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 50.0 in stage 5.0 (TID 55) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 50, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 45.0 in stage 5.0 (TID 50) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (47/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 51.0 in stage 5.0 (TID 56) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 51, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 47.0 in stage 5.0 (TID 52) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (48/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 52.0 in stage 5.0 (TID 57) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 52, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 48.0 in stage 5.0 (TID 53) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (49/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 53.0 in stage 5.0 (TID 58) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 53, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 49.0 in stage 5.0 (TID 54) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (50/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 54.0 in stage 5.0 (TID 59) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 54, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 51.0 in stage 5.0 (TID 56) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (51/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 55.0 in stage 5.0 (TID 60) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 55, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 50.0 in stage 5.0 (TID 55) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (52/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 56.0 in stage 5.0 (TID 61) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 56, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 57.0 in stage 5.0 (TID 62) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 57, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 58.0 in stage 5.0 (TID 63) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 58, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 53.0 in stage 5.0 (TID 58) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (53/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 54.0 in stage 5.0 (TID 59) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (54/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 52.0 in stage 5.0 (TID 57) in 30 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (55/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 59.0 in stage 5.0 (TID 64) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 59, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 55.0 in stage 5.0 (TID 60) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (56/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 60.0 in stage 5.0 (TID 65) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 60, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 61.0 in stage 5.0 (TID 66) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 61, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 57.0 in stage 5.0 (TID 62) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (57/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 62.0 in stage 5.0 (TID 67) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 62, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 58.0 in stage 5.0 (TID 63) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (58/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 56.0 in stage 5.0 (TID 61) in 29 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (59/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 63.0 in stage 5.0 (TID 68) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 63, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 59.0 in stage 5.0 (TID 64) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (60/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 64.0 in stage 5.0 (TID 69) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 64, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 65.0 in stage 5.0 (TID 70) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 65, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 60.0 in stage 5.0 (TID 65) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (61/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 66.0 in stage 5.0 (TID 71) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 66, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 62.0 in stage 5.0 (TID 67) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (62/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 63.0 in stage 5.0 (TID 68) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (63/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 61.0 in stage 5.0 (TID 66) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (64/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 67.0 in stage 5.0 (TID 72) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 67, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 68.0 in stage 5.0 (TID 73) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 68, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 64.0 in stage 5.0 (TID 69) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (65/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 69.0 in stage 5.0 (TID 74) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 69, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 66.0 in stage 5.0 (TID 71) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (66/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 70.0 in stage 5.0 (TID 75) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 70, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 65.0 in stage 5.0 (TID 70) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (67/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 71.0 in stage 5.0 (TID 76) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 71, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 67.0 in stage 5.0 (TID 72) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (68/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 72.0 in stage 5.0 (TID 77) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 72, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 68.0 in stage 5.0 (TID 73) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (69/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 69.0 in stage 5.0 (TID 74) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (70/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 73.0 in stage 5.0 (TID 78) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 73, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 74.0 in stage 5.0 (TID 79) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 74, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 70.0 in stage 5.0 (TID 75) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (71/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 75.0 in stage 5.0 (TID 80) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 75, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 71.0 in stage 5.0 (TID 76) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (72/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 76.0 in stage 5.0 (TID 81) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 76, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 77.0 in stage 5.0 (TID 82) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 77, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 73.0 in stage 5.0 (TID 78) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (73/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 72.0 in stage 5.0 (TID 77) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (74/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 78.0 in stage 5.0 (TID 83) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 78, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 74.0 in stage 5.0 (TID 79) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (75/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 79.0 in stage 5.0 (TID 84) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 79, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 75.0 in stage 5.0 (TID 80) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (76/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 80.0 in stage 5.0 (TID 85) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 80, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 76.0 in stage 5.0 (TID 81) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (77/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 81.0 in stage 5.0 (TID 86) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 81, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 77.0 in stage 5.0 (TID 82) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (78/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 82.0 in stage 5.0 (TID 87) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 82, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 78.0 in stage 5.0 (TID 83) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (79/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 83.0 in stage 5.0 (TID 88) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 83, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 79.0 in stage 5.0 (TID 84) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (80/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 84.0 in stage 5.0 (TID 89) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 84, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 82.0 in stage 5.0 (TID 87) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (81/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 85.0 in stage 5.0 (TID 90) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 85, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 81.0 in stage 5.0 (TID 86) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (82/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 86.0 in stage 5.0 (TID 91) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 86, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 87.0 in stage 5.0 (TID 92) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 87, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 80.0 in stage 5.0 (TID 85) in 27 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (83/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 83.0 in stage 5.0 (TID 88) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (84/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 88.0 in stage 5.0 (TID 93) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 88, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 89.0 in stage 5.0 (TID 94) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 89, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 84.0 in stage 5.0 (TID 89) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (85/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 85.0 in stage 5.0 (TID 90) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (86/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 90.0 in stage 5.0 (TID 95) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 90, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 86.0 in stage 5.0 (TID 91) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (87/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 91.0 in stage 5.0 (TID 96) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 91, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 87.0 in stage 5.0 (TID 92) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (88/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 92.0 in stage 5.0 (TID 97) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 92, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 88.0 in stage 5.0 (TID 93) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (89/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 93.0 in stage 5.0 (TID 98) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 93, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 90.0 in stage 5.0 (TID 95) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (90/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 94.0 in stage 5.0 (TID 99) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 94, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 89.0 in stage 5.0 (TID 94) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (91/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 95.0 in stage 5.0 (TID 100) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 95, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 91.0 in stage 5.0 (TID 96) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (92/120) 2022-03-09 16:31:04 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:04 YarnAllocator: INFO: Driver requested a total number of 3 executor(s) for resource profile id: 0. 2022-03-09 16:31:04 YarnAllocator: INFO: Will request 1 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:04 ExecutorAllocationManager: INFO: Requesting 2 new executors because tasks are backlogged (new desired total will be 3 for resource profile id: 0) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 96.0 in stage 5.0 (TID 101) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 96, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 93.0 in stage 5.0 (TID 98) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (93/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 97.0 in stage 5.0 (TID 102) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 97, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 92.0 in stage 5.0 (TID 97) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (94/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 98.0 in stage 5.0 (TID 103) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 98, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 95.0 in stage 5.0 (TID 100) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (95/120) 2022-03-09 16:31:04 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 99.0 in stage 5.0 (TID 104) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 99, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 94.0 in stage 5.0 (TID 99) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (96/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 100.0 in stage 5.0 (TID 105) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 100, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 97.0 in stage 5.0 (TID 102) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (97/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 101.0 in stage 5.0 (TID 106) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 101, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 96.0 in stage 5.0 (TID 101) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (98/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 102.0 in stage 5.0 (TID 107) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 102, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 99.0 in stage 5.0 (TID 104) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (99/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 103.0 in stage 5.0 (TID 108) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 103, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 98.0 in stage 5.0 (TID 103) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (100/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 104.0 in stage 5.0 (TID 109) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 104, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 100.0 in stage 5.0 (TID 105) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (101/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 105.0 in stage 5.0 (TID 110) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 105, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 101.0 in stage 5.0 (TID 106) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (102/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 106.0 in stage 5.0 (TID 111) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 106, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 102.0 in stage 5.0 (TID 107) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (103/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 107.0 in stage 5.0 (TID 112) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 107, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 103.0 in stage 5.0 (TID 108) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (104/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 108.0 in stage 5.0 (TID 113) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 108, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 105.0 in stage 5.0 (TID 110) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (105/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 109.0 in stage 5.0 (TID 114) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 109, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 104.0 in stage 5.0 (TID 109) in 23 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (106/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 110.0 in stage 5.0 (TID 115) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 110, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 106.0 in stage 5.0 (TID 111) in 20 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (107/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 111.0 in stage 5.0 (TID 116) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 111, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 107.0 in stage 5.0 (TID 112) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (108/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 112.0 in stage 5.0 (TID 117) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 112, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 110.0 in stage 5.0 (TID 115) in 18 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (109/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 113.0 in stage 5.0 (TID 118) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 113, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 109.0 in stage 5.0 (TID 114) in 22 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (110/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 114.0 in stage 5.0 (TID 119) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 114, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 108.0 in stage 5.0 (TID 113) in 26 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (111/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 115.0 in stage 5.0 (TID 120) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 115, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 111.0 in stage 5.0 (TID 116) in 24 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (112/120) 2022-03-09 16:31:04 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:04 YarnAllocator: INFO: Driver requested a total number of 2 executor(s) for resource profile id: 0. 2022-03-09 16:31:04 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 2 executors. 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 116.0 in stage 5.0 (TID 121) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 116, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 112.0 in stage 5.0 (TID 117) in 18 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (113/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 117.0 in stage 5.0 (TID 122) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 117, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 113.0 in stage 5.0 (TID 118) in 19 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (114/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 118.0 in stage 5.0 (TID 123) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 118, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 114.0 in stage 5.0 (TID 119) in 19 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (115/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Starting task 119.0 in stage 5.0 (TID 124) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 119, NODE_LOCAL, 4558 bytes) taskResourceAssignments Map() 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 115.0 in stage 5.0 (TID 120) in 19 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (116/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 116.0 in stage 5.0 (TID 121) in 21 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (117/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 117.0 in stage 5.0 (TID 122) in 18 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (118/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 118.0 in stage 5.0 (TID 123) in 18 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (119/120) 2022-03-09 16:31:04 TaskSetManager: INFO: Finished task 119.0 in stage 5.0 (TID 124) in 19 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (120/120) 2022-03-09 16:31:04 YarnScheduler: INFO: Removed TaskSet 5.0, whose tasks have all completed, from pool default 2022-03-09 16:31:04 DAGScheduler: INFO: ResultStage 5 (runJob at ContextRDD.scala:238) finished in 1.211 s 2022-03-09 16:31:04 DAGScheduler: INFO: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:31:04 YarnScheduler: INFO: Killing all running tasks in stage 5: Stage finished 2022-03-09 16:31:04 DAGScheduler: INFO: Job 4 finished: runJob at ContextRDD.scala:238, took 2.631745 s 2022-03-09 16:31:04 MemoryStore: INFO: Block broadcast_23 stored as values in memory (estimated size 27.3 KiB, free 12.6 GiB) 2022-03-09 16:31:04 MemoryStore: INFO: Block broadcast_23_piece0 stored as bytes in memory (estimated size 1615.0 B, free 12.6 GiB) 2022-03-09 16:31:04 BlockManagerInfo: INFO: Added broadcast_23_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 1615.0 B, free: 12.6 GiB) 2022-03-09 16:31:04 SparkContext: INFO: Created broadcast 23 from broadcast at RVDPartitioner.scala:93 2022-03-09 16:31:04 root: INFO: encoder cache hit 2022-03-09 16:31:04 root: INFO: decoder cache hit 2022-03-09 16:31:04 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:04 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0. 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_8_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 11.6 KiB, free: 12.6 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_8_piece0 on kc-w-0.c.broad-mpg-gnomad.internal:41631 in memory (size: 11.6 KiB, free: 2.9 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_14_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 49.0 B, free: 12.6 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_14_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 49.0 B, free: 2.9 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_21_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 39.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_21_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 39.7 KiB, free: 12.6 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_15_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 21.1 KiB, free: 2.9 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_15_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 21.1 KiB, free: 12.6 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_22_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 12.4 KiB, free: 12.6 GiB) 2022-03-09 16:31:05 BlockManagerInfo: INFO: Removed broadcast_22_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 12.4 KiB, free: 2.9 GiB) 2022-03-09 16:31:07 root: INFO: encoder cache hit 2022-03-09 16:31:07 root: INFO: instruction count: 3: __C868HailClassLoaderContainer. 2022-03-09 16:31:07 root: INFO: instruction count: 3: __C868HailClassLoaderContainer. 2022-03-09 16:31:07 root: INFO: instruction count: 3: __C869FSContainer. 2022-03-09 16:31:07 root: INFO: instruction count: 3: __C869FSContainer. 2022-03-09 16:31:08 root: INFO: instruction count: 3: __C870indexwriter. 2022-03-09 16:31:08 root: INFO: instruction count: 141: __C870indexwriter.apply 2022-03-09 16:31:08 root: INFO: instruction count: 413: __C870indexwriter.__m879writeInternalNode 2022-03-09 16:31:08 root: INFO: instruction count: 25: __C870indexwriter.__m880ENCODE_SBaseStructPointer_TO_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 35: __C870indexwriter.__m881ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 53: __C870indexwriter.__m882ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64ANDr_int64ANDr_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 4: __C870indexwriter.__m883ENCODE_SInt64$_TO_r_int64 2022-03-09 16:31:08 root: INFO: instruction count: 60: __C870indexwriter.__m884ENCODE_SBaseStructPointer_TO_r_struct_of_o_binaryANDr_int32ANDr_int32END 2022-03-09 16:31:08 root: INFO: instruction count: 16: __C870indexwriter.__m885ENCODE_SStringPointer_TO_o_binary 2022-03-09 16:31:08 root: INFO: instruction count: 4: __C870indexwriter.__m886ENCODE_SInt32$_TO_r_int32 2022-03-09 16:31:08 root: INFO: instruction count: 1: __C870indexwriter.__m887ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-03-09 16:31:08 root: INFO: instruction count: 324: __C870indexwriter.__m889writeLeafNode 2022-03-09 16:31:08 root: INFO: instruction count: 37: __C870indexwriter.__m890ENCODE_SBaseStructPointer_TO_o_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 35: __C870indexwriter.__m891ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 29: __C870indexwriter.__m892ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_o_binaryANDr_int32ANDr_int32ENDANDr_int64ANDr_struct_of_ENDEND 2022-03-09 16:31:08 root: INFO: instruction count: 52: __C870indexwriter.__m893flush 2022-03-09 16:31:08 root: INFO: instruction count: 182: __C870indexwriter.init 2022-03-09 16:31:08 root: INFO: instruction count: 4: __C870indexwriter.setObjects 2022-03-09 16:31:08 root: INFO: instruction count: 28: __C870indexwriter.close 2022-03-09 16:31:08 root: INFO: instruction count: 7: __C870indexwriter.trackedOS 2022-03-09 16:31:08 root: INFO: instruction count: 9: __C870indexwriter.setPartitionIndex 2022-03-09 16:31:08 root: INFO: instruction count: 4: __C870indexwriter.addPartitionRegion 2022-03-09 16:31:08 root: INFO: instruction count: 4: __C870indexwriter.setPool 2022-03-09 16:31:08 root: INFO: instruction count: 3: __C870indexwriter.addHailClassLoader 2022-03-09 16:31:08 root: INFO: instruction count: 3: __C870indexwriter.addFS 2022-03-09 16:31:08 root: INFO: encoder cache hit 2022-03-09 16:31:08 SparkContext: INFO: Starting job: collect at ContextRDD.scala:176 2022-03-09 16:31:08 DAGScheduler: INFO: Registering RDD 30 (mapPartitions at ContextRDD.scala:168) as input to shuffle 1 2022-03-09 16:31:08 DAGScheduler: INFO: Got job 5 (collect at ContextRDD.scala:176) with 120 output partitions 2022-03-09 16:31:08 DAGScheduler: INFO: Final stage: ResultStage 7 (collect at ContextRDD.scala:176) 2022-03-09 16:31:08 DAGScheduler: INFO: Parents of final stage: List(ShuffleMapStage 6) 2022-03-09 16:31:08 DAGScheduler: INFO: Missing parents: List(ShuffleMapStage 6) 2022-03-09 16:31:08 DAGScheduler: INFO: Submitting ShuffleMapStage 6 (MapPartitionsRDD[30] at mapPartitions at ContextRDD.scala:168), which has no missing parents 2022-03-09 16:31:08 MemoryStore: INFO: Block broadcast_24 stored as values in memory (estimated size 87.8 KiB, free 12.6 GiB) 2022-03-09 16:31:08 MemoryStore: INFO: Block broadcast_24_piece0 stored as bytes in memory (estimated size 40.9 KiB, free 12.6 GiB) 2022-03-09 16:31:08 BlockManagerInfo: INFO: Added broadcast_24_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 40.9 KiB, free: 12.6 GiB) 2022-03-09 16:31:08 SparkContext: INFO: Created broadcast 24 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:31:08 DAGScheduler: INFO: Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[30] at mapPartitions at ContextRDD.scala:168) (first 15 tasks are for partitions Vector(0)) 2022-03-09 16:31:08 YarnScheduler: INFO: Adding task set 6.0 with 1 tasks resource profile 0 2022-03-09 16:31:08 FairSchedulableBuilder: INFO: Added task set TaskSet_6.0 tasks to pool default 2022-03-09 16:31:08 TaskSetManager: INFO: Starting task 0.0 in stage 6.0 (TID 125) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, PROCESS_LOCAL, 4695 bytes) taskResourceAssignments Map() 2022-03-09 16:31:08 BlockManagerInfo: INFO: Added broadcast_24_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 40.9 KiB, free: 2.9 GiB) 2022-03-09 16:31:08 BlockManagerInfo: INFO: Added broadcast_23_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 1615.0 B, free: 2.9 GiB) 2022-03-09 16:31:08 TaskSetManager: INFO: Finished task 0.0 in stage 6.0 (TID 125) in 607 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/1) 2022-03-09 16:31:08 YarnScheduler: INFO: Removed TaskSet 6.0, whose tasks have all completed, from pool default 2022-03-09 16:31:08 DAGScheduler: INFO: ShuffleMapStage 6 (mapPartitions at ContextRDD.scala:168) finished in 0.617 s 2022-03-09 16:31:08 DAGScheduler: INFO: looking for newly runnable stages 2022-03-09 16:31:08 DAGScheduler: INFO: running: Set() 2022-03-09 16:31:08 DAGScheduler: INFO: waiting: Set(ResultStage 7) 2022-03-09 16:31:08 DAGScheduler: INFO: failed: Set() 2022-03-09 16:31:08 DAGScheduler: INFO: Submitting ResultStage 7 (MapPartitionsRDD[37] at mapPartitions at ContextRDD.scala:168), which has no missing parents 2022-03-09 16:31:08 MemoryStore: INFO: Block broadcast_25 stored as values in memory (estimated size 142.3 KiB, free 12.6 GiB) 2022-03-09 16:31:08 MemoryStore: INFO: Block broadcast_25_piece0 stored as bytes in memory (estimated size 58.7 KiB, free 12.6 GiB) 2022-03-09 16:31:08 BlockManagerInfo: INFO: Added broadcast_25_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 58.7 KiB, free: 12.6 GiB) 2022-03-09 16:31:08 SparkContext: INFO: Created broadcast 25 from broadcast at DAGScheduler.scala:1388 2022-03-09 16:31:08 DAGScheduler: INFO: Submitting 120 missing tasks from ResultStage 7 (MapPartitionsRDD[37] at mapPartitions at ContextRDD.scala:168) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2022-03-09 16:31:08 YarnScheduler: INFO: Adding task set 7.0 with 120 tasks resource profile 0 2022-03-09 16:31:08 FairSchedulableBuilder: INFO: Added task set TaskSet_7.0 tasks to pool default 2022-03-09 16:31:08 TaskSetManager: INFO: Starting task 0.0 in stage 7.0 (TID 126) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 0, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:08 TaskSetManager: INFO: Starting task 1.0 in stage 7.0 (TID 127) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 1, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:08 TaskSetManager: INFO: Starting task 2.0 in stage 7.0 (TID 128) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 2, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:08 TaskSetManager: INFO: Starting task 3.0 in stage 7.0 (TID 129) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 3, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:08 BlockManagerInfo: INFO: Added broadcast_25_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:44485 (size: 58.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:09 MapOutputTrackerMasterEndpoint: INFO: Asked to send map output locations for shuffle 1 to 10.128.0.79:48968 2022-03-09 16:31:09 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:09 YarnAllocator: INFO: Driver requested a total number of 3 executor(s) for resource profile id: 0. 2022-03-09 16:31:09 ExecutorAllocationManager: INFO: Requesting 2 new executors because tasks are backlogged (new desired total will be 3 for resource profile id: 0) 2022-03-09 16:31:09 YarnAllocator: INFO: Will request 1 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:09 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:10 TaskSetManager: INFO: Starting task 4.0 in stage 7.0 (TID 130) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 4, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:10 TaskSetManager: INFO: Finished task 1.0 in stage 7.0 (TID 127) in 1402 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (1/120) 2022-03-09 16:31:10 TaskSetManager: INFO: Starting task 5.0 in stage 7.0 (TID 131) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 5, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:10 TaskSetManager: INFO: Starting task 6.0 in stage 7.0 (TID 132) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 6, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:10 TaskSetManager: INFO: Finished task 3.0 in stage 7.0 (TID 129) in 1460 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (2/120) 2022-03-09 16:31:10 TaskSetManager: INFO: Starting task 7.0 in stage 7.0 (TID 133) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 7, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:10 TaskSetManager: INFO: Finished task 2.0 in stage 7.0 (TID 128) in 1461 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (3/120) 2022-03-09 16:31:10 TaskSetManager: INFO: Finished task 0.0 in stage 7.0 (TID 126) in 1463 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (4/120) 2022-03-09 16:31:10 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:10 YarnAllocator: INFO: Driver requested a total number of 4 executor(s) for resource profile id: 0. 2022-03-09 16:31:10 ExecutorAllocationManager: INFO: Requesting 1 new executor because tasks are backlogged (new desired total will be 4 for resource profile id: 0) 2022-03-09 16:31:10 YarnAllocator: INFO: Will request 1 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:10 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:10 YarnAllocator: INFO: Launching container container_1646839922409_0005_01_000003 on host kc-w-0.c.broad-mpg-gnomad.internal for executor with ID 3 for ResourceProfile Id 0 2022-03-09 16:31:10 YarnAllocator: INFO: Received 1 containers from YARN, launching executors on 1 of them. 2022-03-09 16:31:11 TaskSetManager: INFO: Starting task 8.0 in stage 7.0 (TID 134) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 8, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:11 TaskSetManager: INFO: Finished task 5.0 in stage 7.0 (TID 131) in 732 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (5/120) 2022-03-09 16:31:11 TaskSetManager: INFO: Starting task 9.0 in stage 7.0 (TID 135) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 9, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:11 TaskSetManager: INFO: Finished task 7.0 in stage 7.0 (TID 133) in 730 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (6/120) 2022-03-09 16:31:11 TaskSetManager: INFO: Starting task 10.0 in stage 7.0 (TID 136) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 10, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:11 TaskSetManager: INFO: Finished task 4.0 in stage 7.0 (TID 130) in 803 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (7/120) 2022-03-09 16:31:11 TaskSetManager: INFO: Starting task 11.0 in stage 7.0 (TID 137) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 11, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:11 TaskSetManager: INFO: Finished task 6.0 in stage 7.0 (TID 132) in 915 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (8/120) 2022-03-09 16:31:11 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:11 YarnAllocator: INFO: Driver requested a total number of 6 executor(s) for resource profile id: 0. 2022-03-09 16:31:11 ExecutorAllocationManager: INFO: Requesting 2 new executors because tasks are backlogged (new desired total will be 6 for resource profile id: 0) 2022-03-09 16:31:11 YarnAllocator: INFO: Will request 2 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:11 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:11 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:11 YarnAllocator: INFO: Launching container container_1646839922409_0005_01_000004 on host kc-w-1.c.broad-mpg-gnomad.internal for executor with ID 4 for ResourceProfile Id 0 2022-03-09 16:31:11 YarnAllocator: INFO: Received 1 containers from YARN, launching executors on 1 of them. 2022-03-09 16:31:11 TaskSetManager: INFO: Starting task 12.0 in stage 7.0 (TID 138) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 12, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:11 TaskSetManager: INFO: Finished task 10.0 in stage 7.0 (TID 136) in 848 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (9/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 13.0 in stage 7.0 (TID 139) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 13, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 8.0 in stage 7.0 (TID 134) in 939 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (10/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 14.0 in stage 7.0 (TID 140) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 14, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 9.0 in stage 7.0 (TID 135) in 960 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (11/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 15.0 in stage 7.0 (TID 141) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 15, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 11.0 in stage 7.0 (TID 137) in 817 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (12/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 16.0 in stage 7.0 (TID 142) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 16, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 12.0 in stage 7.0 (TID 138) in 742 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (13/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 17.0 in stage 7.0 (TID 143) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 17, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 13.0 in stage 7.0 (TID 139) in 674 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (14/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 18.0 in stage 7.0 (TID 144) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 18, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 14.0 in stage 7.0 (TID 140) in 769 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (15/120) 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 19.0 in stage 7.0 (TID 145) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 19, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 20.0 in stage 7.0 (TID 146) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 20, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 21.0 in stage 7.0 (TID 147) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 21, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 22.0 in stage 7.0 (TID 148) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 22, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Starting task 23.0 in stage 7.0 (TID 149) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 23, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:12 TaskSetManager: INFO: Finished task 15.0 in stage 7.0 (TID 141) in 770 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (16/120) 2022-03-09 16:31:12 BlockManagerInfo: INFO: Added broadcast_25_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:41631 (size: 58.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:12 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:12 YarnAllocator: INFO: Driver requested a total number of 10 executor(s) for resource profile id: 0. 2022-03-09 16:31:12 ExecutorAllocationManager: INFO: Requesting 4 new executors because tasks are backlogged (new desired total will be 10 for resource profile id: 0) 2022-03-09 16:31:12 YarnAllocator: INFO: Will request 4 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:12 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:12 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:12 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:12 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 MapOutputTrackerMasterEndpoint: INFO: Asked to send map output locations for shuffle 1 to 10.128.0.78:37152 2022-03-09 16:31:13 TaskSetManager: INFO: Starting task 24.0 in stage 7.0 (TID 150) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 24, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:13 TaskSetManager: INFO: Finished task 17.0 in stage 7.0 (TID 143) in 838 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (17/120) 2022-03-09 16:31:13 TaskSetManager: INFO: Starting task 25.0 in stage 7.0 (TID 151) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 25, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:13 TaskSetManager: INFO: Finished task 18.0 in stage 7.0 (TID 144) in 821 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (18/120) 2022-03-09 16:31:13 TaskSetManager: INFO: Starting task 26.0 in stage 7.0 (TID 152) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 26, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:13 TaskSetManager: INFO: Finished task 16.0 in stage 7.0 (TID 142) in 1005 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (19/120) 2022-03-09 16:31:13 TaskSetManager: INFO: Starting task 27.0 in stage 7.0 (TID 153) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 27, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:13 TaskSetManager: INFO: Finished task 23.0 in stage 7.0 (TID 149) in 904 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (20/120) 2022-03-09 16:31:13 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:13 YarnAllocator: INFO: Driver requested a total number of 18 executor(s) for resource profile id: 0. 2022-03-09 16:31:13 ExecutorAllocationManager: INFO: Requesting 8 new executors because tasks are backlogged (new desired total will be 18 for resource profile id: 0) 2022-03-09 16:31:13 YarnAllocator: INFO: Will request 8 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.78:37188) with ID 3, ResourceProfileId 0 2022-03-09 16:31:13 ExecutorMonitor: INFO: New executor 3 has registered (new total is 4) 2022-03-09 16:31:14 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-0.c.broad-mpg-gnomad.internal:33925 with 2.9 GiB RAM, BlockManagerId(3, kc-w-0.c.broad-mpg-gnomad.internal, 33925, None) 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 20.0 in stage 7.0 (TID 146) in 1429 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (21/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 22.0 in stage 7.0 (TID 148) in 1441 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (22/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 21.0 in stage 7.0 (TID 147) in 1444 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (23/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 19.0 in stage 7.0 (TID 145) in 1493 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (24/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Starting task 28.0 in stage 7.0 (TID 154) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 28, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 24.0 in stage 7.0 (TID 150) in 833 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (25/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Starting task 29.0 in stage 7.0 (TID 155) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 29, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 25.0 in stage 7.0 (TID 151) in 833 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (26/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Starting task 30.0 in stage 7.0 (TID 156) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 30, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 26.0 in stage 7.0 (TID 152) in 804 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (27/120) 2022-03-09 16:31:14 TaskSetManager: INFO: Starting task 31.0 in stage 7.0 (TID 157) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 31, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:14 TaskSetManager: INFO: Finished task 27.0 in stage 7.0 (TID 153) in 837 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (28/120) 2022-03-09 16:31:14 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:14 YarnAllocator: INFO: Driver requested a total number of 23 executor(s) for resource profile id: 0. 2022-03-09 16:31:14 ExecutorAllocationManager: INFO: Requesting 5 new executors because tasks are backlogged (new desired total will be 23 for resource profile id: 0) 2022-03-09 16:31:14 YarnAllocator: INFO: Will request 5 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:14 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:14 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:14 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:14 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:14 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 32.0 in stage 7.0 (TID 158) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 32, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 28.0 in stage 7.0 (TID 154) in 644 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (29/120) 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 33.0 in stage 7.0 (TID 159) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 33, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 30.0 in stage 7.0 (TID 156) in 621 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (30/120) 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 34.0 in stage 7.0 (TID 160) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 34, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 31.0 in stage 7.0 (TID 157) in 707 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (31/120) 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 35.0 in stage 7.0 (TID 161) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 35, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 33.0 in stage 7.0 (TID 159) in 561 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (32/120) 2022-03-09 16:31:15 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:15 YarnAllocator: INFO: Driver requested a total number of 22 executor(s) for resource profile id: 0. 2022-03-09 16:31:15 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 22 executors. 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 36.0 in stage 7.0 (TID 162) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 36, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 34.0 in stage 7.0 (TID 160) in 515 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (33/120) 2022-03-09 16:31:15 TaskSetManager: INFO: Starting task 37.0 in stage 7.0 (TID 163) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 37, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:15 TaskSetManager: INFO: Finished task 32.0 in stage 7.0 (TID 158) in 881 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (34/120) 2022-03-09 16:31:16 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.79:49336) with ID 4, ResourceProfileId 0 2022-03-09 16:31:16 ExecutorMonitor: INFO: New executor 4 has registered (new total is 5) 2022-03-09 16:31:16 BlockManagerMasterEndpoint: INFO: Registering block manager kc-w-1.c.broad-mpg-gnomad.internal:45587 with 2.9 GiB RAM, BlockManagerId(4, kc-w-1.c.broad-mpg-gnomad.internal, 45587, None) 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 38.0 in stage 7.0 (TID 164) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 38, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Finished task 35.0 in stage 7.0 (TID 161) in 534 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (35/120) 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 39.0 in stage 7.0 (TID 165) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 39, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 40.0 in stage 7.0 (TID 166) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 40, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 41.0 in stage 7.0 (TID 167) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 41, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 42.0 in stage 7.0 (TID 168) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 42, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 43.0 in stage 7.0 (TID 169) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 43, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Finished task 36.0 in stage 7.0 (TID 162) in 544 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (36/120) 2022-03-09 16:31:16 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:16 YarnAllocator: INFO: Driver requested a total number of 21 executor(s) for resource profile id: 0. 2022-03-09 16:31:16 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 21 executors. 2022-03-09 16:31:16 BlockManagerInfo: INFO: Added broadcast_25_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:45587 (size: 58.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 44.0 in stage 7.0 (TID 170) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 44, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Finished task 37.0 in stage 7.0 (TID 163) in 815 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (37/120) 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 45.0 in stage 7.0 (TID 171) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 45, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Finished task 38.0 in stage 7.0 (TID 164) in 576 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (38/120) 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 46.0 in stage 7.0 (TID 172) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 46, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 47.0 in stage 7.0 (TID 173) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 47, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 48.0 in stage 7.0 (TID 174) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 48, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 49.0 in stage 7.0 (TID 175) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 49, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 50.0 in stage 7.0 (TID 176) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 50, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 51.0 in stage 7.0 (TID 177) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 51, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 52.0 in stage 7.0 (TID 178) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 52, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:16 TaskSetManager: INFO: Starting task 53.0 in stage 7.0 (TID 179) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 53, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Starting task 54.0 in stage 7.0 (TID 180) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 54, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 43.0 in stage 7.0 (TID 169) in 644 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (39/120) 2022-03-09 16:31:17 TaskSetManager: INFO: Starting task 55.0 in stage 7.0 (TID 181) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 55, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 44.0 in stage 7.0 (TID 170) in 554 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (40/120) 2022-03-09 16:31:17 BlockManagerInfo: INFO: Added broadcast_25_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:33925 (size: 58.7 KiB, free: 2.9 GiB) 2022-03-09 16:31:17 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:17 YarnAllocator: INFO: Driver requested a total number of 20 executor(s) for resource profile id: 0. 2022-03-09 16:31:17 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 20 executors. 2022-03-09 16:31:17 TaskSetManager: INFO: Starting task 56.0 in stage 7.0 (TID 182) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 56, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 45.0 in stage 7.0 (TID 171) in 662 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (41/120) 2022-03-09 16:31:17 TaskSetManager: INFO: Starting task 57.0 in stage 7.0 (TID 183) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 57, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 54.0 in stage 7.0 (TID 180) in 655 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (42/120) 2022-03-09 16:31:17 TaskSetManager: INFO: Starting task 58.0 in stage 7.0 (TID 184) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 58, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 55.0 in stage 7.0 (TID 181) in 562 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (43/120) 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 49.0 in stage 7.0 (TID 175) in 1075 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (44/120) 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 51.0 in stage 7.0 (TID 177) in 1089 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (45/120) 2022-03-09 16:31:17 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:17 YarnAllocator: INFO: Driver requested a total number of 19 executor(s) for resource profile id: 0. 2022-03-09 16:31:17 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 19 executors. 2022-03-09 16:31:17 TaskSetManager: INFO: Finished task 53.0 in stage 7.0 (TID 179) in 1107 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (46/120) 2022-03-09 16:31:18 TaskSetManager: INFO: Starting task 59.0 in stage 7.0 (TID 185) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 59, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:18 TaskSetManager: INFO: Finished task 56.0 in stage 7.0 (TID 182) in 555 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (47/120) 2022-03-09 16:31:18 TaskSetManager: INFO: Finished task 47.0 in stage 7.0 (TID 173) in 1426 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (48/120) 2022-03-09 16:31:18 TaskSetManager: INFO: Starting task 60.0 in stage 7.0 (TID 186) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 60, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:18 TaskSetManager: INFO: Finished task 57.0 in stage 7.0 (TID 183) in 618 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (49/120) 2022-03-09 16:31:18 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:18 YarnAllocator: INFO: Driver requested a total number of 18 executor(s) for resource profile id: 0. 2022-03-09 16:31:18 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 18 executors. 2022-03-09 16:31:18 TaskSetManager: INFO: Starting task 61.0 in stage 7.0 (TID 187) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 61, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:18 TaskSetManager: INFO: Finished task 58.0 in stage 7.0 (TID 184) in 662 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (50/120) 2022-03-09 16:31:18 MapOutputTrackerMasterEndpoint: INFO: Asked to send map output locations for shuffle 1 to 10.128.0.79:49336 2022-03-09 16:31:19 TaskSetManager: INFO: Starting task 62.0 in stage 7.0 (TID 188) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 62, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:19 TaskSetManager: INFO: Finished task 60.0 in stage 7.0 (TID 186) in 779 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (51/120) 2022-03-09 16:31:19 MapOutputTrackerMasterEndpoint: INFO: Asked to send map output locations for shuffle 1 to 10.128.0.78:37188 2022-03-09 16:31:19 TaskSetManager: INFO: Starting task 63.0 in stage 7.0 (TID 189) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 63, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:19 TaskSetManager: INFO: Finished task 62.0 in stage 7.0 (TID 188) in 551 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (52/120) 2022-03-09 16:31:19 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:19 YarnAllocator: INFO: Driver requested a total number of 17 executor(s) for resource profile id: 0. 2022-03-09 16:31:19 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 17 executors. 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 64.0 in stage 7.0 (TID 190) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 64, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Finished task 63.0 in stage 7.0 (TID 189) in 446 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (53/120) 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 65.0 in stage 7.0 (TID 191) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 65, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Finished task 59.0 in stage 7.0 (TID 185) in 2270 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (54/120) 2022-03-09 16:31:20 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-w-1.c.broad-mpg-gnomad.internal:45587 (size: 33.4 KiB, free: 2.9 GiB) 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 66.0 in stage 7.0 (TID 192) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 66, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Finished task 61.0 in stage 7.0 (TID 187) in 2118 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (55/120) 2022-03-09 16:31:20 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc-w-0.c.broad-mpg-gnomad.internal:33925 (size: 33.4 KiB, free: 2.9 GiB) 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 67.0 in stage 7.0 (TID 193) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 67, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Finished task 64.0 in stage 7.0 (TID 190) in 685 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (56/120) 2022-03-09 16:31:20 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:20 YarnAllocator: INFO: Driver requested a total number of 16 executor(s) for resource profile id: 0. 2022-03-09 16:31:20 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 16 executors. 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 68.0 in stage 7.0 (TID 194) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 68, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 69.0 in stage 7.0 (TID 195) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 69, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 70.0 in stage 7.0 (TID 196) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 70, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 71.0 in stage 7.0 (TID 197) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 71, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Starting task 72.0 in stage 7.0 (TID 198) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 72, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:20 TaskSetManager: INFO: Finished task 65.0 in stage 7.0 (TID 191) in 618 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (57/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Starting task 73.0 in stage 7.0 (TID 199) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 73, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 67.0 in stage 7.0 (TID 193) in 778 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (58/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Starting task 74.0 in stage 7.0 (TID 200) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 74, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 66.0 in stage 7.0 (TID 192) in 915 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (59/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Starting task 75.0 in stage 7.0 (TID 201) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 75, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 72.0 in stage 7.0 (TID 198) in 834 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (60/120) 2022-03-09 16:31:21 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:21 YarnAllocator: INFO: Driver requested a total number of 15 executor(s) for resource profile id: 0. 2022-03-09 16:31:21 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 15 executors. 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 68.0 in stage 7.0 (TID 194) in 951 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (61/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 70.0 in stage 7.0 (TID 196) in 959 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (62/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 69.0 in stage 7.0 (TID 195) in 966 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (63/120) 2022-03-09 16:31:21 TaskSetManager: INFO: Finished task 71.0 in stage 7.0 (TID 197) in 994 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (64/120) 2022-03-09 16:31:21 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:21 YarnAllocator: INFO: Driver requested a total number of 14 executor(s) for resource profile id: 0. 2022-03-09 16:31:21 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 14 executors. 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 76.0 in stage 7.0 (TID 202) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 76, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 74.0 in stage 7.0 (TID 200) in 649 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (65/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 77.0 in stage 7.0 (TID 203) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 77, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 73.0 in stage 7.0 (TID 199) in 690 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (66/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 78.0 in stage 7.0 (TID 204) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 78, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 75.0 in stage 7.0 (TID 201) in 609 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (67/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 50.0 in stage 7.0 (TID 176) in 5514 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (68/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 48.0 in stage 7.0 (TID 174) in 5516 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (69/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 52.0 in stage 7.0 (TID 178) in 5515 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (70/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 46.0 in stage 7.0 (TID 172) in 5542 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (71/120) 2022-03-09 16:31:22 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:22 YarnAllocator: INFO: Driver requested a total number of 13 executor(s) for resource profile id: 0. 2022-03-09 16:31:22 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 13 executors. 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 79.0 in stage 7.0 (TID 205) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 79, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 40.0 in stage 7.0 (TID 166) in 6171 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (72/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 80.0 in stage 7.0 (TID 206) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 80, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 39.0 in stage 7.0 (TID 165) in 6173 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (73/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 81.0 in stage 7.0 (TID 207) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 81, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 42.0 in stage 7.0 (TID 168) in 6173 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (74/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 82.0 in stage 7.0 (TID 208) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 82, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 41.0 in stage 7.0 (TID 167) in 6174 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (75/120) 2022-03-09 16:31:22 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:22 YarnAllocator: INFO: Driver requested a total number of 12 executor(s) for resource profile id: 0. 2022-03-09 16:31:22 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 12 executors. 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 83.0 in stage 7.0 (TID 209) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 83, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 76.0 in stage 7.0 (TID 202) in 627 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (76/120) 2022-03-09 16:31:22 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:22 YarnAllocator: INFO: Driver requested a total number of 11 executor(s) for resource profile id: 0. 2022-03-09 16:31:22 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 11 executors. 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 84.0 in stage 7.0 (TID 210) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 84, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 77.0 in stage 7.0 (TID 203) in 663 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (77/120) 2022-03-09 16:31:22 TaskSetManager: INFO: Starting task 85.0 in stage 7.0 (TID 211) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 85, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:22 TaskSetManager: INFO: Finished task 78.0 in stage 7.0 (TID 204) in 621 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (78/120) 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 86.0 in stage 7.0 (TID 212) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 86, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 80.0 in stage 7.0 (TID 206) in 863 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (79/120) 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 87.0 in stage 7.0 (TID 213) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 87, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 82.0 in stage 7.0 (TID 208) in 915 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (80/120) 2022-03-09 16:31:23 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:23 YarnAllocator: INFO: Driver requested a total number of 10 executor(s) for resource profile id: 0. 2022-03-09 16:31:23 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 10 executors. 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 88.0 in stage 7.0 (TID 214) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 88, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 79.0 in stage 7.0 (TID 205) in 1013 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (81/120) 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 89.0 in stage 7.0 (TID 215) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 89, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 83.0 in stage 7.0 (TID 209) in 782 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (82/120) 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 90.0 in stage 7.0 (TID 216) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 90, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 84.0 in stage 7.0 (TID 210) in 929 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (83/120) 2022-03-09 16:31:23 TaskSetManager: INFO: Starting task 91.0 in stage 7.0 (TID 217) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 91, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:23 TaskSetManager: INFO: Finished task 29.0 in stage 7.0 (TID 155) in 9366 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (84/120) 2022-03-09 16:31:23 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:23 YarnAllocator: INFO: Driver requested a total number of 9 executor(s) for resource profile id: 0. 2022-03-09 16:31:23 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 9 executors. 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 92.0 in stage 7.0 (TID 218) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 92, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 81.0 in stage 7.0 (TID 207) in 1545 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (85/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 93.0 in stage 7.0 (TID 219) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 93, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 86.0 in stage 7.0 (TID 212) in 684 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (86/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 94.0 in stage 7.0 (TID 220) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 94, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 89.0 in stage 7.0 (TID 215) in 600 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (87/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 95.0 in stage 7.0 (TID 221) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 95, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 88.0 in stage 7.0 (TID 214) in 704 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (88/120) 2022-03-09 16:31:24 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:24 YarnAllocator: INFO: Driver requested a total number of 8 executor(s) for resource profile id: 0. 2022-03-09 16:31:24 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 8 executors. 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 96.0 in stage 7.0 (TID 222) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 96, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 90.0 in stage 7.0 (TID 216) in 557 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (89/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 97.0 in stage 7.0 (TID 223) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 97, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 91.0 in stage 7.0 (TID 217) in 738 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (90/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 98.0 in stage 7.0 (TID 224) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 98, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 92.0 in stage 7.0 (TID 218) in 639 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (91/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Marking task 87 in stage 7.0 (on kc-w-1.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(1 speculatable tasks in this taskset now) 2022-03-09 16:31:24 TaskSetManager: INFO: Marking task 85 in stage 7.0 (on kc-w-1.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(2 speculatable tasks in this taskset now) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 99.0 in stage 7.0 (TID 225) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 99, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 95.0 in stage 7.0 (TID 221) in 588 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (92/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 100.0 in stage 7.0 (TID 226) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 100, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 94.0 in stage 7.0 (TID 220) in 648 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (93/120) 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 101.0 in stage 7.0 (TID 227) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 101, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 102.0 in stage 7.0 (TID 228) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 102, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 103.0 in stage 7.0 (TID 229) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 103, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 104.0 in stage 7.0 (TID 230) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 104, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 105.0 in stage 7.0 (TID 231) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 105, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 106.0 in stage 7.0 (TID 232) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 106, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 107.0 in stage 7.0 (TID 233) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 107, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 108.0 in stage 7.0 (TID 234) (kc-w-0.c.broad-mpg-gnomad.internal, executor 3, partition 108, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Starting task 109.0 in stage 7.0 (TID 235) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 109, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:24 TaskSetManager: INFO: Finished task 96.0 in stage 7.0 (TID 222) in 515 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (94/120) 2022-03-09 16:31:24 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:24 YarnAllocator: INFO: Driver requested a total number of 7 executor(s) for resource profile id: 0. 2022-03-09 16:31:24 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 7 executors. 2022-03-09 16:31:25 TaskSetManager: INFO: Starting task 110.0 in stage 7.0 (TID 236) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 110, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 97.0 in stage 7.0 (TID 223) in 573 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (95/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Marking task 93 in stage 7.0 (on kc-w-1.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1223 ms(3 speculatable tasks in this taskset now) 2022-03-09 16:31:25 TaskSetManager: INFO: Starting task 111.0 in stage 7.0 (TID 237) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 111, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 109.0 in stage 7.0 (TID 235) in 595 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (96/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Starting task 112.0 in stage 7.0 (TID 238) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 112, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 98.0 in stage 7.0 (TID 224) in 951 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (97/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 101.0 in stage 7.0 (TID 227) in 760 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (98/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Starting task 113.0 in stage 7.0 (TID 239) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 113, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 85.0 in stage 7.0 (TID 211) in 2743 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (99/120) 2022-03-09 16:31:25 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:25 YarnAllocator: INFO: Driver requested a total number of 6 executor(s) for resource profile id: 0. 2022-03-09 16:31:25 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 6 executors. 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 106.0 in stage 7.0 (TID 232) in 990 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (100/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 102.0 in stage 7.0 (TID 228) in 994 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (101/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 103.0 in stage 7.0 (TID 229) in 996 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (102/120) 2022-03-09 16:31:25 TaskSetManager: INFO: Finished task 108.0 in stage 7.0 (TID 234) in 1027 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 3) (103/120) 2022-03-09 16:31:25 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:25 YarnAllocator: INFO: Driver requested a total number of 5 executor(s) for resource profile id: 0. 2022-03-09 16:31:25 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 5 executors. 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 114.0 in stage 7.0 (TID 240) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 114, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 111.0 in stage 7.0 (TID 237) in 558 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (104/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 115.0 in stage 7.0 (TID 241) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 115, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 100.0 in stage 7.0 (TID 226) in 1259 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (105/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Marking task 107 in stage 7.0 (on kc-w-0.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(4 speculatable tasks in this taskset now) 2022-03-09 16:31:26 TaskSetManager: INFO: Marking task 104 in stage 7.0 (on kc-w-0.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(5 speculatable tasks in this taskset now) 2022-03-09 16:31:26 TaskSetManager: INFO: Marking task 105 in stage 7.0 (on kc-w-0.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(6 speculatable tasks in this taskset now) 2022-03-09 16:31:26 TaskSetManager: INFO: Marking task 99 in stage 7.0 (on kc-w-1.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1232 ms(7 speculatable tasks in this taskset now) 2022-03-09 16:31:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:26 YarnAllocator: INFO: Driver requested a total number of 6 executor(s) for resource profile id: 0. 2022-03-09 16:31:26 ExecutorAllocationManager: INFO: Requesting 1 new executor because tasks are backlogged (new desired total will be 6 for resource profile id: 0) 2022-03-09 16:31:26 YarnAllocator: INFO: Will request 1 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 14592 MB memory. with custom resources: 2022-03-09 16:31:26 YarnAllocator: INFO: Submitted container request for host kc-w-1.c.broad-mpg-gnomad.internal. 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 107.0 in stage 7.0 (TID 233) in 1362 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (106/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 116.0 in stage 7.0 (TID 242) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 116, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 87.0 in stage 7.0 (TID 213) in 2834 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (107/120) 2022-03-09 16:31:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:26 YarnAllocator: INFO: Driver requested a total number of 5 executor(s) for resource profile id: 0. 2022-03-09 16:31:26 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 5 executors. 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 117.0 in stage 7.0 (TID 243) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 117, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 99.0 in stage 7.0 (TID 225) in 1562 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (108/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 118.0 in stage 7.0 (TID 244) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 118, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 93.0 in stage 7.0 (TID 219) in 2347 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (109/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 119.0 in stage 7.0 (TID 245) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 119, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 113.0 in stage 7.0 (TID 239) in 671 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (110/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 105.1 in stage 7.0 (TID 246) (kc-w-1.c.broad-mpg-gnomad.internal, executor 2, partition 105, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 110.0 in stage 7.0 (TID 236) in 1215 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (111/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Starting task 104.1 in stage 7.0 (TID 247) (kc-w-1.c.broad-mpg-gnomad.internal, executor 4, partition 104, NODE_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 112.0 in stage 7.0 (TID 238) in 763 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (112/120) 2022-03-09 16:31:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:26 YarnAllocator: INFO: Driver requested a total number of 4 executor(s) for resource profile id: 0. 2022-03-09 16:31:26 YarnAllocator: INFO: Canceling requests for 1 executor container(s) to have a new desired total 4 executors. 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 114.0 in stage 7.0 (TID 240) in 667 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (113/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 115.0 in stage 7.0 (TID 241) in 623 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (114/120) 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 116.0 in stage 7.0 (TID 242) in 646 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (115/120) 2022-03-09 16:31:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:26 YarnAllocator: INFO: Driver requested a total number of 3 executor(s) for resource profile id: 0. 2022-03-09 16:31:26 TaskSetManager: INFO: Killing attempt 0 for task 105.0 in stage 7.0 (TID 231) on kc-w-0.c.broad-mpg-gnomad.internal as the attempt 1 succeeded on kc-w-1.c.broad-mpg-gnomad.internal 2022-03-09 16:31:26 TaskSetManager: INFO: Finished task 105.1 in stage 7.0 (TID 246) in 611 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 2) (116/120) 2022-03-09 16:31:27 TaskSetManager: WARN: Lost task 105.0 in stage 7.0 (TID 231) (kc-w-0.c.broad-mpg-gnomad.internal executor 1): TaskKilled (another attempt succeeded) 2022-03-09 16:31:27 TaskSetManager: INFO: task 105.0 in stage 7.0 (TID 231) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-03-09 16:31:27 TaskSetManager: INFO: Finished task 117.0 in stage 7.0 (TID 243) in 700 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (117/120) 2022-03-09 16:31:27 TaskSetManager: INFO: Finished task 118.0 in stage 7.0 (TID 244) in 675 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (118/120) 2022-03-09 16:31:27 TaskSetManager: INFO: Killing attempt 0 for task 104.0 in stage 7.0 (TID 230) on kc-w-0.c.broad-mpg-gnomad.internal as the attempt 1 succeeded on kc-w-1.c.broad-mpg-gnomad.internal 2022-03-09 16:31:27 TaskSetManager: INFO: Finished task 104.1 in stage 7.0 (TID 247) in 693 ms on kc-w-1.c.broad-mpg-gnomad.internal (executor 4) (119/120) 2022-03-09 16:31:27 TaskSetManager: WARN: Lost task 104.0 in stage 7.0 (TID 230) (kc-w-0.c.broad-mpg-gnomad.internal executor 3): TaskKilled (another attempt succeeded) 2022-03-09 16:31:27 TaskSetManager: INFO: task 104.0 in stage 7.0 (TID 230) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-03-09 16:31:27 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:27 YarnAllocator: INFO: Driver requested a total number of 2 executor(s) for resource profile id: 0. 2022-03-09 16:31:27 TaskSetManager: INFO: Marking task 119 in stage 7.0 (on kc-w-1.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 1223 ms(6 speculatable tasks in this taskset now) 2022-03-09 16:31:27 TaskSetManager: INFO: Starting task 119.1 in stage 7.0 (TID 248) (kc-w-0.c.broad-mpg-gnomad.internal, executor 1, partition 119, RACK_LOCAL, 4282 bytes) taskResourceAssignments Map() 2022-03-09 16:31:32 TaskSetManager: INFO: Killing attempt 0 for task 119.0 in stage 7.0 (TID 245) on kc-w-1.c.broad-mpg-gnomad.internal as the attempt 1 succeeded on kc-w-0.c.broad-mpg-gnomad.internal 2022-03-09 16:31:32 TaskSetManager: INFO: Finished task 119.1 in stage 7.0 (TID 248) in 4599 ms on kc-w-0.c.broad-mpg-gnomad.internal (executor 1) (120/120) 2022-03-09 16:31:32 DAGScheduler: INFO: ResultStage 7 (collect at ContextRDD.scala:176) finished in 23.307 s 2022-03-09 16:31:32 DAGScheduler: INFO: Job 5 is finished. Cancelling potential speculative or zombie tasks for this job 2022-03-09 16:31:32 YarnScheduler: INFO: Killing all running tasks in stage 7: Stage finished 2022-03-09 16:31:32 DAGScheduler: INFO: Job 5 finished: collect at ContextRDD.scala:176, took 23.946587 s 2022-03-09 16:31:32 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:31:32 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0. 2022-03-09 16:31:32 BlockManagerInfo: INFO: Removed broadcast_24_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 40.9 KiB, free: 12.6 GiB) 2022-03-09 16:31:32 BlockManagerInfo: INFO: Removed broadcast_24_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 40.9 KiB, free: 2.9 GiB) 2022-03-09 16:31:32 TaskSetManager: WARN: Lost task 119.0 in stage 7.0 (TID 245) (kc-w-1.c.broad-mpg-gnomad.internal executor 2): TaskKilled (Stage finished) 2022-03-09 16:31:32 TaskSetManager: INFO: task 119.0 in stage 7.0 (TID 245) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-03-09 16:31:32 YarnScheduler: INFO: Removed TaskSet 7.0, whose tasks have all completed, from pool default 2022-03-09 16:31:33 Hail: INFO: wrote table with 120 rows in 120 partitions to gs://gnomad-tmp/kc/mxra5_repart_test.ht Total size: 10.16 MiB * Rows: 10.16 MiB * Globals: 12.00 B * Smallest partition: 1 rows (86.69 KiB) * Largest partition: 1 rows (86.70 KiB) 2022-03-09 16:31:33 root: INFO: took 32.188s 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (Begin) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (Begin) 2022-03-09 16:31:33 root: INFO: finished execution of query hail_query_6 2022-03-09 16:31:33 root: INFO: RegionPool: FREE: 192.0K allocated (192.0K blocks / 0 chunks), regions.size = 3, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode total 32.282s self 6.166ms children 32.276s %children 99.98% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 42.371ms self 0.020ms children 42.351ms %children 99.95% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 42.242ms self 5.028ms children 37.214ms %children 88.10% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 37.214ms self 0.148ms children 37.067ms %children 99.60% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.263ms self 0.263ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.606ms self 1.606ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 7.370ms self 7.370ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 4.101ms self 4.101ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.190ms self 0.190ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.271ms self 4.271ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.527ms self 0.527ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.850ms self 0.850ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.806ms self 0.806ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.222ms self 3.222ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.287ms self 0.287ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 5.006ms self 5.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.318ms self 0.318ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.768ms self 0.768ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.895ms self 0.895ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.096ms self 3.096ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.151ms self 0.151ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.339ms self 3.339ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.554ms self 0.033ms children 0.520ms %children 93.98% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.431ms self 0.431ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 43.214ms self 0.029ms children 43.185ms %children 99.93% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 43.094ms self 4.187ms children 38.907ms %children 90.28% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 38.907ms self 0.180ms children 38.727ms %children 99.54% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.350ms self 0.350ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.277ms self 1.277ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.056ms self 1.056ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 4.012ms self 4.012ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.276ms self 0.276ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.823ms self 4.823ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.292ms self 0.292ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.803ms self 0.803ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.842ms self 0.842ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 3.521ms self 3.521ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.277ms self 0.277ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 7.181ms self 7.181ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.890ms self 0.890ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.719ms self 1.719ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.904ms self 0.904ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 5.945ms self 5.945ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.382ms self 4.382ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.439ms self 0.010ms children 0.428ms %children 97.71% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.307ms self 0.307ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 32.188s self 0.032ms children 32.188s %children 100.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 32.188s self 32.052s children 136.680ms %children 0.42% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 12.848ms self 0.015ms children 12.833ms %children 99.88% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.126ms self 0.126ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 12.688ms self 1.720ms children 10.967ms %children 86.44% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 10.967ms self 0.109ms children 10.858ms %children 99.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.507ms self 0.507ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.111ms self 0.111ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.321ms self 0.321ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.040ms self 1.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.193ms self 0.193ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.247ms self 1.247ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.406ms self 0.406ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.109ms self 0.109ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.472ms self 0.472ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.333ms self 1.333ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.133ms self 0.133ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.572ms self 1.572ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.352ms self 0.352ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.370ms self 0.370ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.554ms self 1.554ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.141ms self 0.141ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.854ms self 0.854ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.242ms self 0.010ms children 0.232ms %children 96.02% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.177ms self 0.177ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 11.501ms self 0.014ms children 11.487ms %children 99.87% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 11.429ms self 2.105ms children 9.325ms %children 81.59% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 9.325ms self 0.091ms children 9.233ms %children 99.02% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.295ms self 0.295ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.173ms self 0.173ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.321ms self 0.321ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.128ms self 1.128ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.132ms self 0.132ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.171ms self 1.171ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.290ms self 0.290ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.123ms self 0.123ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.262ms self 0.262ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.977ms self 0.977ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.133ms self 0.133ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.160ms self 1.160ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.232ms self 0.232ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.118ms self 0.118ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.255ms self 0.255ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.043ms self 1.043ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.294ms self 1.294ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 5.681ms self 0.017ms children 5.664ms %children 99.71% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 5.515ms self 5.515ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.108ms self 0.108ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 13.385ms self 0.014ms children 13.372ms %children 99.90% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 13.327ms self 2.174ms children 11.153ms %children 83.69% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 11.153ms self 0.078ms children 11.075ms %children 99.30% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.219ms self 0.219ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.185ms self 0.185ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.305ms self 0.305ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.232ms self 1.232ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.123ms self 0.123ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.150ms self 1.150ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.279ms self 0.279ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.162ms self 0.162ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.255ms self 0.255ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.975ms self 0.975ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.133ms self 0.133ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.143ms self 1.143ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.220ms self 0.220ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.124ms self 0.124ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.274ms self 0.274ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 2.695ms self 2.695ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.189ms self 0.189ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.412ms self 1.412ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 1.780ms self 1.780ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 3.130ms self 0.012ms children 3.118ms %children 99.62% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.132ms self 0.132ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 2.980ms self 0.773ms children 2.207ms %children 74.05% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 2.207ms self 0.086ms children 2.120ms %children 96.08% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.176ms self 0.176ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.233ms self 0.233ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.421ms self 0.421ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.202ms self 0.202ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.494ms self 0.494ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.081ms self 0.081ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.299ms self 0.299ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.133ms self 0.006ms children 0.127ms %children 95.34% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 1.300ms self 0.007ms children 1.294ms %children 99.48% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.282ms self 0.581ms children 0.701ms %children 54.68% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.701ms self 0.023ms children 0.678ms %children 96.69% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.133ms self 0.133ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.221ms self 0.221ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.151ms self 0.151ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 0.438ms self 0.006ms children 0.432ms %children 98.55% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 0.387ms self 0.387ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 1.278ms self 0.007ms children 1.271ms %children 99.46% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 1.264ms self 0.628ms children 0.636ms %children 50.34% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.636ms self 0.020ms children 0.616ms %children 96.85% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.204ms self 0.204ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.232ms self 0.232ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 0.295ms self 0.295ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 3.053ms self 0.010ms children 3.043ms %children 99.68% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.148ms self 0.148ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 2.890ms self 0.779ms children 2.111ms %children 73.05% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 2.111ms self 0.061ms children 2.050ms %children 97.11% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.237ms self 0.237ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.147ms self 0.147ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.347ms self 0.347ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.418ms self 0.418ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.105ms self 0.105ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.287ms self 0.287ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.265ms self 0.265ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.039ms self 0.004ms children 0.034ms %children 89.04% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 1.417ms self 0.006ms children 1.411ms %children 99.58% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.406ms self 0.586ms children 0.819ms %children 58.28% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.819ms self 0.019ms children 0.801ms %children 97.74% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.332ms self 0.332ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.279ms self 0.279ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 2.325ms self 0.008ms children 2.316ms %children 99.64% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 2.255ms self 2.255ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 2.165ms self 0.008ms children 2.156ms %children 99.61% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 2.147ms self 0.998ms children 1.148ms %children 53.50% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 1.148ms self 0.029ms children 1.119ms %children 97.43% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.097ms self 0.097ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.127ms self 0.127ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.362ms self 0.362ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.390ms self 0.390ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 0.493ms self 0.493ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 7.808ms self 0.014ms children 7.794ms %children 99.82% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 7.738ms self 1.743ms children 5.995ms %children 77.48% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 5.995ms self 0.067ms children 5.928ms %children 98.89% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.286ms self 0.286ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.128ms self 0.128ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.199ms self 0.199ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.617ms self 0.617ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.137ms self 0.137ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.093ms self 1.093ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.087ms self 0.087ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.183ms self 0.183ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.587ms self 0.587ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.662ms self 0.662ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.131ms self 0.131ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.154ms self 0.154ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.573ms self 0.573ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.717ms self 0.717ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.070ms self 0.007ms children 0.064ms %children 90.43% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 6.589ms self 0.011ms children 6.578ms %children 99.83% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 6.564ms self 1.487ms children 5.077ms %children 77.35% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 5.077ms self 0.058ms children 5.019ms %children 98.85% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.128ms self 0.128ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.253ms self 0.253ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.608ms self 0.608ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.764ms self 0.764ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.152ms self 0.152ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.078ms self 0.078ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.156ms self 0.156ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.454ms self 0.454ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.629ms self 0.629ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.108ms self 0.108ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.165ms self 0.165ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.450ms self 0.450ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.656ms self 0.656ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 1.022ms self 0.008ms children 1.014ms %children 99.20% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 0.984ms self 0.984ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 5.890ms self 0.008ms children 5.882ms %children 99.86% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 5.872ms self 0.980ms children 4.892ms %children 83.32% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 4.892ms self 0.042ms children 4.850ms %children 99.14% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.079ms self 0.079ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.133ms self 0.133ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.351ms self 0.351ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.634ms self 0.634ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.514ms self 1.514ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.633ms self 0.633ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.100ms self 0.100ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.400ms self 0.400ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.424ms self 0.424ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 0.754ms self 0.754ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 2.599ms self 0.013ms children 2.586ms %children 99.51% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 2.488ms self 1.049ms children 1.439ms %children 57.85% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 1.439ms self 0.051ms children 1.388ms %children 96.45% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.283ms self 0.283ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.176ms self 0.176ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.215ms self 0.215ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.120ms self 0.120ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.540ms self 0.540ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.045ms self 0.005ms children 0.040ms %children 89.13% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 1.431ms self 0.006ms children 1.425ms %children 99.56% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.419ms self 0.700ms children 0.719ms %children 50.69% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.719ms self 0.022ms children 0.697ms %children 96.89% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.074ms self 0.074ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.098ms self 0.098ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.214ms self 0.214ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.231ms self 0.231ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 0.392ms self 0.005ms children 0.387ms %children 98.67% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 0.363ms self 0.363ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 1.367ms self 0.008ms children 1.359ms %children 99.43% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 1.353ms self 0.707ms children 0.646ms %children 47.76% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.646ms self 0.021ms children 0.625ms %children 96.76% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.166ms self 0.166ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.220ms self 0.220ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 0.521ms self 0.521ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR total 16.180ms self 0.018ms children 16.163ms %children 99.89% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation total 16.086ms self 3.921ms children 12.165ms %children 75.63% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 12.165ms self 0.058ms children 12.107ms %children 99.52% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.612ms self 0.612ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.216ms self 0.216ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.437ms self 0.437ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.035ms self 1.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.934ms self 2.934ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.220ms self 0.220ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.295ms self 0.295ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.969ms self 0.969ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.620ms self 1.620ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.173ms self 0.173ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.294ms self 0.294ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.087ms self 1.087ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.092ms self 0.092ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.604ms self 1.604ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, initial IR/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR total 0.186ms self 0.009ms children 0.176ms %children 95.09% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/LoweringTransformation total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InlineApplyIR/Verify total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR total 13.801ms self 0.010ms children 13.791ms %children 99.93% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 13.775ms self 2.773ms children 11.002ms %children 79.87% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 11.002ms self 0.065ms children 10.937ms %children 99.41% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.225ms self 0.225ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.423ms self 0.423ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.450ms self 0.450ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.215ms self 1.215ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.127ms self 0.127ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.157ms self 2.157ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.209ms self 0.209ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.166ms self 0.166ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.336ms self 0.336ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.937ms self 0.937ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.086ms self 0.086ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.533ms self 1.533ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.182ms self 0.182ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.141ms self 0.141ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.307ms self 0.307ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.774ms self 0.774ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.082ms self 0.082ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 1.585ms self 1.585ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after InlineApplyIR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs total 1.812ms self 0.014ms children 1.798ms %children 99.24% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/LoweringTransformation total 1.729ms self 1.729ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerArrayAggsToRunAggs/Verify total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 13.020ms self 0.014ms children 13.006ms %children 99.90% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 12.990ms self 2.408ms children 10.582ms %children 81.47% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 10.582ms self 0.055ms children 10.527ms %children 99.48% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.168ms self 0.168ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.319ms self 0.319ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.889ms self 0.889ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.686ms self 1.686ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.215ms self 0.215ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.151ms self 0.151ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.282ms self 0.282ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 1.435ms self 1.435ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.749ms self 1.749ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.225ms self 0.225ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.276ms self 0.276ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.309ms self 0.309ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.842ms self 0.842ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.085ms self 0.085ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 1.510ms self 1.510ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/EmitContext.analyze total 1.691ms self 1.691ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.775ms self 0.007ms children 0.768ms %children 99.06% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.763ms self 0.216ms children 0.547ms %children 71.69% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.547ms self 0.018ms children 0.529ms %children 96.69% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.265ms self 0.265ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/Compile total 0.091ms self 0.091ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/RunCompiledVoidFunction total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: timing SparkBackend.parse_table_ir total 260.018ms self 260.018ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: JSON: JObject(List((name,JString(NPartitionsTable)))) 2022-03-09 16:31:33 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: timing SparkBackend.parse_value_ir total 1.496ms self 1.496ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: starting execution of query hail_query_7 of initial size 2 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_repart_test.ht ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_repart_test.ht ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_repart_test.ht ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2: (TableToValueApply "{\"name\":\"NPartitionsTable\"}" (TableRead None False (TableNativeReader gs://gnomad-tmp/kc/mxra5_repart_test.ht ))) 2022-03-09 16:31:33 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:31:33 root: INFO: lowering result: TableToValueApply 2022-03-09 16:31:33 root: INFO: compiling and evaluating result: TableToValueApply 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 9: (Let (ArrayRef -1 (ToArray (ReadPartition Struct{min_window_size:Int32} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{min_window_size:+EInt32}\",\"_vType\":\"Struct{min_window_size:Int32}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://gnoma..."))) (I32 0)) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{path:String}] ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{path:String}] ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{path:String}] ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{path:String}] ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 3: (StreamLen (ToStream False (Literal Array[Struct{path:String}] ))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 4: (MakeTuple (0) (StreamLen (ToStream False (Literal Array[Struct{path:String}] )))) 2022-03-09 16:31:33 root: INFO: encoder cache miss (7 hits, 15 misses, 0.318) 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C918HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C918HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C919FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C919FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C920etypeEncode. 2022-03-09 16:31:33 root: INFO: instruction count: 7: __C920etypeEncode.apply 2022-03-09 16:31:33 root: INFO: instruction count: 25: __C920etypeEncode.__m921ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryENDEND 2022-03-09 16:31:33 root: INFO: instruction count: 35: __C920etypeEncode.__m922ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryEND 2022-03-09 16:31:33 root: INFO: instruction count: 13: __C920etypeEncode.__m923ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-03-09 16:31:33 root: INFO: instruction count: 16: __C920etypeEncode.__m924ENCODE_SStringPointer_TO_r_binary 2022-03-09 16:31:33 MemoryStore: INFO: Block broadcast_26 stored as values in memory (estimated size 12.6 KiB, free 12.6 GiB) 2022-03-09 16:31:33 MemoryStore: INFO: Block broadcast_26_piece0 stored as bytes in memory (estimated size 5.6 KiB, free 12.6 GiB) 2022-03-09 16:31:33 BlockManagerInfo: INFO: Added broadcast_26_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 5.6 KiB, free: 12.6 GiB) 2022-03-09 16:31:33 SparkContext: INFO: Created broadcast 26 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C897HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C897HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C898FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C898FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C899Compiled. 2022-03-09 16:31:33 root: INFO: instruction count: 25: __C899Compiled.apply 2022-03-09 16:31:33 root: INFO: instruction count: 22: __C899Compiled.__m900split_StreamLen 2022-03-09 16:31:33 root: INFO: instruction count: 9: __C899Compiled.setPartitionIndex 2022-03-09 16:31:33 root: INFO: instruction count: 4: __C899Compiled.addPartitionRegion 2022-03-09 16:31:33 root: INFO: instruction count: 4: __C899Compiled.setPool 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C899Compiled.addHailClassLoader 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C899Compiled.addFS 2022-03-09 16:31:33 root: INFO: instruction count: 55: __C899Compiled.addLiterals 2022-03-09 16:31:33 root: INFO: instruction count: 27: __C899Compiled.__m914DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryENDEND_TO_SBaseStructPointer 2022-03-09 16:31:33 root: INFO: instruction count: 58: __C899Compiled.__m915INPLACE_DECODE_r_array_of_r_struct_of_r_binaryEND_TO_r_array_of_r_struct_of_r_stringEND 2022-03-09 16:31:33 root: INFO: instruction count: 17: __C899Compiled.__m916INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-03-09 16:31:33 root: INFO: instruction count: 31: __C899Compiled.__m917INPLACE_DECODE_r_binary_TO_r_string 2022-03-09 16:31:33 root: INFO: took 163.687ms 2022-03-09 16:31:33 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-03-09 16:31:33 root: INFO: lowering result: I32 2022-03-09 16:31:33 root: INFO: compiling and evaluating result: I32 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (I32 120)) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:31:33 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 1: (Literal Tuple[Int32] ) 2022-03-09 16:31:33 root: INFO: encoder cache hit 2022-03-09 16:31:33 MemoryStore: INFO: Block broadcast_27 stored as values in memory (estimated size 80.0 B, free 12.6 GiB) 2022-03-09 16:31:33 MemoryStore: INFO: Block broadcast_27_piece0 stored as bytes in memory (estimated size 61.0 B, free 12.6 GiB) 2022-03-09 16:31:33 BlockManagerInfo: INFO: Added broadcast_27_piece0 in memory on kc-m.c.broad-mpg-gnomad.internal:44249 (size: 61.0 B, free: 12.6 GiB) 2022-03-09 16:31:33 SparkContext: INFO: Created broadcast 27 from broadcast at SparkBackend.scala:280 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C925HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C925HailClassLoaderContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C926FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C926FSContainer. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C927Compiled. 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C927Compiled.apply 2022-03-09 16:31:33 root: INFO: instruction count: 9: __C927Compiled.setPartitionIndex 2022-03-09 16:31:33 root: INFO: instruction count: 4: __C927Compiled.addPartitionRegion 2022-03-09 16:31:33 root: INFO: instruction count: 4: __C927Compiled.setPool 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C927Compiled.addHailClassLoader 2022-03-09 16:31:33 root: INFO: instruction count: 3: __C927Compiled.addFS 2022-03-09 16:31:33 root: INFO: instruction count: 35: __C927Compiled.addLiterals 2022-03-09 16:31:33 root: INFO: instruction count: 27: __C927Compiled.__m932DECODE_r_struct_of_r_struct_of_r_int32ENDEND_TO_SBaseStructPointer 2022-03-09 16:31:33 root: INFO: instruction count: 17: __C927Compiled.__m933INPLACE_DECODE_r_struct_of_r_int32END_TO_r_tuple_of_r_int32END 2022-03-09 16:31:33 root: INFO: instruction count: 10: __C927Compiled.__m934INPLACE_DECODE_r_int32_TO_r_int32 2022-03-09 16:31:33 root: INFO: took 14.238ms 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (I32 120) 2022-03-09 16:31:33 root: INFO: encoder cache hit 2022-03-09 16:31:33 root: INFO: finished execution of query hail_query_7 2022-03-09 16:31:33 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode total 183.972ms self 1.159ms children 182.813ms %children 99.37% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 2.234ms self 0.013ms children 2.221ms %children 99.42% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 2.209ms self 0.859ms children 1.351ms %children 61.13% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 1.351ms self 0.048ms children 1.303ms %children 96.45% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.185ms self 0.185ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.148ms self 0.148ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.351ms self 0.351ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.492ms self 0.492ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.134ms self 0.007ms children 0.127ms %children 94.65% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.115ms self 0.115ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 1.357ms self 0.008ms children 1.349ms %children 99.44% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 1.344ms self 0.493ms children 0.851ms %children 63.31% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.851ms self 0.032ms children 0.819ms %children 96.22% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.251ms self 0.251ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.348ms self 0.348ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.224ms self 0.005ms children 0.218ms %children 97.58% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.197ms self 0.197ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 178.228ms self 0.009ms children 178.219ms %children 100.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 178.207ms self 120.934ms children 57.273ms %children 32.14% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 3.067ms self 0.014ms children 3.053ms %children 99.53% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.214ms self 0.214ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 2.834ms self 1.032ms children 1.801ms %children 63.57% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 1.801ms self 0.033ms children 1.768ms %children 98.14% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.139ms self 0.139ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.211ms self 0.211ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.546ms self 0.546ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.124ms self 0.124ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.203ms self 0.203ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.247ms self 0.247ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.065ms self 0.006ms children 0.059ms %children 90.22% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.653ms self 0.006ms children 0.647ms %children 99.01% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.643ms self 0.254ms children 0.388ms %children 60.41% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.388ms self 0.029ms children 0.359ms %children 92.42% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.129ms self 0.129ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.082ms self 0.082ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.043ms self 0.005ms children 0.038ms %children 89.12% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.033ms self 0.004ms children 0.030ms %children 88.98% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.713ms self 0.006ms children 0.707ms %children 99.13% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.702ms self 0.314ms children 0.389ms %children 55.35% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.389ms self 0.016ms children 0.373ms %children 95.88% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.102ms self 0.102ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.155ms self 0.155ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 35.662ms self 29.815ms children 5.847ms %children 16.39% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 2.009ms self 0.008ms children 2.000ms %children 99.58% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 1.992ms self 0.685ms children 1.308ms %children 65.63% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 1.308ms self 0.028ms children 1.280ms %children 97.86% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.285ms self 0.285ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.343ms self 0.343ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.121ms self 0.121ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.242ms self 0.242ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.261ms self 0.261ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.037ms self 0.006ms children 0.032ms %children 85.09% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 1.381ms self 0.006ms children 1.375ms %children 99.54% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.369ms self 0.420ms children 0.949ms %children 69.32% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.949ms self 0.021ms children 0.928ms %children 97.75% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.457ms self 0.457ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.091ms self 0.091ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.197ms self 0.197ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.710ms self 0.006ms children 0.704ms %children 99.11% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.681ms self 0.681ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 1.129ms self 0.007ms children 1.122ms %children 99.36% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 1.116ms self 0.455ms children 0.660ms %children 59.18% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.660ms self 0.022ms children 0.639ms %children 96.72% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.247ms self 0.247ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.165ms self 0.165ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.581ms self 0.581ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 3.120ms self 3.120ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 0.385ms self 0.007ms children 0.377ms %children 98.06% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 0.370ms self 0.148ms children 0.222ms %children 60.01% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.222ms self 0.010ms children 0.212ms %children 95.54% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.025ms self 0.003ms children 0.023ms %children 89.99% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.169ms self 0.004ms children 0.165ms %children 97.64% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.000ms self 0.000ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.163ms self 0.071ms children 0.092ms %children 56.36% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.092ms self 0.007ms children 0.085ms %children 92.53% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.017ms self 0.003ms children 0.015ms %children 85.23% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.014ms self 0.002ms children 0.012ms %children 84.99% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.167ms self 0.003ms children 0.164ms %children 98.06% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.162ms self 0.077ms children 0.085ms %children 52.39% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.085ms self 0.007ms children 0.078ms %children 91.98% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.019ms self 0.019ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 10.885ms self 9.719ms children 1.165ms %children 10.71% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 0.572ms self 0.005ms children 0.568ms %children 99.18% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 0.565ms self 0.308ms children 0.256ms %children 45.42% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 0.256ms self 0.014ms children 0.242ms %children 94.35% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.108ms self 0.108ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.015ms self 0.002ms children 0.013ms %children 83.71% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 0.198ms self 0.004ms children 0.195ms %children 98.16% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 0.192ms self 0.098ms children 0.095ms %children 49.22% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 0.095ms self 0.007ms children 0.088ms %children 92.66% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 0.094ms self 0.003ms children 0.091ms %children 96.90% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 0.191ms self 0.004ms children 0.187ms %children 97.97% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 0.185ms self 0.095ms children 0.089ms %children 48.31% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 0.089ms self 0.007ms children 0.082ms %children 92.49% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 2.242ms self 2.242ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 0.441ms self 0.007ms children 0.434ms %children 98.41% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.430ms self 0.190ms children 0.239ms %children 55.73% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.239ms self 0.034ms children 0.206ms %children 86.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/Compile total 0.160ms self 0.160ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2022-03-09 16:31:33 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-03-09 16:32:27 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 4 2022-03-09 16:32:27 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 4 2022-03-09 16:32:27 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 4. 2022-03-09 16:32:27 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:32:27 ExecutorAllocationManager: INFO: Executors 4 removed due to idle timeout. 2022-03-09 16:32:27 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 3 2022-03-09 16:32:27 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 3 2022-03-09 16:32:27 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 3. 2022-03-09 16:32:27 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:32:27 ExecutorAllocationManager: INFO: Executors 3 removed due to idle timeout. 2022-03-09 16:32:30 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 3. 2022-03-09 16:32:30 DAGScheduler: INFO: Executor lost: 3 (epoch 2) 2022-03-09 16:32:30 BlockManagerMasterEndpoint: INFO: Trying to remove executor 3 from BlockManagerMaster. 2022-03-09 16:32:30 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(3, kc-w-0.c.broad-mpg-gnomad.internal, 33925, None) 2022-03-09 16:32:30 BlockManagerMaster: INFO: Removed 3 successfully in removeExecutor 2022-03-09 16:32:30 YarnScheduler: INFO: Executor 3 on kc-w-0.c.broad-mpg-gnomad.internal killed by driver. 2022-03-09 16:32:30 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 4. 2022-03-09 16:32:30 DAGScheduler: INFO: Executor lost: 4 (epoch 2) 2022-03-09 16:32:30 BlockManagerMasterEndpoint: INFO: Trying to remove executor 4 from BlockManagerMaster. 2022-03-09 16:32:30 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(4, kc-w-1.c.broad-mpg-gnomad.internal, 45587, None) 2022-03-09 16:32:30 BlockManagerMaster: INFO: Removed 4 successfully in removeExecutor 2022-03-09 16:32:30 YarnScheduler: INFO: Executor 4 on kc-w-1.c.broad-mpg-gnomad.internal killed by driver. 2022-03-09 16:32:32 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 1 2022-03-09 16:32:32 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 1 2022-03-09 16:32:32 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 1. 2022-03-09 16:32:32 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-03-09 16:32:32 ExecutorAllocationManager: INFO: Executors 1 removed due to idle timeout. 2022-03-09 16:32:34 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 1. 2022-03-09 16:32:34 DAGScheduler: INFO: Executor lost: 1 (epoch 2) 2022-03-09 16:32:34 BlockManagerMasterEndpoint: INFO: Trying to remove executor 1 from BlockManagerMaster. 2022-03-09 16:32:34 YarnScheduler: INFO: Executor 1 on kc-w-0.c.broad-mpg-gnomad.internal killed by driver. 2022-03-09 16:32:34 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(1, kc-w-0.c.broad-mpg-gnomad.internal, 41631, None) 2022-03-09 16:32:34 BlockManagerMaster: INFO: Removed 1 successfully in removeExecutor 2022-03-09 16:40:52 BlockManagerInfo: INFO: Removed broadcast_18_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 1725.0 B, free: 12.6 GiB) 2022-03-09 16:40:52 BlockManagerInfo: INFO: Removed broadcast_18_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 1725.0 B, free: 2.9 GiB) 2022-03-09 16:40:52 BlockManagerInfo: INFO: Removed broadcast_25_piece0 on kc-m.c.broad-mpg-gnomad.internal:44249 in memory (size: 58.7 KiB, free: 12.6 GiB) 2022-03-09 16:40:52 BlockManagerInfo: INFO: Removed broadcast_25_piece0 on kc-w-1.c.broad-mpg-gnomad.internal:44485 in memory (size: 58.7 KiB, free: 2.9 GiB)