2022-06-03 16:49:08 Hail: INFO: Running Hail version 0.2.95-513139587f57 2022-06-03 16:49:09 SparkContext: WARN: Using an existing SparkContext; some configuration may not take effect. 2022-06-03 16:49:12 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.34:56786) with ID 2, ResourceProfileId 0 2022-06-03 16:49:12 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.17:36024) with ID 1, ResourceProfileId 0 2022-06-03 16:49:12 ExecutorMonitor: INFO: New executor 2 has registered (new total is 1) 2022-06-03 16:49:12 ExecutorMonitor: INFO: New executor 1 has registered (new total is 2) 2022-06-03 16:49:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-16.c.broad-mpg-gnomad.internal:39807 with 5.1 GiB RAM, BlockManagerId(2, kc1-w-16.c.broad-mpg-gnomad.internal, 39807, None) 2022-06-03 16:49:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-9.c.broad-mpg-gnomad.internal:43647 with 5.1 GiB RAM, BlockManagerId(1, kc1-w-9.c.broad-mpg-gnomad.internal, 43647, None) 2022-06-03 16:50:13 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:50:13 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0. 2022-06-03 16:50:13 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 2 2022-06-03 16:50:13 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 2 2022-06-03 16:50:13 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 2. 2022-06-03 16:50:13 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:50:13 ExecutorAllocationManager: INFO: Executors 2 removed due to idle timeout. 2022-06-03 16:50:16 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 2. 2022-06-03 16:50:16 DAGScheduler: INFO: Executor lost: 2 (epoch 0) 2022-06-03 16:50:16 BlockManagerMasterEndpoint: INFO: Trying to remove executor 2 from BlockManagerMaster. 2022-06-03 16:50:16 YarnScheduler: INFO: Executor 2 on kc1-w-16.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:50:16 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(2, kc1-w-16.c.broad-mpg-gnomad.internal, 39807, None) 2022-06-03 16:50:16 BlockManagerMaster: INFO: Removed 2 successfully in removeExecutor 2022-06-03 16:50:52 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:50:52 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 50.2 KiB, free 21.7 GiB) 2022-06-03 16:50:53 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 21.7 GiB) 2022-06-03 16:50:53 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 3.2 KiB, free: 21.7 GiB) 2022-06-03 16:50:53 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:313 2022-06-03 16:50:54 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:50:54 root: INFO: timing SparkBackend.parse_table_ir total 2.181s self 2.181s children 0.000ms %children 0.00% 2022-06-03 16:50:54 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:50:54 root: INFO: JSON: JObject(List((name,JString(TableCalculateNewPartitions)), (nPartitions,JInt(30000)))) 2022-06-03 16:50:54 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:50:54 root: INFO: timing SparkBackend.parse_value_ir total 39.010ms self 39.010ms children 0.000ms %children 0.00% 2022-06-03 16:50:54 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:50:54 root: INFO: starting execution of query hail_query_1 of initial size 2 2022-06-03 16:50:54 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":30000}" (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht ))) 2022-06-03 16:50:54 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":30000}" (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht ))) 2022-06-03 16:50:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":30000}" (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht ))) 2022-06-03 16:50:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2: (TableToValueApply "{\"name\":\"TableCalculateNewPartitions\",\"nPartitions\":30000}" (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht ))) 2022-06-03 16:50:55 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:50:55 root: INFO: lowering result: TableToValueApply 2022-06-03 16:50:55 root: INFO: compiling and evaluating result: TableToValueApply 2022-06-03 16:50:55 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 118: (Let __iruid_8 (StreamFlatMap __iruid_7 (ToStream False (Let __iruid_1 (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)) (Let __iruid_6 (CollectDistributedArray __iruid_2 __iruid_3 (ToStream False (Literal Array[Struct{path:String}] )) (MakeStruct (__iruid_1 (Ref __iruid_1))) (Let __iruid_1 (GetField __iruid_1 (Ref __iruid_3)) (StreamAgg __iruid_4 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_2))) (ToArray (StreamFlatMap __iruid_5 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_4)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_4) (Ref __iruid_4))) (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_4) (Ref __iruid_4))))) (ToStream False (Ref __iruid_5))))))) (Ref __iruid_6)))) (ToStream False (Ref __iruid_7))) (Let __iruid_12 (ToArray (StreamFlatMap __iruid_11 (StreamGroupByKey () (ToStream False (ArraySort __iruid_9 __iruid_10 (Ref __iruid_8) (ApplyComparisonOp LT (Ref __iruid_9) (Ref __iruid_10))))) (StreamTake (Ref __iruid_11) (I32 1)))) (Let __iruid_13 (ArrayLen (Ref __iruid_12)) (Let __iruid_14 (If (ApplyComparisonOp LT (Ref __iruid_13) (I32 30000)) (Ref __iruid_13) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_14) (I32 0)) (MakeArray Array[Interval[Struct{}]]) (Let __iruid_15 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_13) (ApplyBinaryPrimOp Subtract (Ref __iruid_14) (I32 1))) (Ref __iruid_14)) (ToArray (StreamMap __iruid_16 (StreamRange -1 False (I32 0) (Ref __iruid_13) (Ref __iruid_15)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_16) (Ref __iruid_15)) (ApplyBinaryPrimOp Subtract (Ref __iruid_13) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_12) (Ref __iruid_16)) (ArrayRef -1 (Ref __iruid_12) (ApplyBinaryPrimOp Add (Ref __iruid_16) (Ref __iruid_15))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_12) (Ref __iruid_16)) (ArrayRef -1 (Ref __iruid_12) (ApplyBinaryPrimOp Subtract (Ref __iruid_13) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:56 root: INFO: Prune: MakeStruct: eliminating field '__iruid_1' 2022-06-03 16:50:56 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 120: (Let __iruid_55 (ToArray (StreamFlatMap __iruid_56 (StreamGroupByKey () (ToStream False (ArraySort __iruid_57 __iruid_58 (StreamFlatMap __iruid_59 (ToStream False (CollectDistributedArray __iruid_60 __iruid_61 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_62 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_60))) (ToArray (StreamFlatMap __iruid_63 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_62)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_64 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_62) (Ref __iruid_62)))) (If (IsNA (Ref __iruid_64)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_65 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_62) (Ref __iruid_62)))) (If (IsNA (Ref __iruid_65)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_63))))))) (ToStream False (Ref __iruid_59))) (ApplyComparisonOp LT (Ref __iruid_57) (Ref __iruid_58))))) (StreamTake (Ref __iruid_56) (I32 1)))) (Let __iruid_66 (ArrayLen (Ref __iruid_55)) (Let __iruid_67 (If (ApplyComparisonOp LT (Ref __iruid_66) (I32 30000)) (Ref __iruid_66) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_67) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_68 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_66) (ApplyBinaryPrimOp Subtract (Ref __iruid_67) (I32 1))) (Ref __iruid_67)) (ToArray (StreamMap __iruid_69 (StreamRange -1 False (I32 0) (Ref __iruid_66) (Ref __iruid_68)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_69) (Ref __iruid_68)) (ApplyBinaryPrimOp Subtract (Ref __iruid_66) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_55) (Ref __iruid_69)) (ArrayRef -1 (Ref __iruid_55) (ApplyBinaryPrimOp Add (Ref __iruid_69) (Ref __iruid_68))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_55) (Ref __iruid_69)) (ArrayRef -1 (Ref __iruid_55) (ApplyBinaryPrimOp Subtract (Ref __iruid_66) (I32 1))) (True) (True)))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 120: (Let __iruid_55 (ToArray (StreamFlatMap __iruid_56 (StreamGroupByKey () (ToStream False (ArraySort __iruid_57 __iruid_58 (StreamFlatMap __iruid_59 (ToStream False (CollectDistributedArray __iruid_60 __iruid_61 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_62 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_60))) (ToArray (StreamFlatMap __iruid_63 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_62)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_64 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_62) (Ref __iruid_62)))) (If (IsNA (Ref __iruid_64)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_65 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_62) (Ref __iruid_62)))) (If (IsNA (Ref __iruid_65)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_63))))))) (ToStream False (Ref __iruid_59))) (ApplyComparisonOp LT (Ref __iruid_57) (Ref __iruid_58))))) (StreamTake (Ref __iruid_56) (I32 1)))) (Let __iruid_66 (ArrayLen (Ref __iruid_55)) (Let __iruid_67 (If (ApplyComparisonOp LT (Ref __iruid_66) (I32 30000)) (Ref __iruid_66) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_67) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_68 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_66) (ApplyBinaryPrimOp Subtract (Ref __iruid_67) (I32 1))) (Ref __iruid_67)) (ToArray (StreamMap __iruid_69 (StreamRange -1 False (I32 0) (Ref __iruid_66) (Ref __iruid_68)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_69) (Ref __iruid_68)) (ApplyBinaryPrimOp Subtract (Ref __iruid_66) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_55) (Ref __iruid_69)) (ArrayRef -1 (Ref __iruid_55) (ApplyBinaryPrimOp Add (Ref __iruid_69) (Ref __iruid_68))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_55) (Ref __iruid_69)) (ArrayRef -1 (Ref __iruid_55) (ApplyBinaryPrimOp Subtract (Ref __iruid_66) (I32 1))) (True) (True)))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 120: (Let __iruid_100 (ToArray (StreamFlatMap __iruid_101 (StreamGroupByKey () (ToStream False (ArraySort __iruid_102 __iruid_103 (StreamFlatMap __iruid_104 (ToStream False (CollectDistributedArray __iruid_105 __iruid_106 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_107 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_105))) (ToArray (StreamFlatMap __iruid_108 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_107)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_109 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_107) (Ref __iruid_107)))) (If (IsNA (Ref __iruid_109)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_110 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_107) (Ref __iruid_107)))) (If (IsNA (Ref __iruid_110)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_108))))))) (ToStream False (Ref __iruid_104))) (ApplyComparisonOp LT (Ref __iruid_102) (Ref __iruid_103))))) (StreamTake (Ref __iruid_101) (I32 1)))) (Let __iruid_111 (ArrayLen (Ref __iruid_100)) (Let __iruid_112 (If (ApplyComparisonOp LT (Ref __iruid_111) (I32 30000)) (Ref __iruid_111) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_112) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_113 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_111) (ApplyBinaryPrimOp Subtract (Ref __iruid_112) (I32 1))) (Ref __iruid_112)) (ToArray (StreamMap __iruid_114 (StreamRange -1 False (I32 0) (Ref __iruid_111) (Ref __iruid_113)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_114) (Ref __iruid_113)) (ApplyBinaryPrimOp Subtract (Ref __iruid_111) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_100) (Ref __iruid_114)) (ArrayRef -1 (Ref __iruid_100) (ApplyBinaryPrimOp Add (Ref __iruid_114) (Ref __iruid_113))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_100) (Ref __iruid_114)) (ArrayRef -1 (Ref __iruid_100) (ApplyBinaryPrimOp Subtract (Ref __iruid_111) (I32 1))) (True) (True)))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 120: (Let __iruid_100 (ToArray (StreamFlatMap __iruid_101 (StreamGroupByKey () (ToStream False (ArraySort __iruid_102 __iruid_103 (StreamFlatMap __iruid_104 (ToStream False (CollectDistributedArray __iruid_105 __iruid_106 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_107 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_105))) (ToArray (StreamFlatMap __iruid_108 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_107)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_109 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_107) (Ref __iruid_107)))) (If (IsNA (Ref __iruid_109)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_110 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_107) (Ref __iruid_107)))) (If (IsNA (Ref __iruid_110)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_108))))))) (ToStream False (Ref __iruid_104))) (ApplyComparisonOp LT (Ref __iruid_102) (Ref __iruid_103))))) (StreamTake (Ref __iruid_101) (I32 1)))) (Let __iruid_111 (ArrayLen (Ref __iruid_100)) (Let __iruid_112 (If (ApplyComparisonOp LT (Ref __iruid_111) (I32 30000)) (Ref __iruid_111) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_112) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_113 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_111) (ApplyBinaryPrimOp Subtract (Ref __iruid_112) (I32 1))) (Ref __iruid_112)) (ToArray (StreamMap __iruid_114 (StreamRange -1 False (I32 0) (Ref __iruid_111) (Ref __iruid_113)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_114) (Ref __iruid_113)) (ApplyBinaryPrimOp Subtract (Ref __iruid_111) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_100) (Ref __iruid_114)) (ArrayRef -1 (Ref __iruid_100) (ApplyBinaryPrimOp Add (Ref __iruid_114) (Ref __iruid_113))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_100) (Ref __iruid_114)) (ArrayRef -1 (Ref __iruid_100) (ApplyBinaryPrimOp Subtract (Ref __iruid_111) (I32 1))) (True) (True)))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 120: (Let __iruid_145 (ToArray (StreamFlatMap __iruid_146 (StreamGroupByKey () (ToStream False (ArraySort __iruid_147 __iruid_148 (StreamFlatMap __iruid_149 (ToStream False (CollectDistributedArray __iruid_150 __iruid_151 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_152 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_150))) (ToArray (StreamFlatMap __iruid_153 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_152)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_154 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_152) (Ref __iruid_152)))) (If (IsNA (Ref __iruid_154)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_155 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_152) (Ref __iruid_152)))) (If (IsNA (Ref __iruid_155)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_153))))))) (ToStream False (Ref __iruid_149))) (ApplyComparisonOp LT (Ref __iruid_147) (Ref __iruid_148))))) (StreamTake (Ref __iruid_146) (I32 1)))) (Let __iruid_156 (ArrayLen (Ref __iruid_145)) (Let __iruid_157 (If (ApplyComparisonOp LT (Ref __iruid_156) (I32 30000)) (Ref __iruid_156) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_157) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_158 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_156) (ApplyBinaryPrimOp Subtract (Ref __iruid_157) (I32 1))) (Ref __iruid_157)) (ToArray (StreamMap __iruid_159 (StreamRange -1 False (I32 0) (Ref __iruid_156) (Ref __iruid_158)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_159) (Ref __iruid_158)) (ApplyBinaryPrimOp Subtract (Ref __iruid_156) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_145) (Ref __iruid_159)) (ArrayRef -1 (Ref __iruid_145) (ApplyBinaryPrimOp Add (Ref __iruid_159) (Ref __iruid_158))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_145) (Ref __iruid_159)) (ArrayRef -1 (Ref __iruid_145) (ApplyBinaryPrimOp Subtract (Ref __iruid_156) (I32 1))) (True) (True)))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 121: (MakeTuple (0) (Let __iruid_145 (ToArray (StreamFlatMap __iruid_146 (StreamGroupByKey () (ToStream False (ArraySort __iruid_147 __iruid_148 (StreamFlatMap __iruid_149 (ToStream False (CollectDistributedArray __iruid_150 __iruid_151 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_152 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_150))) (ToArray (StreamFlatMap __iruid_153 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_152)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_154 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_152) (Ref __iruid_152)))) (If (IsNA (Ref __iruid_154)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_155 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_152) (Ref __iruid_152)))) (If (IsNA (Ref __iruid_155)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_153))))))) (ToStream False (Ref __iruid_149))) (ApplyComparisonOp LT (Ref __iruid_147) (Ref __iruid_148))))) (StreamTake (Ref __iruid_146) (I32 1)))) (Let __iruid_156 (ArrayLen (Ref __iruid_145)) (Let __iruid_157 (If (ApplyComparisonOp LT (Ref __iruid_156) (I32 30000)) (Ref __iruid_156) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_157) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_158 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_156) (ApplyBinaryPrimOp Subtract (Ref __iruid_157) (I32 1))) (Ref __iruid_157)) (ToArray (StreamMap __iruid_159 (StreamRange -1 False (I32 0) (Ref __iruid_156) (Ref __iruid_158)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_159) (Ref __iruid_158)) (ApplyBinaryPrimOp Subtract (Ref __iruid_156) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_145) (Ref __iruid_159)) (ArrayRef -1 (Ref __iruid_145) (ApplyBinaryPrimOp Add (Ref __iruid_159) (Ref __iruid_158))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_145) (Ref __iruid_159)) (ArrayRef -1 (Ref __iruid_145) (ApplyBinaryPrimOp Subtract (Ref __iruid_156) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 121: (MakeTuple (0) (Let __iruid_190 (ToArray (StreamFlatMap __iruid_191 (StreamGroupByKey () (ToStream False (ArraySort __iruid_192 __iruid_193 (StreamFlatMap __iruid_194 (ToStream False (CollectDistributedArray __iruid_195 __iruid_196 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_197 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_195))) (ToArray (StreamFlatMap __iruid_198 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_197)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_199 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_197) (Ref __iruid_197)))) (If (IsNA (Ref __iruid_199)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_200 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_197) (Ref __iruid_197)))) (If (IsNA (Ref __iruid_200)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_198))))))) (ToStream False (Ref __iruid_194))) (ApplyComparisonOp LT (Ref __iruid_192) (Ref __iruid_193))))) (StreamTake (Ref __iruid_191) (I32 1)))) (Let __iruid_201 (ArrayLen (Ref __iruid_190)) (Let __iruid_202 (If (ApplyComparisonOp LT (Ref __iruid_201) (I32 30000)) (Ref __iruid_201) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_202) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_203 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_201) (ApplyBinaryPrimOp Subtract (Ref __iruid_202) (I32 1))) (Ref __iruid_202)) (ToArray (StreamMap __iruid_204 (StreamRange -1 False (I32 0) (Ref __iruid_201) (Ref __iruid_203)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_204) (Ref __iruid_203)) (ApplyBinaryPrimOp Subtract (Ref __iruid_201) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_190) (Ref __iruid_204)) (ArrayRef -1 (Ref __iruid_190) (ApplyBinaryPrimOp Add (Ref __iruid_204) (Ref __iruid_203))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_190) (Ref __iruid_204)) (ArrayRef -1 (Ref __iruid_190) (ApplyBinaryPrimOp Subtract (Ref __iruid_201) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 121: (MakeTuple (0) (Let __iruid_190 (ToArray (StreamFlatMap __iruid_191 (StreamGroupByKey () (ToStream False (ArraySort __iruid_192 __iruid_193 (StreamFlatMap __iruid_194 (ToStream False (CollectDistributedArray __iruid_195 __iruid_196 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_197 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_195))) (ToArray (StreamFlatMap __iruid_198 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_197)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_199 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_197) (Ref __iruid_197)))) (If (IsNA (Ref __iruid_199)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_200 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_197) (Ref __iruid_197)))) (If (IsNA (Ref __iruid_200)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_198))))))) (ToStream False (Ref __iruid_194))) (ApplyComparisonOp LT (Ref __iruid_192) (Ref __iruid_193))))) (StreamTake (Ref __iruid_191) (I32 1)))) (Let __iruid_201 (ArrayLen (Ref __iruid_190)) (Let __iruid_202 (If (ApplyComparisonOp LT (Ref __iruid_201) (I32 30000)) (Ref __iruid_201) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_202) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_203 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_201) (ApplyBinaryPrimOp Subtract (Ref __iruid_202) (I32 1))) (Ref __iruid_202)) (ToArray (StreamMap __iruid_204 (StreamRange -1 False (I32 0) (Ref __iruid_201) (Ref __iruid_203)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_204) (Ref __iruid_203)) (ApplyBinaryPrimOp Subtract (Ref __iruid_201) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_190) (Ref __iruid_204)) (ArrayRef -1 (Ref __iruid_190) (ApplyBinaryPrimOp Add (Ref __iruid_204) (Ref __iruid_203))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_190) (Ref __iruid_204)) (ArrayRef -1 (Ref __iruid_190) (ApplyBinaryPrimOp Subtract (Ref __iruid_201) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:57 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 121: (MakeTuple (0) (Let __iruid_235 (ToArray (StreamFlatMap __iruid_236 (StreamGroupByKey () (ToStream False (ArraySort __iruid_237 __iruid_238 (StreamFlatMap __iruid_239 (ToStream False (CollectDistributedArray __iruid_240 __iruid_241 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (StreamAgg __iruid_242 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_240))) (ToArray (StreamFlatMap __iruid_243 (ToStream False (MakeArray Array[Array[Struct{}]] (ApplyAggOp TakeBy ((I32 4000)) ((SelectFields () (Ref __iruid_242)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (ToArray (StreamMap __iruid_244 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_242) (Ref __iruid_242)))) (If (IsNA (Ref __iruid_244)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_245 (ToStream False (ApplyAggOp TakeBy ((I32 1)) ((Ref __iruid_242) (Ref __iruid_242)))) (If (IsNA (Ref __iruid_245)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_243))))))) (ToStream False (Ref __iruid_239))) (ApplyComparisonOp LT (Ref __iruid_237) (Ref __iruid_238))))) (StreamTake (Ref __iruid_236) (I32 1)))) (Let __iruid_246 (ArrayLen (Ref __iruid_235)) (Let __iruid_247 (If (ApplyComparisonOp LT (Ref __iruid_246) (I32 30000)) (Ref __iruid_246) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_247) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_248 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_246) (ApplyBinaryPrimOp Subtract (Ref __iruid_247) (I32 1))) (Ref __iruid_247)) (ToArray (StreamMap __iruid_249 (StreamRange -1 False (I32 0) (Ref __iruid_246) (Ref __iruid_248)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_249) (Ref __iruid_248)) (ApplyBinaryPrimOp Subtract (Ref __iruid_246) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_235) (Ref __iruid_249)) (ArrayRef -1 (Ref __iruid_235) (ApplyBinaryPrimOp Add (Ref __iruid_249) (Ref __iruid_248))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_235) (Ref __iruid_249)) (ArrayRef -1 (Ref __iruid_235) (ApplyBinaryPrimOp Subtract (Ref __iruid_246) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:57 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 139: (MakeTuple (0) (Let __iruid_235 (ToArray (StreamFlatMap __iruid_236 (StreamGroupByKey () (ToStream False (ArraySort __iruid_237 __iruid_238 (StreamFlatMap __iruid_239 (ToStream False (CollectDistributedArray __iruid_240 __iruid_241 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (Let __iruid_250 (RunAgg ((TakeByStateSig +PCStruct{} +PFloat64) (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}) (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) (Begin (Begin (InitOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64)) ((I32 4000))) (InitOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((I32 1))) (InitOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((I32 1)))) (StreamFor __iruid_242 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_240))) (Begin (SeqOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64)) ((SelectFields () (Ref __iruid_242)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (SeqOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((Ref __iruid_242) (Ref __iruid_242))) (SeqOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((Ref __iruid_242) (Ref __iruid_242)))))) (MakeTuple (0 1 2) (ResultOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64))) (ResultOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}))) (ResultOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}))))) (ToArray (StreamFlatMap __iruid_243 (ToStream False (MakeArray Array[Array[Struct{}]] (GetTupleElement 0 (Ref __iruid_250)) (ToArray (StreamMap __iruid_244 (ToStream False (GetTupleElement 1 (Ref __iruid_250))) (If (IsNA (Ref __iruid_244)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_245 (ToStream False (GetTupleElement 2 (Ref __iruid_250))) (If (IsNA (Ref __iruid_245)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_243))))))) (ToStream False (Ref __iruid_239))) (ApplyComparisonOp LT (Ref __iruid_237) (Ref __iruid_238))))) (StreamTake (Ref __iruid_236) (I32 1)))) (Let __iruid_246 (ArrayLen (Ref __iruid_235)) (Let __iruid_247 (If (ApplyComparisonOp LT (Ref __iruid_246) (I32 30000)) (Ref __iruid_246) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_247) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_248 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_246) (ApplyBinaryPrimOp Subtract (Ref __iruid_247) (I32 1))) (Ref __iruid_247)) (ToArray (StreamMap __iruid_249 (StreamRange -1 False (I32 0) (Ref __iruid_246) (Ref __iruid_248)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_249) (Ref __iruid_248)) (ApplyBinaryPrimOp Subtract (Ref __iruid_246) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_235) (Ref __iruid_249)) (ArrayRef -1 (Ref __iruid_235) (ApplyBinaryPrimOp Add (Ref __iruid_249) (Ref __iruid_248))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_235) (Ref __iruid_249)) (ArrayRef -1 (Ref __iruid_235) (ApplyBinaryPrimOp Subtract (Ref __iruid_246) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:57 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 139: (MakeTuple (0) (Let __iruid_283 (ToArray (StreamFlatMap __iruid_284 (StreamGroupByKey () (ToStream False (ArraySort __iruid_285 __iruid_286 (StreamFlatMap __iruid_287 (ToStream False (CollectDistributedArray __iruid_288 __iruid_289 (ToStream False (Literal Array[Struct{path:String}] )) (Literal Struct{} ) (Let __iruid_290 (RunAgg ((TakeByStateSig +PCStruct{} +PFloat64) (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}) (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) (Begin (Begin (InitOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64)) ((I32 4000))) (InitOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((I32 1))) (InitOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((I32 1)))) (StreamFor __iruid_291 (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_288))) (Begin (SeqOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64)) ((SelectFields () (Ref __iruid_291)) (ApplySeeded rand_unif 1 Float64 (F64 0.0) (F64 1.0)))) (SeqOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((Ref __iruid_291) (Ref __iruid_291))) (SeqOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]})) ((Ref __iruid_291) (Ref __iruid_291)))))) (MakeTuple (0 1 2) (ResultOp 0 (TakeBy (TakeByStateSig +PCStruct{} +PFloat64))) (ResultOp 1 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}))) (ResultOp 2 (TakeBy (TakeByStateSig +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]} +PCStruct{locus:PCLocus(GRCh37),alleles:PCArray[PCString],polyphen:PFloat64,sift:+PCStruct{prediction:PCString,score:PFloat64},codons:PCString,most_severe_consequence:PCString,transcript:PCString,ref:PCString,alt:PCString,interval:PCInterval[PCLocus(GRCh37)],oe:PFloat64,misbad:PFloat64,fitted_score:PFloat64,gnomad_scores:+PCArray[+PFloat64]}))))) (ToArray (StreamFlatMap __iruid_292 (ToStream False (MakeArray Array[Array[Struct{}]] (GetTupleElement 0 (Ref __iruid_290)) (ToArray (StreamMap __iruid_293 (ToStream False (GetTupleElement 1 (Ref __iruid_290))) (If (IsNA (Ref __iruid_293)) (NA Struct{}) (Literal Struct{} )))) (ToArray (StreamMap __iruid_294 (ToStream False (GetTupleElement 2 (Ref __iruid_290))) (If (IsNA (Ref __iruid_294)) (NA Struct{}) (Literal Struct{} )))))) (ToStream False (Ref __iruid_292))))))) (ToStream False (Ref __iruid_287))) (ApplyComparisonOp LT (Ref __iruid_285) (Ref __iruid_286))))) (StreamTake (Ref __iruid_284) (I32 1)))) (Let __iruid_295 (ArrayLen (Ref __iruid_283)) (Let __iruid_296 (If (ApplyComparisonOp LT (Ref __iruid_295) (I32 30000)) (Ref __iruid_295) (I32 30000)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_296) (I32 0)) (Literal Array[Interval[Struct{}]] ) (Let __iruid_297 (ApplyBinaryPrimOp RoundToNegInfDivide (ApplyBinaryPrimOp Add (Ref __iruid_295) (ApplyBinaryPrimOp Subtract (Ref __iruid_296) (I32 1))) (Ref __iruid_296)) (ToArray (StreamMap __iruid_298 (StreamRange -1 False (I32 0) (Ref __iruid_295) (Ref __iruid_297)) (If (ApplyComparisonOp LT (ApplyBinaryPrimOp Add (Ref __iruid_298) (Ref __iruid_297)) (ApplyBinaryPrimOp Subtract (Ref __iruid_295) (I32 1))) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_283) (Ref __iruid_298)) (ArrayRef -1 (Ref __iruid_283) (ApplyBinaryPrimOp Add (Ref __iruid_298) (Ref __iruid_297))) (True) (False)) (ApplySpecial -1 Interval () Interval[Struct{}] (ArrayRef -1 (Ref __iruid_283) (Ref __iruid_298)) (ArrayRef -1 (Ref __iruid_283) (ApplyBinaryPrimOp Subtract (Ref __iruid_295) (I32 1))) (True) (True))))))))))) 2022-06-03 16:50:58 root: INFO: encoder cache miss (0 hits, 1 misses, 0.000) 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C213HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C213HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C214FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C214FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C215etypeEncode. 2022-06-03 16:50:59 root: INFO: instruction count: 7: __C215etypeEncode.apply 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C215etypeEncode.__m216ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-06-03 16:50:59 root: INFO: instruction count: 1: __C215etypeEncode.__m217ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:50:59 MemoryStore: INFO: Block broadcast_1 stored as values in memory (estimated size 96.0 B, free 21.7 GiB) 2022-06-03 16:50:59 MemoryStore: INFO: Block broadcast_1_piece0 stored as bytes in memory (estimated size 57.0 B, free 21.7 GiB) 2022-06-03 16:50:59 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 57.0 B, free: 21.7 GiB) 2022-06-03 16:50:59 SparkContext: INFO: Created broadcast 1 from broadcast at SparkBackend.scala:313 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C17HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C17HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C18FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C18FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C19collect_distributed_array. 2022-06-03 16:50:59 root: INFO: instruction count: 357: __C19collect_distributed_array.apply 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C19collect_distributed_array.apply 2022-06-03 16:50:59 root: INFO: instruction count: 27: __C19collect_distributed_array.__m20DECODE_r_struct_of_r_struct_of_r_binaryENDEND_TO_SBaseStructPointer 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C19collect_distributed_array.__m21INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:50:59 root: INFO: instruction count: 31: __C19collect_distributed_array.__m22INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:50:59 root: INFO: instruction count: 27: __C19collect_distributed_array.__m24DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:50:59 root: INFO: instruction count: 8: __C19collect_distributed_array.__m25INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C19collect_distributed_array.__m40i_gt_j 2022-06-03 16:50:59 root: INFO: instruction count: 79: __C19collect_distributed_array.__m41ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C19collect_distributed_array.__m42ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 8: __C19collect_distributed_array.__m43ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C19collect_distributed_array.__m44ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 8: __C19collect_distributed_array.__m45ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 18: __C19collect_distributed_array.__m46swap 2022-06-03 16:50:59 root: INFO: instruction count: 74: __C19collect_distributed_array.__m47rebalance_up 2022-06-03 16:50:59 root: INFO: instruction count: 150: __C19collect_distributed_array.__m48rebalance_down 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C19collect_distributed_array.__m61i_gt_j 2022-06-03 16:50:59 root: INFO: instruction count: 71: __C19collect_distributed_array.__m62ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C19collect_distributed_array.__m63ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 177: __C19collect_distributed_array.__m64ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 34: __C19collect_distributed_array.__m65ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 74: __C19collect_distributed_array.__m66ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C19collect_distributed_array.__m68ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 89: __C19collect_distributed_array.__m69ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 34: __C19collect_distributed_array.__m70ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 188: __C19collect_distributed_array.__m71ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 30: __C19collect_distributed_array.__m72ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 30: __C19collect_distributed_array.__m73ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C19collect_distributed_array.__m74ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 199: __C19collect_distributed_array.__m75ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 34: __C19collect_distributed_array.__m76ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 398: __C19collect_distributed_array.__m77ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 15: __C19collect_distributed_array.__m78ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 82: __C19collect_distributed_array.__m79ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 18: __C19collect_distributed_array.__m80swap 2022-06-03 16:50:59 root: INFO: instruction count: 74: __C19collect_distributed_array.__m81rebalance_up 2022-06-03 16:50:59 root: INFO: instruction count: 150: __C19collect_distributed_array.__m82rebalance_down 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C19collect_distributed_array.__m95i_gt_j 2022-06-03 16:50:59 root: INFO: instruction count: 18: __C19collect_distributed_array.__m96swap 2022-06-03 16:50:59 root: INFO: instruction count: 74: __C19collect_distributed_array.__m97rebalance_up 2022-06-03 16:50:59 root: INFO: instruction count: 150: __C19collect_distributed_array.__m98rebalance_down 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C19collect_distributed_array.__m99begin_group_0 2022-06-03 16:50:59 root: INFO: instruction count: 366: __C19collect_distributed_array.__m100begin_group_0 2022-06-03 16:50:59 root: INFO: instruction count: 113: __C19collect_distributed_array.__m101split_StreamFor 2022-06-03 16:50:59 root: INFO: instruction count: 382: __C19collect_distributed_array.__m106DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryANDo_float64ANDr_struct_of_o_binaryANDo_float64ENDANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolENDANDo_float64ANDo_float64ANDo_float64ANDr_array_of_r_float64END_TO_SBaseStructPointer 2022-06-03 16:50:59 root: INFO: instruction count: 26: __C19collect_distributed_array.__m107INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus 2022-06-03 16:50:59 root: INFO: instruction count: 10: __C19collect_distributed_array.__m108INPLACE_DECODE_r_int32_TO_r_int32 2022-06-03 16:50:59 root: INFO: instruction count: 114: __C19collect_distributed_array.__m109INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string 2022-06-03 16:50:59 root: INFO: instruction count: 31: __C19collect_distributed_array.__m110INPLACE_DECODE_o_binary_TO_o_string 2022-06-03 16:50:59 root: INFO: instruction count: 10: __C19collect_distributed_array.__m111INPLACE_DECODE_o_float64_TO_o_float64 2022-06-03 16:50:59 root: INFO: instruction count: 75: __C19collect_distributed_array.__m112INPLACE_DECODE_r_struct_of_o_binaryANDo_float64END_TO_r_struct_of_o_stringANDo_float64END 2022-06-03 16:50:59 root: INFO: instruction count: 93: __C19collect_distributed_array.__m113INPLACE_DECODE_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_o_interval_of_o_locus 2022-06-03 16:50:59 root: INFO: instruction count: 10: __C19collect_distributed_array.__m114INPLACE_DECODE_r_bool_TO_r_bool 2022-06-03 16:50:59 root: INFO: instruction count: 58: __C19collect_distributed_array.__m115INPLACE_DECODE_r_array_of_r_float64_TO_r_array_of_r_float64 2022-06-03 16:50:59 root: INFO: instruction count: 10: __C19collect_distributed_array.__m116INPLACE_DECODE_r_float64_TO_r_float64 2022-06-03 16:50:59 root: INFO: instruction count: 38: __C19collect_distributed_array.__m118begin_group_0 2022-06-03 16:50:59 root: INFO: instruction count: 397: __C19collect_distributed_array.__m120take_by_seqop 2022-06-03 16:50:59 root: INFO: instruction count: 597: __C19collect_distributed_array.__m122take_by_seqop 2022-06-03 16:50:59 root: INFO: instruction count: 120: __C19collect_distributed_array.__m124take_by_garbage_collect 2022-06-03 16:50:59 root: INFO: instruction count: 597: __C19collect_distributed_array.__m125take_by_seqop 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C19collect_distributed_array.__m127ord_compare 2022-06-03 16:50:59 root: INFO: instruction count: 177: __C19collect_distributed_array.__m128ord_compareNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 120: __C19collect_distributed_array.__m129take_by_garbage_collect 2022-06-03 16:50:59 root: INFO: instruction count: 165: __C19collect_distributed_array.__m130take_by_result 2022-06-03 16:50:59 root: INFO: instruction count: 28: __C19collect_distributed_array.__m131result_quicksort 2022-06-03 16:50:59 root: INFO: instruction count: 19: __C19collect_distributed_array.__m132quicksort_swap 2022-06-03 16:50:59 root: INFO: instruction count: 193: __C19collect_distributed_array.__m133quicksort_partition 2022-06-03 16:50:59 root: INFO: instruction count: 90: __C19collect_distributed_array.__m134take_by_result 2022-06-03 16:50:59 root: INFO: instruction count: 28: __C19collect_distributed_array.__m135result_quicksort 2022-06-03 16:50:59 root: INFO: instruction count: 19: __C19collect_distributed_array.__m136quicksort_swap 2022-06-03 16:50:59 root: INFO: instruction count: 193: __C19collect_distributed_array.__m137quicksort_partition 2022-06-03 16:50:59 root: INFO: instruction count: 90: __C19collect_distributed_array.__m138take_by_result 2022-06-03 16:50:59 root: INFO: instruction count: 28: __C19collect_distributed_array.__m139result_quicksort 2022-06-03 16:50:59 root: INFO: instruction count: 19: __C19collect_distributed_array.__m140quicksort_swap 2022-06-03 16:50:59 root: INFO: instruction count: 193: __C19collect_distributed_array.__m141quicksort_partition 2022-06-03 16:50:59 root: INFO: instruction count: 584: __C19collect_distributed_array.__m160split_ToArray 2022-06-03 16:50:59 root: INFO: instruction count: 192: __C19collect_distributed_array.__m162split_ToArray 2022-06-03 16:50:59 root: INFO: instruction count: 192: __C19collect_distributed_array.__m174split_ToArray 2022-06-03 16:50:59 root: INFO: instruction count: 12: __C19collect_distributed_array.__m203setup_jab 2022-06-03 16:50:59 root: INFO: instruction count: 36: __C19collect_distributed_array.__m208ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_o_struct_of_ENDEND 2022-06-03 16:50:59 root: INFO: instruction count: 111: __C19collect_distributed_array.__m209ENCODE_SIndexablePointer_TO_r_array_of_o_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 1: __C19collect_distributed_array.__m210ENCODE_SBaseStructPointer_TO_o_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 21: __C19collect_distributed_array.setPartitionIndex 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C19collect_distributed_array.addPartitionRegion 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C19collect_distributed_array.setPool 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C19collect_distributed_array.addHailClassLoader 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C19collect_distributed_array.addFS 2022-06-03 16:50:59 root: INFO: instruction count: 43: __C19collect_distributed_array.addReferenceGenomes 2022-06-03 16:50:59 root: INFO: instruction count: 35: __C19collect_distributed_array.addLiterals 2022-06-03 16:50:59 root: INFO: instruction count: 277: __C19collect_distributed_array.apply_region12_17 2022-06-03 16:50:59 root: INFO: instruction count: 118: __C19collect_distributed_array.__m64ord_compareNonnull_region17_24 2022-06-03 16:50:59 root: INFO: instruction count: 100: __C19collect_distributed_array.__m64ord_compareNonnull_region130_137 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region178_185 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region162_169 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region146_153 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region114_121 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region98_105 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region82_89 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region66_73 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region50_57 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m64ord_compareNonnull_region33_40 2022-06-03 16:50:59 root: INFO: instruction count: 78: __C19collect_distributed_array.__m64ord_compareNonnull_region138_145 2022-06-03 16:50:59 root: INFO: instruction count: 78: __C19collect_distributed_array.__m64ord_compareNonnull_region25_32 2022-06-03 16:50:59 root: INFO: instruction count: 626: __C19collect_distributed_array.__m64ord_compareNonnull_region16_195 2022-06-03 16:50:59 root: INFO: instruction count: 212: __C19collect_distributed_array.__m122take_by_seqop_region3_9 2022-06-03 16:50:59 root: INFO: instruction count: 204: __C19collect_distributed_array.__m122take_by_seqop_region167_173 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m122take_by_seqop_region174_185 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m122take_by_seqop_region89_100 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m122take_by_seqop_region10_21 2022-06-03 16:50:59 root: INFO: instruction count: 310: __C19collect_distributed_array.__m122take_by_seqop_region302_319 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m122take_by_seqop_region228_245 2022-06-03 16:50:59 root: INFO: instruction count: 311: __C19collect_distributed_array.__m122take_by_seqop_region143_160 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m122take_by_seqop_region64_81 2022-06-03 16:50:59 root: INFO: instruction count: 132: __C19collect_distributed_array.__m122take_by_seqop_region161_163 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m122take_by_seqop_region246_252 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m122take_by_seqop_region82_88 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m122take_by_seqop_region260_266 2022-06-03 16:50:59 root: INFO: instruction count: 85: __C19collect_distributed_array.__m122take_by_seqop_region321_323 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region295_301 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region288_294 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region281_287 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region274_280 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region267_273 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region221_227 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region214_220 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region207_213 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region200_206 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region193_199 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region186_192 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region136_142 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region129_135 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region122_128 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region115_121 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region108_114 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region101_107 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region57_63 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region50_56 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m122take_by_seqop_region43_49 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m124take_by_garbage_collect_region85_96 2022-06-03 16:50:59 root: INFO: instruction count: 305: __C19collect_distributed_array.__m124take_by_garbage_collect_region139_156 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m124take_by_garbage_collect_region60_77 2022-06-03 16:50:59 root: INFO: instruction count: 180: __C19collect_distributed_array.__m124take_by_garbage_collect_region4_10 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m124take_by_garbage_collect_region78_84 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m124take_by_garbage_collect_region18_24 2022-06-03 16:50:59 root: INFO: instruction count: 85: __C19collect_distributed_array.__m124take_by_garbage_collect_region157_159 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region132_138 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region125_131 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region118_124 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region111_117 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region104_110 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m124take_by_garbage_collect_region97_103 2022-06-03 16:50:59 root: INFO: instruction count: 589: __C19collect_distributed_array.__m124take_by_garbage_collect_region2_161 2022-06-03 16:50:59 root: INFO: instruction count: 212: __C19collect_distributed_array.__m125take_by_seqop_region3_9 2022-06-03 16:50:59 root: INFO: instruction count: 204: __C19collect_distributed_array.__m125take_by_seqop_region167_173 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m125take_by_seqop_region174_185 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m125take_by_seqop_region89_100 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m125take_by_seqop_region10_21 2022-06-03 16:50:59 root: INFO: instruction count: 310: __C19collect_distributed_array.__m125take_by_seqop_region302_319 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m125take_by_seqop_region228_245 2022-06-03 16:50:59 root: INFO: instruction count: 311: __C19collect_distributed_array.__m125take_by_seqop_region143_160 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m125take_by_seqop_region64_81 2022-06-03 16:50:59 root: INFO: instruction count: 132: __C19collect_distributed_array.__m125take_by_seqop_region161_163 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m125take_by_seqop_region246_252 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m125take_by_seqop_region82_88 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m125take_by_seqop_region260_266 2022-06-03 16:50:59 root: INFO: instruction count: 85: __C19collect_distributed_array.__m125take_by_seqop_region321_323 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region295_301 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region288_294 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region281_287 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region274_280 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region267_273 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region221_227 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region214_220 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region207_213 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region200_206 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region193_199 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region186_192 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region136_142 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region129_135 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region122_128 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region115_121 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region108_114 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region101_107 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region57_63 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region50_56 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m125take_by_seqop_region43_49 2022-06-03 16:50:59 root: INFO: instruction count: 118: __C19collect_distributed_array.__m128ord_compareNonnull_region17_24 2022-06-03 16:50:59 root: INFO: instruction count: 100: __C19collect_distributed_array.__m128ord_compareNonnull_region130_137 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region178_185 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region162_169 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region146_153 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region114_121 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region98_105 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region82_89 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region66_73 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region50_57 2022-06-03 16:50:59 root: INFO: instruction count: 64: __C19collect_distributed_array.__m128ord_compareNonnull_region33_40 2022-06-03 16:50:59 root: INFO: instruction count: 78: __C19collect_distributed_array.__m128ord_compareNonnull_region138_145 2022-06-03 16:50:59 root: INFO: instruction count: 78: __C19collect_distributed_array.__m128ord_compareNonnull_region25_32 2022-06-03 16:50:59 root: INFO: instruction count: 626: __C19collect_distributed_array.__m128ord_compareNonnull_region16_195 2022-06-03 16:50:59 root: INFO: instruction count: 216: __C19collect_distributed_array.__m129take_by_garbage_collect_region85_96 2022-06-03 16:50:59 root: INFO: instruction count: 305: __C19collect_distributed_array.__m129take_by_garbage_collect_region139_156 2022-06-03 16:50:59 root: INFO: instruction count: 331: __C19collect_distributed_array.__m129take_by_garbage_collect_region60_77 2022-06-03 16:50:59 root: INFO: instruction count: 180: __C19collect_distributed_array.__m129take_by_garbage_collect_region4_10 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m129take_by_garbage_collect_region78_84 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m129take_by_garbage_collect_region18_24 2022-06-03 16:50:59 root: INFO: instruction count: 85: __C19collect_distributed_array.__m129take_by_garbage_collect_region157_159 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region132_138 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region125_131 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region118_124 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region111_117 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region104_110 2022-06-03 16:50:59 root: INFO: instruction count: 73: __C19collect_distributed_array.__m129take_by_garbage_collect_region97_103 2022-06-03 16:50:59 root: INFO: instruction count: 589: __C19collect_distributed_array.__m129take_by_garbage_collect_region2_161 2022-06-03 16:50:59 root: INFO: instruction count: 311: __C19collect_distributed_array.__m134take_by_result_region68_85 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m134take_by_result_region12_18 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m134take_by_result_region26_32 2022-06-03 16:50:59 root: INFO: instruction count: 99: __C19collect_distributed_array.__m134take_by_result_region5_11 2022-06-03 16:50:59 root: INFO: instruction count: 570: __C19collect_distributed_array.__m134take_by_result_region4_89 2022-06-03 16:50:59 root: INFO: instruction count: 311: __C19collect_distributed_array.__m138take_by_result_region68_85 2022-06-03 16:50:59 root: INFO: instruction count: 106: __C19collect_distributed_array.__m138take_by_result_region12_18 2022-06-03 16:50:59 root: INFO: instruction count: 94: __C19collect_distributed_array.__m138take_by_result_region26_32 2022-06-03 16:50:59 root: INFO: instruction count: 99: __C19collect_distributed_array.__m138take_by_result_region5_11 2022-06-03 16:50:59 root: INFO: instruction count: 570: __C19collect_distributed_array.__m138take_by_result_region4_89 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C218applySpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C225__m64ord_compareNonnullSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C311__m122take_by_seqopSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C359__m124take_by_garbage_collectSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C386__m125take_by_seqopSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C434__m128ord_compareNonnullSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C520__m129take_by_garbage_collectSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C547__m134take_by_resultSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C558__m138take_by_resultSpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C569__m160split_ToArraySpills. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C67RGContainer_GRCh37. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C67RGContainer_GRCh37. 2022-06-03 16:50:59 root: INFO: encoder cache miss (0 hits, 2 misses, 0.000) 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C682HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C682HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C683FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C683FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C684etypeEncode. 2022-06-03 16:50:59 root: INFO: instruction count: 7: __C684etypeEncode.apply 2022-06-03 16:50:59 root: INFO: instruction count: 57: __C684etypeEncode.__m685ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryENDANDr_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolENDANDr_struct_of_ENDEND 2022-06-03 16:50:59 root: INFO: instruction count: 35: __C684etypeEncode.__m686ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryEND 2022-06-03 16:50:59 root: INFO: instruction count: 13: __C684etypeEncode.__m687ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:50:59 root: INFO: instruction count: 16: __C684etypeEncode.__m688ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:50:59 root: INFO: instruction count: 53: __C684etypeEncode.__m689ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND 2022-06-03 16:50:59 root: INFO: instruction count: 41: __C684etypeEncode.__m690ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND 2022-06-03 16:50:59 root: INFO: instruction count: 1: __C684etypeEncode.__m691ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C684etypeEncode.__m692ENCODE_SBoolean$_TO_r_bool 2022-06-03 16:50:59 MemoryStore: INFO: Block broadcast_2 stored as values in memory (estimated size 17.4 KiB, free 21.7 GiB) 2022-06-03 16:50:59 MemoryStore: INFO: Block broadcast_2_piece0 stored as bytes in memory (estimated size 6.2 KiB, free 21.7 GiB) 2022-06-03 16:50:59 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 6.2 KiB, free: 21.7 GiB) 2022-06-03 16:50:59 SparkContext: INFO: Created broadcast 2 from broadcast at SparkBackend.scala:313 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C1HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C1HailClassLoaderContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C2FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C2FSContainer. 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C3Compiled. 2022-06-03 16:50:59 root: INFO: instruction count: 149: __C3Compiled.apply 2022-06-03 16:50:59 root: INFO: instruction count: 426: __C3Compiled.__m4split_ToArray 2022-06-03 16:50:59 root: INFO: instruction count: 229: __C3Compiled.__m6split_ArraySort 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.setBackend 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C3Compiled.__m575ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDEND 2022-06-03 16:50:59 root: INFO: instruction count: 13: __C3Compiled.__m576ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:50:59 root: INFO: instruction count: 16: __C3Compiled.__m577ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C3Compiled.__m578ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-06-03 16:50:59 root: INFO: instruction count: 1: __C3Compiled.__m579ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.setObjects 2022-06-03 16:50:59 root: INFO: instruction count: 27: __C3Compiled.__m581DECODE_r_struct_of_r_array_of_o_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:50:59 root: INFO: instruction count: 114: __C3Compiled.__m582INPLACE_DECODE_r_array_of_o_struct_of_END_TO_r_array_of_o_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 8: __C3Compiled.__m583INPLACE_DECODE_o_struct_of_END_TO_o_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 12: __C3Compiled.__m602setup_jab 2022-06-03 16:50:59 root: INFO: instruction count: 31: __C3Compiled.__m605dependent_sorting_func 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C3Compiled.__m608ord_lt 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.__m609ord_ltNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 202: __C3Compiled.__m610arraySorter_outer 2022-06-03 16:50:59 root: INFO: instruction count: 66: __C3Compiled.__m611arraySorter_merge 2022-06-03 16:50:59 root: INFO: instruction count: 36: __C3Compiled.__m612arraySorter_splitMerge 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.__m631ord_equivNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 12: __C3Compiled.__m639setup_jab 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C3Compiled.__m648ord_lt 2022-06-03 16:50:59 root: INFO: instruction count: 14: __C3Compiled.__m649ord_ltNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 11: __C3Compiled.__m651ord_equiv 2022-06-03 16:50:59 root: INFO: instruction count: 14: __C3Compiled.__m652ord_equivNonnull 2022-06-03 16:50:59 root: INFO: instruction count: 445: __C3Compiled.__m657split_ToArray 2022-06-03 16:50:59 root: INFO: instruction count: 35: __C3Compiled.__m667arrayref_bounds_check 2022-06-03 16:50:59 root: INFO: instruction count: 9: __C3Compiled.setPartitionIndex 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.addPartitionRegion 2022-06-03 16:50:59 root: INFO: instruction count: 4: __C3Compiled.setPool 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C3Compiled.addHailClassLoader 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C3Compiled.addFS 2022-06-03 16:50:59 root: INFO: instruction count: 89: __C3Compiled.addLiterals 2022-06-03 16:50:59 root: INFO: instruction count: 45: __C3Compiled.__m674DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryENDANDr_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolENDANDr_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:50:59 root: INFO: instruction count: 58: __C3Compiled.__m675INPLACE_DECODE_r_array_of_r_struct_of_r_binaryEND_TO_r_array_of_r_struct_of_r_stringEND 2022-06-03 16:50:59 root: INFO: instruction count: 17: __C3Compiled.__m676INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:50:59 root: INFO: instruction count: 31: __C3Compiled.__m677INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:50:59 root: INFO: instruction count: 58: __C3Compiled.__m678INPLACE_DECODE_r_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND_TO_r_array_of_r_interval_of_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 44: __C3Compiled.__m679INPLACE_DECODE_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 8: __C3Compiled.__m680INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:50:59 root: INFO: instruction count: 10: __C3Compiled.__m681INPLACE_DECODE_r_bool_TO_r_bool 2022-06-03 16:50:59 root: INFO: instruction count: 487: __C3Compiled.__m6split_ArraySort_region4_47 2022-06-03 16:50:59 root: INFO: instruction count: 3: __C693__m6split_ArraySortSpills. 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_3 stored as values in memory (estimated size 64.0 B, free 21.7 GiB) 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_3_piece0 stored as bytes in memory (estimated size 49.0 B, free 21.7 GiB) 2022-06-03 16:51:00 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 49.0 B, free: 21.7 GiB) 2022-06-03 16:51:00 SparkContext: INFO: Created broadcast 3 from broadcast at SparkBackend.scala:313 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_4 stored as values in memory (estimated size 424.7 KiB, free 21.7 GiB) 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_4_piece0 stored as bytes in memory (estimated size 33.4 KiB, free 21.7 GiB) 2022-06-03 16:51:00 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 33.4 KiB, free: 21.7 GiB) 2022-06-03 16:51:00 SparkContext: INFO: Created broadcast 4 from broadcast at SparkBackend.scala:313 2022-06-03 16:51:00 SparkContext: INFO: Starting job: collect at SparkBackend.scala:321 2022-06-03 16:51:00 DAGScheduler: INFO: Got job 0 (collect at SparkBackend.scala:321) with 150 output partitions 2022-06-03 16:51:00 DAGScheduler: INFO: Final stage: ResultStage 0 (collect at SparkBackend.scala:321) 2022-06-03 16:51:00 DAGScheduler: INFO: Parents of final stage: List() 2022-06-03 16:51:00 DAGScheduler: INFO: Missing parents: List() 2022-06-03 16:51:00 DAGScheduler: INFO: Submitting ResultStage 0 (SparkBackendComputeRDD[0] at RDD at SparkBackend.scala:753), which has no missing parents 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_5 stored as values in memory (estimated size 178.4 KiB, free 21.7 GiB) 2022-06-03 16:51:00 MemoryStore: INFO: Block broadcast_5_piece0 stored as bytes in memory (estimated size 77.9 KiB, free 21.7 GiB) 2022-06-03 16:51:00 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 77.9 KiB, free: 21.7 GiB) 2022-06-03 16:51:00 SparkContext: INFO: Created broadcast 5 from broadcast at DAGScheduler.scala:1388 2022-06-03 16:51:00 DAGScheduler: INFO: Submitting 150 missing tasks from ResultStage 0 (SparkBackendComputeRDD[0] at RDD at SparkBackend.scala:753) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 2022-06-03 16:51:00 YarnScheduler: INFO: Adding task set 0.0 with 150 tasks resource profile 0 2022-06-03 16:51:00 FairSchedulableBuilder: INFO: Added task set TaskSet_0.0 tasks to pool default 2022-06-03 16:51:00 TaskSetManager: INFO: Starting task 0.0 in stage 0.0 (TID 0) (kc1-w-9.c.broad-mpg-gnomad.internal, executor 1, partition 0, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:00 TaskSetManager: INFO: Starting task 1.0 in stage 0.0 (TID 1) (kc1-w-9.c.broad-mpg-gnomad.internal, executor 1, partition 1, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:00 TaskSetManager: INFO: Starting task 2.0 in stage 0.0 (TID 2) (kc1-w-9.c.broad-mpg-gnomad.internal, executor 1, partition 2, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:00 TaskSetManager: INFO: Starting task 3.0 in stage 0.0 (TID 3) (kc1-w-9.c.broad-mpg-gnomad.internal, executor 1, partition 3, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:01 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-9.c.broad-mpg-gnomad.internal:43647 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:01 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:01 YarnAllocator: INFO: Driver requested a total number of 2 executor(s) for resource profile id: 0. 2022-06-03 16:51:01 ExecutorAllocationManager: INFO: Requesting 1 new executor because tasks are backlogged (new desired total will be 2 for resource profile id: 0) 2022-06-03 16:51:01 YarnAllocator: INFO: Will request 1 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:01 YarnAllocator: INFO: Submitted 1 unlocalized container requests. 2022-06-03 16:51:01 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000004 on host kc1-w-30.c.broad-mpg-gnomad.internal for executor with ID 3 for ResourceProfile Id 0 2022-06-03 16:51:01 YarnAllocator: INFO: Received 1 containers from YARN, launching executors on 1 of them. 2022-06-03 16:51:02 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-9.c.broad-mpg-gnomad.internal:43647 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:02 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-9.c.broad-mpg-gnomad.internal:43647 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:02 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-9.c.broad-mpg-gnomad.internal:43647 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:02 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:02 YarnAllocator: INFO: Driver requested a total number of 4 executor(s) for resource profile id: 0. 2022-06-03 16:51:02 ExecutorAllocationManager: INFO: Requesting 2 new executors because tasks are backlogged (new desired total will be 4 for resource profile id: 0) 2022-06-03 16:51:02 YarnAllocator: INFO: Will request 2 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:02 YarnAllocator: INFO: Submitted 2 unlocalized container requests. 2022-06-03 16:51:02 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000007 on host kc1-w-22.c.broad-mpg-gnomad.internal for executor with ID 4 for ResourceProfile Id 0 2022-06-03 16:51:02 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000008 on host kc1-w-33.c.broad-mpg-gnomad.internal for executor with ID 5 for ResourceProfile Id 0 2022-06-03 16:51:02 YarnAllocator: INFO: Received 2 containers from YARN, launching executors on 2 of them. 2022-06-03 16:51:03 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:03 YarnAllocator: INFO: Driver requested a total number of 8 executor(s) for resource profile id: 0. 2022-06-03 16:51:03 ExecutorAllocationManager: INFO: Requesting 4 new executors because tasks are backlogged (new desired total will be 8 for resource profile id: 0) 2022-06-03 16:51:03 YarnAllocator: INFO: Will request 4 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:03 YarnAllocator: INFO: Submitted 4 unlocalized container requests. 2022-06-03 16:51:03 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000013 on host kc1-w-40.c.broad-mpg-gnomad.internal for executor with ID 6 for ResourceProfile Id 0 2022-06-03 16:51:03 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000014 on host kc1-w-33.c.broad-mpg-gnomad.internal for executor with ID 7 for ResourceProfile Id 0 2022-06-03 16:51:03 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000015 on host kc1-w-22.c.broad-mpg-gnomad.internal for executor with ID 8 for ResourceProfile Id 0 2022-06-03 16:51:03 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000016 on host kc1-w-17.c.broad-mpg-gnomad.internal for executor with ID 9 for ResourceProfile Id 0 2022-06-03 16:51:03 YarnAllocator: INFO: Received 4 containers from YARN, launching executors on 4 of them. 2022-06-03 16:51:04 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:04 YarnAllocator: INFO: Driver requested a total number of 16 executor(s) for resource profile id: 0. 2022-06-03 16:51:04 ExecutorAllocationManager: INFO: Requesting 8 new executors because tasks are backlogged (new desired total will be 16 for resource profile id: 0) 2022-06-03 16:51:04 YarnAllocator: INFO: Will request 8 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:04 YarnAllocator: INFO: Submitted 8 unlocalized container requests. 2022-06-03 16:51:04 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000025 on host kc1-w-40.c.broad-mpg-gnomad.internal for executor with ID 10 for ResourceProfile Id 0 2022-06-03 16:51:04 YarnAllocator: INFO: Received 1 containers from YARN, launching executors on 1 of them. 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000026 on host kc1-w-14.c.broad-mpg-gnomad.internal for executor with ID 11 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000027 on host kc1-w-36.c.broad-mpg-gnomad.internal for executor with ID 12 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000028 on host kc1-w-24.c.broad-mpg-gnomad.internal for executor with ID 13 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000029 on host kc1-w-18.c.broad-mpg-gnomad.internal for executor with ID 14 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000030 on host kc1-w-28.c.broad-mpg-gnomad.internal for executor with ID 15 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000031 on host kc1-w-4.c.broad-mpg-gnomad.internal for executor with ID 16 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000032 on host kc1-w-7.c.broad-mpg-gnomad.internal for executor with ID 17 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Received 7 containers from YARN, launching executors on 7 of them. 2022-06-03 16:51:05 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:05 YarnAllocator: INFO: Driver requested a total number of 32 executor(s) for resource profile id: 0. 2022-06-03 16:51:05 ExecutorAllocationManager: INFO: Requesting 16 new executors because tasks are backlogged (new desired total will be 32 for resource profile id: 0) 2022-06-03 16:51:05 YarnAllocator: INFO: Will request 16 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:05 YarnAllocator: INFO: Submitted 16 unlocalized container requests. 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000034 on host kc1-w-10.c.broad-mpg-gnomad.internal for executor with ID 18 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000036 on host kc1-w-13.c.broad-mpg-gnomad.internal for executor with ID 19 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000039 on host kc1-w-39.c.broad-mpg-gnomad.internal for executor with ID 20 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000040 on host kc1-w-23.c.broad-mpg-gnomad.internal for executor with ID 21 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000041 on host kc1-w-3.c.broad-mpg-gnomad.internal for executor with ID 22 for ResourceProfile Id 0 2022-06-03 16:51:05 YarnAllocator: INFO: Received 5 containers from YARN, launching executors on 5 of them. 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000054 on host kc1-w-18.c.broad-mpg-gnomad.internal for executor with ID 23 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000055 on host kc1-w-16.c.broad-mpg-gnomad.internal for executor with ID 24 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000056 on host kc1-w-14.c.broad-mpg-gnomad.internal for executor with ID 25 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000057 on host kc1-w-7.c.broad-mpg-gnomad.internal for executor with ID 26 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000058 on host kc1-w-20.c.broad-mpg-gnomad.internal for executor with ID 27 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000059 on host kc1-w-24.c.broad-mpg-gnomad.internal for executor with ID 28 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000060 on host kc1-w-27.c.broad-mpg-gnomad.internal for executor with ID 29 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000061 on host kc1-w-8.c.broad-mpg-gnomad.internal for executor with ID 30 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Received 8 containers from YARN, launching executors on 8 of them. 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000062 on host kc1-w-49.c.broad-mpg-gnomad.internal for executor with ID 31 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000063 on host kc1-w-45.c.broad-mpg-gnomad.internal for executor with ID 32 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000064 on host kc1-w-15.c.broad-mpg-gnomad.internal for executor with ID 33 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Received 11 containers from YARN, launching executors on 3 of them. 2022-06-03 16:51:06 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.39:54804) with ID 4, ResourceProfileId 0 2022-06-03 16:51:06 ExecutorMonitor: INFO: New executor 4 has registered (new total is 3) 2022-06-03 16:51:06 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.53:43236) with ID 5, ResourceProfileId 0 2022-06-03 16:51:06 ExecutorMonitor: INFO: New executor 5 has registered (new total is 4) 2022-06-03 16:51:06 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:06 YarnAllocator: INFO: Driver requested a total number of 38 executor(s) for resource profile id: 0. 2022-06-03 16:51:06 ExecutorAllocationManager: INFO: Requesting 6 new executors because tasks are backlogged (new desired total will be 38 for resource profile id: 0) 2022-06-03 16:51:06 YarnAllocator: INFO: Will request 6 executor container(s) for ResourceProfile Id: 0, each with 4 core(s) and 25292 MB memory. with custom resources: 2022-06-03 16:51:06 YarnAllocator: INFO: Submitted 6 unlocalized container requests. 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000074 on host kc1-w-13.c.broad-mpg-gnomad.internal for executor with ID 34 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000076 on host kc1-w-39.c.broad-mpg-gnomad.internal for executor with ID 35 for ResourceProfile Id 0 2022-06-03 16:51:06 YarnAllocator: INFO: Received 2 containers from YARN, launching executors on 2 of them. 2022-06-03 16:51:06 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-22.c.broad-mpg-gnomad.internal:39851 with 5.1 GiB RAM, BlockManagerId(4, kc1-w-22.c.broad-mpg-gnomad.internal, 39851, None) 2022-06-03 16:51:06 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-33.c.broad-mpg-gnomad.internal:34997 with 5.1 GiB RAM, BlockManagerId(5, kc1-w-33.c.broad-mpg-gnomad.internal, 34997, None) 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 4.0 in stage 0.0 (TID 4) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 4, partition 4, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 5.0 in stage 0.0 (TID 5) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 4, partition 5, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 6.0 in stage 0.0 (TID 6) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 4, partition 6, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 7.0 in stage 0.0 (TID 7) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 4, partition 7, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 8.0 in stage 0.0 (TID 8) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 5, partition 8, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 9.0 in stage 0.0 (TID 9) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 5, partition 9, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 10.0 in stage 0.0 (TID 10) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 5, partition 10, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:06 TaskSetManager: INFO: Starting task 11.0 in stage 0.0 (TID 11) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 5, partition 11, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.53:43240) with ID 7, ResourceProfileId 0 2022-06-03 16:51:07 ExecutorMonitor: INFO: New executor 7 has registered (new total is 5) 2022-06-03 16:51:07 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:39851 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:07 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:34997 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:07 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-33.c.broad-mpg-gnomad.internal:44711 with 5.1 GiB RAM, BlockManagerId(7, kc1-w-33.c.broad-mpg-gnomad.internal, 44711, None) 2022-06-03 16:51:07 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.39:54808) with ID 8, ResourceProfileId 0 2022-06-03 16:51:07 ExecutorMonitor: INFO: New executor 8 has registered (new total is 6) 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 12.0 in stage 0.0 (TID 12) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 7, partition 12, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 13.0 in stage 0.0 (TID 13) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 7, partition 13, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 14.0 in stage 0.0 (TID 14) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 7, partition 14, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 15.0 in stage 0.0 (TID 15) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 7, partition 15, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000082 on host kc1-w-20.c.broad-mpg-gnomad.internal for executor with ID 36 for ResourceProfile Id 0 2022-06-03 16:51:07 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000083 on host kc1-w-27.c.broad-mpg-gnomad.internal for executor with ID 37 for ResourceProfile Id 0 2022-06-03 16:51:07 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000084 on host kc1-w-8.c.broad-mpg-gnomad.internal for executor with ID 38 for ResourceProfile Id 0 2022-06-03 16:51:07 YarnAllocator: INFO: Launching container container_1654272173047_0006_01_000085 on host kc1-w-21.c.broad-mpg-gnomad.internal for executor with ID 39 for ResourceProfile Id 0 2022-06-03 16:51:07 YarnAllocator: INFO: Received 4 containers from YARN, launching executors on 4 of them. 2022-06-03 16:51:07 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-22.c.broad-mpg-gnomad.internal:33423 with 5.1 GiB RAM, BlockManagerId(8, kc1-w-22.c.broad-mpg-gnomad.internal, 33423, None) 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 16.0 in stage 0.0 (TID 16) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 8, partition 16, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 17.0 in stage 0.0 (TID 17) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 8, partition 17, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 18.0 in stage 0.0 (TID 18) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 8, partition 18, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 19.0 in stage 0.0 (TID 19) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 8, partition 19, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:44711 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:07 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.62:34108) with ID 6, ResourceProfileId 0 2022-06-03 16:51:07 ExecutorMonitor: INFO: New executor 6 has registered (new total is 7) 2022-06-03 16:51:07 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-40.c.broad-mpg-gnomad.internal:37615 with 5.1 GiB RAM, BlockManagerId(6, kc1-w-40.c.broad-mpg-gnomad.internal, 37615, None) 2022-06-03 16:51:07 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:33423 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 20.0 in stage 0.0 (TID 20) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 6, partition 20, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 21.0 in stage 0.0 (TID 21) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 6, partition 21, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 22.0 in stage 0.0 (TID 22) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 6, partition 22, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:07 TaskSetManager: INFO: Starting task 23.0 in stage 0.0 (TID 23) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 6, partition 23, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:08 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.62:34112) with ID 10, ResourceProfileId 0 2022-06-03 16:51:08 ExecutorMonitor: INFO: New executor 10 has registered (new total is 8) 2022-06-03 16:51:08 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-40.c.broad-mpg-gnomad.internal:41683 with 5.1 GiB RAM, BlockManagerId(10, kc1-w-40.c.broad-mpg-gnomad.internal, 41683, None) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:37615 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:08 TaskSetManager: INFO: Starting task 24.0 in stage 0.0 (TID 24) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 10, partition 24, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:08 TaskSetManager: INFO: Starting task 25.0 in stage 0.0 (TID 25) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 10, partition 25, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:08 TaskSetManager: INFO: Starting task 26.0 in stage 0.0 (TID 26) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 10, partition 26, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:08 TaskSetManager: INFO: Starting task 27.0 in stage 0.0 (TID 27) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 10, partition 27, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:39851 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:34997 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:41683 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:39851 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:34997 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:39851 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:08 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:34997 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:44711 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:44711 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.29:34502) with ID 25, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 25 has registered (new total is 9) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-33.c.broad-mpg-gnomad.internal:44711 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:33423 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.29:34504) with ID 11, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 11 has registered (new total is 10) 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-14.c.broad-mpg-gnomad.internal:44979 with 5.1 GiB RAM, BlockManagerId(25, kc1-w-14.c.broad-mpg-gnomad.internal, 44979, None) 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.49:58152) with ID 3, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 3 has registered (new total is 11) 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-14.c.broad-mpg-gnomad.internal:46275 with 5.1 GiB RAM, BlockManagerId(11, kc1-w-14.c.broad-mpg-gnomad.internal, 46275, None) 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-30.c.broad-mpg-gnomad.internal:41593 with 5.1 GiB RAM, BlockManagerId(3, kc1-w-30.c.broad-mpg-gnomad.internal, 41593, None) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:33423 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 28.0 in stage 0.0 (TID 28) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 28, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 29.0 in stage 0.0 (TID 29) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 29, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 30.0 in stage 0.0 (TID 30) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 30, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 31.0 in stage 0.0 (TID 31) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 31, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.35:58334) with ID 14, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 14 has registered (new total is 12) 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 32.0 in stage 0.0 (TID 32) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 11, partition 32, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 33.0 in stage 0.0 (TID 33) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 11, partition 33, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 34.0 in stage 0.0 (TID 34) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 11, partition 34, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 35.0 in stage 0.0 (TID 35) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 11, partition 35, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.35:58336) with ID 23, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 23 has registered (new total is 13) 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 36.0 in stage 0.0 (TID 36) (kc1-w-30.c.broad-mpg-gnomad.internal, executor 3, partition 36, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 37.0 in stage 0.0 (TID 37) (kc1-w-30.c.broad-mpg-gnomad.internal, executor 3, partition 37, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 38.0 in stage 0.0 (TID 38) (kc1-w-30.c.broad-mpg-gnomad.internal, executor 3, partition 38, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 39.0 in stage 0.0 (TID 39) (kc1-w-30.c.broad-mpg-gnomad.internal, executor 3, partition 39, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-18.c.broad-mpg-gnomad.internal:36891 with 5.1 GiB RAM, BlockManagerId(14, kc1-w-18.c.broad-mpg-gnomad.internal, 36891, None) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-22.c.broad-mpg-gnomad.internal:33423 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:37615 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-18.c.broad-mpg-gnomad.internal:35929 with 5.1 GiB RAM, BlockManagerId(23, kc1-w-18.c.broad-mpg-gnomad.internal, 35929, None) 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 40.0 in stage 0.0 (TID 40) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 14, partition 40, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 41.0 in stage 0.0 (TID 41) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 14, partition 41, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 42.0 in stage 0.0 (TID 42) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 14, partition 42, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 43.0 in stage 0.0 (TID 43) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 14, partition 43, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:44979 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:46275 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 44.0 in stage 0.0 (TID 44) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 44, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 45.0 in stage 0.0 (TID 45) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 45, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 46.0 in stage 0.0 (TID 46) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 46, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 TaskSetManager: INFO: Starting task 47.0 in stage 0.0 (TID 47) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 47, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-30.c.broad-mpg-gnomad.internal:41593 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.34:56794) with ID 24, ResourceProfileId 0 2022-06-03 16:51:09 ExecutorMonitor: INFO: New executor 24 has registered (new total is 14) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:37615 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-16.c.broad-mpg-gnomad.internal:34507 with 5.1 GiB RAM, BlockManagerId(24, kc1-w-16.c.broad-mpg-gnomad.internal, 34507, None) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:41683 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:37615 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:09 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:36891 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.33:43940) with ID 19, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 19 has registered (new total is 15) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 48.0 in stage 0.0 (TID 48) (kc1-w-16.c.broad-mpg-gnomad.internal, executor 24, partition 48, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 49.0 in stage 0.0 (TID 49) (kc1-w-16.c.broad-mpg-gnomad.internal, executor 24, partition 49, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 50.0 in stage 0.0 (TID 50) (kc1-w-16.c.broad-mpg-gnomad.internal, executor 24, partition 50, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 51.0 in stage 0.0 (TID 51) (kc1-w-16.c.broad-mpg-gnomad.internal, executor 24, partition 51, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:41683 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.40:48022) with ID 21, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 21 has registered (new total is 16) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.22:43030) with ID 30, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 30 has registered (new total is 17) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-13.c.broad-mpg-gnomad.internal:43623 with 5.1 GiB RAM, BlockManagerId(19, kc1-w-13.c.broad-mpg-gnomad.internal, 43623, None) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-23.c.broad-mpg-gnomad.internal:33523 with 5.1 GiB RAM, BlockManagerId(21, kc1-w-23.c.broad-mpg-gnomad.internal, 33523, None) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.33:43942) with ID 34, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 34 has registered (new total is 18) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-40.c.broad-mpg-gnomad.internal:41683 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-8.c.broad-mpg-gnomad.internal:36667 with 5.1 GiB RAM, BlockManagerId(30, kc1-w-8.c.broad-mpg-gnomad.internal, 36667, None) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 52.0 in stage 0.0 (TID 52) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 19, partition 52, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 53.0 in stage 0.0 (TID 53) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 19, partition 53, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 54.0 in stage 0.0 (TID 54) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 19, partition 54, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 55.0 in stage 0.0 (TID 55) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 19, partition 55, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 56.0 in stage 0.0 (TID 56) (kc1-w-23.c.broad-mpg-gnomad.internal, executor 21, partition 56, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 57.0 in stage 0.0 (TID 57) (kc1-w-23.c.broad-mpg-gnomad.internal, executor 21, partition 57, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 58.0 in stage 0.0 (TID 58) (kc1-w-23.c.broad-mpg-gnomad.internal, executor 21, partition 58, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 59.0 in stage 0.0 (TID 59) (kc1-w-23.c.broad-mpg-gnomad.internal, executor 21, partition 59, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.60:56276) with ID 20, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 20 has registered (new total is 19) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-13.c.broad-mpg-gnomad.internal:43607 with 5.1 GiB RAM, BlockManagerId(34, kc1-w-13.c.broad-mpg-gnomad.internal, 43607, None) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.37:60716) with ID 27, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 27 has registered (new total is 20) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.60:56278) with ID 35, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 35 has registered (new total is 21) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 60.0 in stage 0.0 (TID 60) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 30, partition 60, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 61.0 in stage 0.0 (TID 61) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 30, partition 61, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 62.0 in stage 0.0 (TID 62) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 30, partition 62, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 63.0 in stage 0.0 (TID 63) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 30, partition 63, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-39.c.broad-mpg-gnomad.internal:37865 with 5.1 GiB RAM, BlockManagerId(20, kc1-w-39.c.broad-mpg-gnomad.internal, 37865, None) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-20.c.broad-mpg-gnomad.internal:40647 with 5.1 GiB RAM, BlockManagerId(27, kc1-w-20.c.broad-mpg-gnomad.internal, 40647, None) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.45:53112) with ID 29, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 29 has registered (new total is 22) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 64.0 in stage 0.0 (TID 64) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 64, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 65.0 in stage 0.0 (TID 65) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 65, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 66.0 in stage 0.0 (TID 66) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 66, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 67.0 in stage 0.0 (TID 67) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 67, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-16.c.broad-mpg-gnomad.internal:34507 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-39.c.broad-mpg-gnomad.internal:43981 with 5.1 GiB RAM, BlockManagerId(35, kc1-w-39.c.broad-mpg-gnomad.internal, 43981, None) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-27.c.broad-mpg-gnomad.internal:35819 with 5.1 GiB RAM, BlockManagerId(29, kc1-w-27.c.broad-mpg-gnomad.internal, 35819, None) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 68.0 in stage 0.0 (TID 68) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 20, partition 68, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 69.0 in stage 0.0 (TID 69) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 20, partition 69, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 70.0 in stage 0.0 (TID 70) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 20, partition 70, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 71.0 in stage 0.0 (TID 71) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 20, partition 71, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 72.0 in stage 0.0 (TID 72) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 27, partition 72, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 73.0 in stage 0.0 (TID 73) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 27, partition 73, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 74.0 in stage 0.0 (TID 74) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 27, partition 74, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 75.0 in stage 0.0 (TID 75) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 27, partition 75, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-23.c.broad-mpg-gnomad.internal:33523 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43623 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 76.0 in stage 0.0 (TID 76) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 76, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 77.0 in stage 0.0 (TID 77) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 77, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 78.0 in stage 0.0 (TID 78) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 78, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 79.0 in stage 0.0 (TID 79) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 79, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 80.0 in stage 0.0 (TID 80) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 80, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 81.0 in stage 0.0 (TID 81) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 81, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 82.0 in stage 0.0 (TID 82) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 82, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 83.0 in stage 0.0 (TID 83) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 83, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:36667 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:37865 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.22:43034) with ID 38, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 38 has registered (new total is 23) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:40647 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-8.c.broad-mpg-gnomad.internal:44645 with 5.1 GiB RAM, BlockManagerId(38, kc1-w-8.c.broad-mpg-gnomad.internal, 44645, None) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:35819 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.37:60720) with ID 36, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 36 has registered (new total is 24) 2022-06-03 16:51:10 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:43981 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:10 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.45:53116) with ID 37, ResourceProfileId 0 2022-06-03 16:51:10 ExecutorMonitor: INFO: New executor 37 has registered (new total is 25) 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 84.0 in stage 0.0 (TID 84) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 84, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 85.0 in stage 0.0 (TID 85) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 85, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 86.0 in stage 0.0 (TID 86) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 86, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:10 TaskSetManager: INFO: Starting task 87.0 in stage 0.0 (TID 87) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 87, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-20.c.broad-mpg-gnomad.internal:45189 with 5.1 GiB RAM, BlockManagerId(36, kc1-w-20.c.broad-mpg-gnomad.internal, 45189, None) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-30.c.broad-mpg-gnomad.internal:41593 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-27.c.broad-mpg-gnomad.internal:43697 with 5.1 GiB RAM, BlockManagerId(37, kc1-w-27.c.broad-mpg-gnomad.internal, 43697, None) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:44979 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 88.0 in stage 0.0 (TID 88) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 36, partition 88, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 89.0 in stage 0.0 (TID 89) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 36, partition 89, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 90.0 in stage 0.0 (TID 90) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 36, partition 90, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 91.0 in stage 0.0 (TID 91) (kc1-w-20.c.broad-mpg-gnomad.internal, executor 36, partition 91, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:46275 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 92.0 in stage 0.0 (TID 92) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 37, partition 92, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 93.0 in stage 0.0 (TID 93) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 37, partition 93, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 94.0 in stage 0.0 (TID 94) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 37, partition 94, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-30.c.broad-mpg-gnomad.internal:41593 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 TaskSetManager: INFO: Starting task 95.0 in stage 0.0 (TID 95) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 37, partition 95, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-30.c.broad-mpg-gnomad.internal:41593 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:44979 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:46275 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:44979 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-14.c.broad-mpg-gnomad.internal:46275 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:36891 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:45189 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:43697 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-23.c.broad-mpg-gnomad.internal:33523 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:36891 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-16.c.broad-mpg-gnomad.internal:34507 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-23.c.broad-mpg-gnomad.internal:33523 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:36891 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-16.c.broad-mpg-gnomad.internal:34507 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:11 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.36:37456) with ID 9, ResourceProfileId 0 2022-06-03 16:51:11 ExecutorMonitor: INFO: New executor 9 has registered (new total is 26) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-23.c.broad-mpg-gnomad.internal:33523 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-17.c.broad-mpg-gnomad.internal:37281 with 5.1 GiB RAM, BlockManagerId(9, kc1-w-17.c.broad-mpg-gnomad.internal, 37281, None) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:36667 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:40647 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:37865 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 96.0 in stage 0.0 (TID 96) (kc1-w-17.c.broad-mpg-gnomad.internal, executor 9, partition 96, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 97.0 in stage 0.0 (TID 97) (kc1-w-17.c.broad-mpg-gnomad.internal, executor 9, partition 97, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 98.0 in stage 0.0 (TID 98) (kc1-w-17.c.broad-mpg-gnomad.internal, executor 9, partition 98, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 99.0 in stage 0.0 (TID 99) (kc1-w-17.c.broad-mpg-gnomad.internal, executor 9, partition 99, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43623 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:36667 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-16.c.broad-mpg-gnomad.internal:34507 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:40647 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:37865 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43623 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:40647 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:35819 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:43981 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-17.c.broad-mpg-gnomad.internal:37281 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:37865 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43623 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:36667 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:43981 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.48:48384) with ID 15, ResourceProfileId 0 2022-06-03 16:51:12 ExecutorMonitor: INFO: New executor 15 has registered (new total is 27) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:35819 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-39.c.broad-mpg-gnomad.internal:43981 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-28.c.broad-mpg-gnomad.internal:38115 with 5.1 GiB RAM, BlockManagerId(15, kc1-w-28.c.broad-mpg-gnomad.internal, 38115, None) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 100.0 in stage 0.0 (TID 100) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 100, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 101.0 in stage 0.0 (TID 101) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 101, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 102.0 in stage 0.0 (TID 102) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 102, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 TaskSetManager: INFO: Starting task 103.0 in stage 0.0 (TID 103) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 103, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:35819 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:12 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.26:46522) with ID 26, ResourceProfileId 0 2022-06-03 16:51:12 ExecutorMonitor: INFO: New executor 26 has registered (new total is 28) 2022-06-03 16:51:12 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.26:46520) with ID 17, ResourceProfileId 0 2022-06-03 16:51:12 ExecutorMonitor: INFO: New executor 17 has registered (new total is 29) 2022-06-03 16:51:12 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:45189 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-7.c.broad-mpg-gnomad.internal:33731 with 5.1 GiB RAM, BlockManagerId(26, kc1-w-7.c.broad-mpg-gnomad.internal, 33731, None) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-7.c.broad-mpg-gnomad.internal:40761 with 5.1 GiB RAM, BlockManagerId(17, kc1-w-7.c.broad-mpg-gnomad.internal, 40761, None) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.24:60602) with ID 18, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 18 has registered (new total is 30) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.57:39918) with ID 12, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 12 has registered (new total is 31) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.18:45992) with ID 16, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 16 has registered (new total is 32) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:45189 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-10.c.broad-mpg-gnomad.internal:34193 with 5.1 GiB RAM, BlockManagerId(18, kc1-w-10.c.broad-mpg-gnomad.internal, 34193, None) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:43697 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-4.c.broad-mpg-gnomad.internal:37077 with 5.1 GiB RAM, BlockManagerId(16, kc1-w-4.c.broad-mpg-gnomad.internal, 37077, None) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-36.c.broad-mpg-gnomad.internal:40831 with 5.1 GiB RAM, BlockManagerId(12, kc1-w-36.c.broad-mpg-gnomad.internal, 40831, None) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.42:46940) with ID 13, ResourceProfileId 0 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.42:46942) with ID 28, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 13 has registered (new total is 33) 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 28 has registered (new total is 34) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-20.c.broad-mpg-gnomad.internal:45189 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 104.0 in stage 0.0 (TID 104) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 104, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 105.0 in stage 0.0 (TID 105) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 105, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 106.0 in stage 0.0 (TID 106) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 106, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 107.0 in stage 0.0 (TID 107) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 107, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 108.0 in stage 0.0 (TID 108) (kc1-w-4.c.broad-mpg-gnomad.internal, executor 16, partition 108, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 109.0 in stage 0.0 (TID 109) (kc1-w-4.c.broad-mpg-gnomad.internal, executor 16, partition 109, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 110.0 in stage 0.0 (TID 110) (kc1-w-4.c.broad-mpg-gnomad.internal, executor 16, partition 110, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 111.0 in stage 0.0 (TID 111) (kc1-w-4.c.broad-mpg-gnomad.internal, executor 16, partition 111, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 112.0 in stage 0.0 (TID 112) (kc1-w-36.c.broad-mpg-gnomad.internal, executor 12, partition 112, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 113.0 in stage 0.0 (TID 113) (kc1-w-36.c.broad-mpg-gnomad.internal, executor 12, partition 113, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 114.0 in stage 0.0 (TID 114) (kc1-w-36.c.broad-mpg-gnomad.internal, executor 12, partition 114, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 115.0 in stage 0.0 (TID 115) (kc1-w-36.c.broad-mpg-gnomad.internal, executor 12, partition 115, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 116.0 in stage 0.0 (TID 116) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 116, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 117.0 in stage 0.0 (TID 117) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 117, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 118.0 in stage 0.0 (TID 118) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 118, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 119.0 in stage 0.0 (TID 119) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 119, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-24.c.broad-mpg-gnomad.internal:39175 with 5.1 GiB RAM, BlockManagerId(13, kc1-w-24.c.broad-mpg-gnomad.internal, 39175, None) 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 120.0 in stage 0.0 (TID 120) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 120, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 121.0 in stage 0.0 (TID 121) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 121, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 122.0 in stage 0.0 (TID 122) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 122, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 123.0 in stage 0.0 (TID 123) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 123, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-24.c.broad-mpg-gnomad.internal:35463 with 5.1 GiB RAM, BlockManagerId(28, kc1-w-24.c.broad-mpg-gnomad.internal, 35463, None) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:43697 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-4.c.broad-mpg-gnomad.internal:37077 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 124.0 in stage 0.0 (TID 124) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 13, partition 124, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 125.0 in stage 0.0 (TID 125) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 13, partition 125, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 126.0 in stage 0.0 (TID 126) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 13, partition 126, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 127.0 in stage 0.0 (TID 127) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 13, partition 127, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 128.0 in stage 0.0 (TID 128) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 28, partition 128, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 129.0 in stage 0.0 (TID 129) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 28, partition 129, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 130.0 in stage 0.0 (TID 130) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 28, partition 130, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 131.0 in stage 0.0 (TID 131) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 28, partition 131, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-27.c.broad-mpg-gnomad.internal:43697 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-36.c.broad-mpg-gnomad.internal:40831 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.27:55708) with ID 22, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 22 has registered (new total is 35) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:33731 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:40761 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-3.c.broad-mpg-gnomad.internal:40029 with 5.1 GiB RAM, BlockManagerId(22, kc1-w-3.c.broad-mpg-gnomad.internal, 40029, None) 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 132.0 in stage 0.0 (TID 132) (kc1-w-3.c.broad-mpg-gnomad.internal, executor 22, partition 132, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 133.0 in stage 0.0 (TID 133) (kc1-w-3.c.broad-mpg-gnomad.internal, executor 22, partition 133, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 134.0 in stage 0.0 (TID 134) (kc1-w-3.c.broad-mpg-gnomad.internal, executor 22, partition 134, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 TaskSetManager: INFO: Starting task 135.0 in stage 0.0 (TID 135) (kc1-w-3.c.broad-mpg-gnomad.internal, executor 22, partition 135, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:35463 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:39175 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-17.c.broad-mpg-gnomad.internal:37281 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:13 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.72:48018) with ID 31, ResourceProfileId 0 2022-06-03 16:51:13 ExecutorMonitor: INFO: New executor 31 has registered (new total is 36) 2022-06-03 16:51:14 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.30:48672) with ID 33, ResourceProfileId 0 2022-06-03 16:51:14 ExecutorMonitor: INFO: New executor 33 has registered (new total is 37) 2022-06-03 16:51:14 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-49.c.broad-mpg-gnomad.internal:46035 with 5.1 GiB RAM, BlockManagerId(31, kc1-w-49.c.broad-mpg-gnomad.internal, 46035, None) 2022-06-03 16:51:14 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-15.c.broad-mpg-gnomad.internal:44743 with 5.1 GiB RAM, BlockManagerId(33, kc1-w-15.c.broad-mpg-gnomad.internal, 44743, None) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-17.c.broad-mpg-gnomad.internal:37281 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-3.c.broad-mpg-gnomad.internal:40029 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.68:46660) with ID 32, ResourceProfileId 0 2022-06-03 16:51:14 ExecutorMonitor: INFO: New executor 32 has registered (new total is 38) 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 136.0 in stage 0.0 (TID 136) (kc1-w-49.c.broad-mpg-gnomad.internal, executor 31, partition 136, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 137.0 in stage 0.0 (TID 137) (kc1-w-49.c.broad-mpg-gnomad.internal, executor 31, partition 137, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 138.0 in stage 0.0 (TID 138) (kc1-w-49.c.broad-mpg-gnomad.internal, executor 31, partition 138, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 139.0 in stage 0.0 (TID 139) (kc1-w-49.c.broad-mpg-gnomad.internal, executor 31, partition 139, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-17.c.broad-mpg-gnomad.internal:37281 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 140.0 in stage 0.0 (TID 140) (kc1-w-15.c.broad-mpg-gnomad.internal, executor 33, partition 140, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 141.0 in stage 0.0 (TID 141) (kc1-w-15.c.broad-mpg-gnomad.internal, executor 33, partition 141, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 142.0 in stage 0.0 (TID 142) (kc1-w-15.c.broad-mpg-gnomad.internal, executor 33, partition 142, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 143.0 in stage 0.0 (TID 143) (kc1-w-15.c.broad-mpg-gnomad.internal, executor 33, partition 143, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-45.c.broad-mpg-gnomad.internal:41707 with 5.1 GiB RAM, BlockManagerId(32, kc1-w-45.c.broad-mpg-gnomad.internal, 41707, None) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-49.c.broad-mpg-gnomad.internal:46035 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 144.0 in stage 0.0 (TID 144) (kc1-w-45.c.broad-mpg-gnomad.internal, executor 32, partition 144, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 145.0 in stage 0.0 (TID 145) (kc1-w-45.c.broad-mpg-gnomad.internal, executor 32, partition 145, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 146.0 in stage 0.0 (TID 146) (kc1-w-45.c.broad-mpg-gnomad.internal, executor 32, partition 146, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 TaskSetManager: INFO: Starting task 147.0 in stage 0.0 (TID 147) (kc1-w-45.c.broad-mpg-gnomad.internal, executor 32, partition 147, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-15.c.broad-mpg-gnomad.internal:44743 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-4.c.broad-mpg-gnomad.internal:37077 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-45.c.broad-mpg-gnomad.internal:41707 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-36.c.broad-mpg-gnomad.internal:40831 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:14 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-4.c.broad-mpg-gnomad.internal:37077 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-36.c.broad-mpg-gnomad.internal:40831 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-4.c.broad-mpg-gnomad.internal:37077 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:40761 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:33731 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-36.c.broad-mpg-gnomad.internal:40831 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.128.0.41:55080) with ID 39, ResourceProfileId 0 2022-06-03 16:51:15 ExecutorMonitor: INFO: New executor 39 has registered (new total is 39) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:33731 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:40761 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:39175 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerMasterEndpoint: INFO: Registering block manager kc1-w-21.c.broad-mpg-gnomad.internal:37871 with 5.1 GiB RAM, BlockManagerId(39, kc1-w-21.c.broad-mpg-gnomad.internal, 37871, None) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-3.c.broad-mpg-gnomad.internal:40029 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:35463 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-3.c.broad-mpg-gnomad.internal:40029 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:40761 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-7.c.broad-mpg-gnomad.internal:33731 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:39175 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:35463 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 TaskSetManager: INFO: Starting task 148.0 in stage 0.0 (TID 148) (kc1-w-21.c.broad-mpg-gnomad.internal, executor 39, partition 148, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:15 TaskSetManager: INFO: Starting task 149.0 in stage 0.0 (TID 149) (kc1-w-21.c.broad-mpg-gnomad.internal, executor 39, partition 149, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-3.c.broad-mpg-gnomad.internal:40029 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:39175 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-24.c.broad-mpg-gnomad.internal:35463 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-49.c.broad-mpg-gnomad.internal:46035 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-15.c.broad-mpg-gnomad.internal:44743 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on kc1-w-21.c.broad-mpg-gnomad.internal:37871 (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-15.c.broad-mpg-gnomad.internal:44743 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-49.c.broad-mpg-gnomad.internal:46035 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:15 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-45.c.broad-mpg-gnomad.internal:41707 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:16 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-49.c.broad-mpg-gnomad.internal:46035 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:16 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-45.c.broad-mpg-gnomad.internal:41707 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:16 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-15.c.broad-mpg-gnomad.internal:44743 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:16 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-45.c.broad-mpg-gnomad.internal:41707 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:16 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on kc1-w-21.c.broad-mpg-gnomad.internal:37871 (size: 33.4 KiB, free: 5.1 GiB) 2022-06-03 16:51:17 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on kc1-w-21.c.broad-mpg-gnomad.internal:37871 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:51:17 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on kc1-w-21.c.broad-mpg-gnomad.internal:37871 (size: 57.0 B, free: 5.1 GiB) 2022-06-03 16:51:26 TaskSetManager: INFO: Finished task 121.0 in stage 0.0 (TID 121) in 13388 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 26) (1/150) 2022-06-03 16:51:32 TaskSetManager: INFO: Finished task 128.0 in stage 0.0 (TID 128) in 19082 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 28) (2/150) 2022-06-03 16:51:32 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:32 YarnAllocator: INFO: Driver requested a total number of 37 executor(s) for resource profile id: 0. 2022-06-03 16:51:33 TaskSetManager: INFO: Finished task 11.0 in stage 0.0 (TID 11) in 26157 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 5) (3/150) 2022-06-03 16:51:34 TaskSetManager: INFO: Finished task 14.0 in stage 0.0 (TID 14) in 26783 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 7) (4/150) 2022-06-03 16:51:34 TaskSetManager: INFO: Finished task 57.0 in stage 0.0 (TID 57) in 24234 ms on kc1-w-23.c.broad-mpg-gnomad.internal (executor 21) (5/150) 2022-06-03 16:51:34 TaskSetManager: INFO: Finished task 66.0 in stage 0.0 (TID 66) in 24185 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (6/150) 2022-06-03 16:51:34 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:34 YarnAllocator: INFO: Driver requested a total number of 36 executor(s) for resource profile id: 0. 2022-06-03 16:51:35 TaskSetManager: INFO: Finished task 135.0 in stage 0.0 (TID 135) in 21330 ms on kc1-w-3.c.broad-mpg-gnomad.internal (executor 22) (7/150) 2022-06-03 16:51:36 TaskSetManager: INFO: Finished task 1.0 in stage 0.0 (TID 1) in 35538 ms on kc1-w-9.c.broad-mpg-gnomad.internal (executor 1) (8/150) 2022-06-03 16:51:36 TaskSetManager: INFO: Finished task 133.0 in stage 0.0 (TID 133) in 22694 ms on kc1-w-3.c.broad-mpg-gnomad.internal (executor 22) (9/150) 2022-06-03 16:51:36 TaskSetManager: INFO: Finished task 134.0 in stage 0.0 (TID 134) in 22979 ms on kc1-w-3.c.broad-mpg-gnomad.internal (executor 22) (10/150) 2022-06-03 16:51:36 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:36 YarnAllocator: INFO: Driver requested a total number of 35 executor(s) for resource profile id: 0. 2022-06-03 16:51:42 TaskSetManager: INFO: Finished task 38.0 in stage 0.0 (TID 38) in 32971 ms on kc1-w-30.c.broad-mpg-gnomad.internal (executor 3) (11/150) 2022-06-03 16:51:42 TaskSetManager: INFO: Finished task 8.0 in stage 0.0 (TID 8) in 35589 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 5) (12/150) 2022-06-03 16:51:44 TaskSetManager: INFO: Finished task 69.0 in stage 0.0 (TID 69) in 33790 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 20) (13/150) 2022-06-03 16:51:44 TaskSetManager: INFO: Finished task 7.0 in stage 0.0 (TID 7) in 37932 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 4) (14/150) 2022-06-03 16:51:44 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:44 YarnAllocator: INFO: Driver requested a total number of 34 executor(s) for resource profile id: 0. 2022-06-03 16:51:46 TaskSetManager: INFO: Finished task 117.0 in stage 0.0 (TID 117) in 33309 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 17) (15/150) 2022-06-03 16:51:46 TaskSetManager: INFO: Finished task 138.0 in stage 0.0 (TID 138) in 32604 ms on kc1-w-49.c.broad-mpg-gnomad.internal (executor 31) (16/150) 2022-06-03 16:51:50 TaskSetManager: INFO: Finished task 13.0 in stage 0.0 (TID 13) in 42766 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 7) (17/150) 2022-06-03 16:51:50 TaskSetManager: INFO: Finished task 143.0 in stage 0.0 (TID 143) in 35915 ms on kc1-w-15.c.broad-mpg-gnomad.internal (executor 33) (18/150) 2022-06-03 16:51:50 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:50 YarnAllocator: INFO: Driver requested a total number of 33 executor(s) for resource profile id: 0. 2022-06-03 16:51:50 TaskSetManager: INFO: Finished task 37.0 in stage 0.0 (TID 37) in 40960 ms on kc1-w-30.c.broad-mpg-gnomad.internal (executor 3) (19/150) 2022-06-03 16:51:53 TaskSetManager: INFO: Finished task 149.0 in stage 0.0 (TID 149) in 38328 ms on kc1-w-21.c.broad-mpg-gnomad.internal (executor 39) (20/150) 2022-06-03 16:51:54 TaskSetManager: INFO: Finished task 132.0 in stage 0.0 (TID 132) in 40578 ms on kc1-w-3.c.broad-mpg-gnomad.internal (executor 22) (21/150) 2022-06-03 16:51:54 TaskSetManager: INFO: Finished task 139.0 in stage 0.0 (TID 139) in 40183 ms on kc1-w-49.c.broad-mpg-gnomad.internal (executor 31) (22/150) 2022-06-03 16:51:54 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:54 YarnAllocator: INFO: Driver requested a total number of 32 executor(s) for resource profile id: 0. 2022-06-03 16:51:54 TaskSetManager: INFO: Finished task 97.0 in stage 0.0 (TID 97) in 42521 ms on kc1-w-17.c.broad-mpg-gnomad.internal (executor 9) (23/150) 2022-06-03 16:51:55 TaskSetManager: INFO: Finished task 83.0 in stage 0.0 (TID 83) in 44810 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 29) (24/150) 2022-06-03 16:51:56 TaskSetManager: INFO: Finished task 33.0 in stage 0.0 (TID 33) in 46651 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 11) (25/150) 2022-06-03 16:51:56 TaskSetManager: INFO: Finished task 148.0 in stage 0.0 (TID 148) in 40718 ms on kc1-w-21.c.broad-mpg-gnomad.internal (executor 39) (26/150) 2022-06-03 16:51:56 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:56 YarnAllocator: INFO: Driver requested a total number of 31 executor(s) for resource profile id: 0. 2022-06-03 16:51:56 TaskSetManager: INFO: Finished task 106.0 in stage 0.0 (TID 106) in 43165 ms on kc1-w-10.c.broad-mpg-gnomad.internal (executor 18) (27/150) 2022-06-03 16:51:56 TaskSetManager: INFO: Finished task 96.0 in stage 0.0 (TID 96) in 44753 ms on kc1-w-17.c.broad-mpg-gnomad.internal (executor 9) (28/150) 2022-06-03 16:51:57 TaskSetManager: INFO: Finished task 141.0 in stage 0.0 (TID 141) in 43079 ms on kc1-w-15.c.broad-mpg-gnomad.internal (executor 33) (29/150) 2022-06-03 16:51:58 TaskSetManager: INFO: Finished task 17.0 in stage 0.0 (TID 17) in 51186 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 8) (30/150) 2022-06-03 16:51:58 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:51:58 YarnAllocator: INFO: Driver requested a total number of 30 executor(s) for resource profile id: 0. 2022-06-03 16:51:59 TaskSetManager: INFO: Finished task 18.0 in stage 0.0 (TID 18) in 51579 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 8) (31/150) 2022-06-03 16:51:59 TaskSetManager: INFO: Finished task 144.0 in stage 0.0 (TID 144) in 44836 ms on kc1-w-45.c.broad-mpg-gnomad.internal (executor 32) (32/150) 2022-06-03 16:51:59 TaskSetManager: INFO: Finished task 137.0 in stage 0.0 (TID 137) in 45604 ms on kc1-w-49.c.broad-mpg-gnomad.internal (executor 31) (33/150) 2022-06-03 16:52:01 TaskSetManager: INFO: Finished task 15.0 in stage 0.0 (TID 15) in 54290 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 7) (34/150) 2022-06-03 16:52:01 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:01 YarnAllocator: INFO: Driver requested a total number of 29 executor(s) for resource profile id: 0. 2022-06-03 16:52:01 TaskSetManager: INFO: Finished task 146.0 in stage 0.0 (TID 146) in 47174 ms on kc1-w-45.c.broad-mpg-gnomad.internal (executor 32) (35/150) 2022-06-03 16:52:02 TaskSetManager: INFO: Finished task 12.0 in stage 0.0 (TID 12) in 55365 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 7) (36/150) 2022-06-03 16:52:03 TaskSetManager: INFO: Finished task 101.0 in stage 0.0 (TID 101) in 50191 ms on kc1-w-28.c.broad-mpg-gnomad.internal (executor 15) (37/150) 2022-06-03 16:52:03 TaskSetManager: INFO: Finished task 113.0 in stage 0.0 (TID 113) in 50212 ms on kc1-w-36.c.broad-mpg-gnomad.internal (executor 12) (38/150) 2022-06-03 16:52:03 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:03 YarnAllocator: INFO: Driver requested a total number of 28 executor(s) for resource profile id: 0. 2022-06-03 16:52:03 TaskSetManager: INFO: Finished task 115.0 in stage 0.0 (TID 115) in 50384 ms on kc1-w-36.c.broad-mpg-gnomad.internal (executor 12) (39/150) 2022-06-03 16:52:04 TaskSetManager: INFO: Finished task 107.0 in stage 0.0 (TID 107) in 50814 ms on kc1-w-10.c.broad-mpg-gnomad.internal (executor 18) (40/150) 2022-06-03 16:52:04 TaskSetManager: INFO: Finished task 102.0 in stage 0.0 (TID 102) in 51546 ms on kc1-w-28.c.broad-mpg-gnomad.internal (executor 15) (41/150) 2022-06-03 16:52:04 TaskSetManager: INFO: Finished task 103.0 in stage 0.0 (TID 103) in 51904 ms on kc1-w-28.c.broad-mpg-gnomad.internal (executor 15) (42/150) 2022-06-03 16:52:04 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:04 YarnAllocator: INFO: Driver requested a total number of 27 executor(s) for resource profile id: 0. 2022-06-03 16:52:05 TaskSetManager: INFO: Finished task 19.0 in stage 0.0 (TID 19) in 57461 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 8) (43/150) 2022-06-03 16:52:05 TaskSetManager: INFO: Finished task 35.0 in stage 0.0 (TID 35) in 55542 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 11) (44/150) 2022-06-03 16:52:05 TaskSetManager: INFO: Finished task 34.0 in stage 0.0 (TID 34) in 55757 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 11) (45/150) 2022-06-03 16:52:05 TaskSetManager: INFO: Finished task 140.0 in stage 0.0 (TID 140) in 51612 ms on kc1-w-15.c.broad-mpg-gnomad.internal (executor 33) (46/150) 2022-06-03 16:52:06 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:06 YarnAllocator: INFO: Driver requested a total number of 26 executor(s) for resource profile id: 0. 2022-06-03 16:52:06 TaskSetManager: INFO: Finished task 65.0 in stage 0.0 (TID 65) in 56456 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (47/150) 2022-06-03 16:52:07 TaskSetManager: INFO: Finished task 124.0 in stage 0.0 (TID 124) in 53625 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 13) (48/150) 2022-06-03 16:52:08 TaskSetManager: INFO: Finished task 10.0 in stage 0.0 (TID 10) in 61601 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 5) (49/150) 2022-06-03 16:52:08 TaskSetManager: INFO: Finished task 127.0 in stage 0.0 (TID 127) in 55176 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 13) (50/150) 2022-06-03 16:52:08 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:08 YarnAllocator: INFO: Driver requested a total number of 25 executor(s) for resource profile id: 0. 2022-06-03 16:52:08 TaskSetManager: INFO: Finished task 0.0 in stage 0.0 (TID 0) in 68221 ms on kc1-w-9.c.broad-mpg-gnomad.internal (executor 1) (51/150) 2022-06-03 16:52:10 TaskSetManager: INFO: Finished task 56.0 in stage 0.0 (TID 56) in 59762 ms on kc1-w-23.c.broad-mpg-gnomad.internal (executor 21) (52/150) 2022-06-03 16:52:10 TaskSetManager: INFO: Finished task 64.0 in stage 0.0 (TID 64) in 60133 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (53/150) 2022-06-03 16:52:12 TaskSetManager: INFO: Finished task 53.0 in stage 0.0 (TID 53) in 62696 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 19) (54/150) 2022-06-03 16:52:12 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:12 YarnAllocator: INFO: Driver requested a total number of 24 executor(s) for resource profile id: 0. 2022-06-03 16:52:15 TaskSetManager: INFO: Finished task 67.0 in stage 0.0 (TID 67) in 64971 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (55/150) 2022-06-03 16:52:15 TaskSetManager: INFO: Finished task 39.0 in stage 0.0 (TID 39) in 66049 ms on kc1-w-30.c.broad-mpg-gnomad.internal (executor 3) (56/150) 2022-06-03 16:52:16 TaskSetManager: INFO: Finished task 60.0 in stage 0.0 (TID 60) in 66383 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 30) (57/150) 2022-06-03 16:52:17 TaskSetManager: INFO: Finished task 63.0 in stage 0.0 (TID 63) in 66842 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 30) (58/150) 2022-06-03 16:52:17 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:17 YarnAllocator: INFO: Driver requested a total number of 23 executor(s) for resource profile id: 0. 2022-06-03 16:52:17 TaskSetManager: INFO: Finished task 99.0 in stage 0.0 (TID 99) in 65175 ms on kc1-w-17.c.broad-mpg-gnomad.internal (executor 9) (59/150) 2022-06-03 16:52:17 TaskSetManager: INFO: Finished task 122.0 in stage 0.0 (TID 122) in 64602 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 26) (60/150) 2022-06-03 16:52:18 TaskSetManager: INFO: Finished task 95.0 in stage 0.0 (TID 95) in 67014 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 37) (61/150) 2022-06-03 16:52:19 TaskSetManager: INFO: Finished task 116.0 in stage 0.0 (TID 116) in 66237 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 17) (62/150) 2022-06-03 16:52:19 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:19 YarnAllocator: INFO: Driver requested a total number of 22 executor(s) for resource profile id: 0. 2022-06-03 16:52:19 TaskSetManager: INFO: Finished task 98.0 in stage 0.0 (TID 98) in 67554 ms on kc1-w-17.c.broad-mpg-gnomad.internal (executor 9) (63/150) 2022-06-03 16:52:19 TaskSetManager: INFO: Finished task 92.0 in stage 0.0 (TID 92) in 68731 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 37) (64/150) 2022-06-03 16:52:20 TaskSetManager: INFO: Finished task 55.0 in stage 0.0 (TID 55) in 70471 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 19) (65/150) 2022-06-03 16:52:20 TaskSetManager: INFO: Finished task 24.0 in stage 0.0 (TID 24) in 72474 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 10) (66/150) 2022-06-03 16:52:20 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:20 YarnAllocator: INFO: Driver requested a total number of 21 executor(s) for resource profile id: 0. 2022-06-03 16:52:22 TaskSetManager: INFO: Finished task 25.0 in stage 0.0 (TID 25) in 73713 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 10) (67/150) 2022-06-03 16:52:23 TaskSetManager: INFO: Finished task 61.0 in stage 0.0 (TID 61) in 72761 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 30) (68/150) 2022-06-03 16:52:23 TaskSetManager: INFO: Finished task 82.0 in stage 0.0 (TID 82) in 72560 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 29) (69/150) 2022-06-03 16:52:23 TaskSetManager: INFO: Finished task 125.0 in stage 0.0 (TID 125) in 70440 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 13) (70/150) 2022-06-03 16:52:24 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:24 YarnAllocator: INFO: Driver requested a total number of 20 executor(s) for resource profile id: 0. 2022-06-03 16:52:24 TaskSetManager: INFO: Finished task 20.0 in stage 0.0 (TID 20) in 76179 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 6) (71/150) 2022-06-03 16:52:24 TaskSetManager: INFO: Finished task 36.0 in stage 0.0 (TID 36) in 74876 ms on kc1-w-30.c.broad-mpg-gnomad.internal (executor 3) (72/150) 2022-06-03 16:52:24 TaskSetManager: INFO: Finished task 23.0 in stage 0.0 (TID 23) in 76484 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 6) (73/150) 2022-06-03 16:52:25 TaskSetManager: INFO: Finished task 21.0 in stage 0.0 (TID 21) in 77757 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 6) (74/150) 2022-06-03 16:52:25 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:25 YarnAllocator: INFO: Driver requested a total number of 19 executor(s) for resource profile id: 0. 2022-06-03 16:52:26 TaskSetManager: INFO: Finished task 111.0 in stage 0.0 (TID 111) in 73249 ms on kc1-w-4.c.broad-mpg-gnomad.internal (executor 16) (75/150) 2022-06-03 16:52:27 TaskSetManager: INFO: Finished task 59.0 in stage 0.0 (TID 59) in 77108 ms on kc1-w-23.c.broad-mpg-gnomad.internal (executor 21) (76/150) 2022-06-03 16:52:28 TaskSetManager: INFO: Finished task 77.0 in stage 0.0 (TID 77) in 78193 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 35) (77/150) 2022-06-03 16:52:32 TaskSetManager: INFO: Finished task 43.0 in stage 0.0 (TID 43) in 83045 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 14) (78/150) 2022-06-03 16:52:32 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:32 YarnAllocator: INFO: Driver requested a total number of 18 executor(s) for resource profile id: 0. 2022-06-03 16:52:32 TaskSetManager: INFO: Finished task 3.0 in stage 0.0 (TID 3) in 92153 ms on kc1-w-9.c.broad-mpg-gnomad.internal (executor 1) (79/150) 2022-06-03 16:52:34 TaskSetManager: INFO: Finished task 32.0 in stage 0.0 (TID 32) in 85412 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 11) (80/150) 2022-06-03 16:52:36 TaskSetManager: INFO: Finished task 31.0 in stage 0.0 (TID 31) in 86964 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 25) (81/150) 2022-06-03 16:52:36 TaskSetManager: INFO: Finished task 44.0 in stage 0.0 (TID 44) in 87105 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 23) (82/150) 2022-06-03 16:52:36 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:36 YarnAllocator: INFO: Driver requested a total number of 17 executor(s) for resource profile id: 0. 2022-06-03 16:52:37 TaskSetManager: INFO: Finished task 29.0 in stage 0.0 (TID 29) in 88142 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 25) (83/150) 2022-06-03 16:52:39 TaskSetManager: INFO: Finished task 27.0 in stage 0.0 (TID 27) in 90741 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 10) (84/150) 2022-06-03 16:52:39 TaskSetManager: INFO: Finished task 30.0 in stage 0.0 (TID 30) in 89738 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 25) (85/150) 2022-06-03 16:52:41 TaskSetManager: INFO: Finished task 114.0 in stage 0.0 (TID 114) in 87986 ms on kc1-w-36.c.broad-mpg-gnomad.internal (executor 12) (86/150) 2022-06-03 16:52:41 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:41 YarnAllocator: INFO: Driver requested a total number of 16 executor(s) for resource profile id: 0. 2022-06-03 16:52:42 TaskSetManager: INFO: Finished task 54.0 in stage 0.0 (TID 54) in 92041 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 19) (87/150) 2022-06-03 16:52:42 TaskSetManager: INFO: Finished task 123.0 in stage 0.0 (TID 123) in 89216 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 26) (88/150) 2022-06-03 16:52:44 TaskSetManager: INFO: Finished task 26.0 in stage 0.0 (TID 26) in 95901 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 10) (89/150) 2022-06-03 16:52:45 TaskSetManager: INFO: Finished task 42.0 in stage 0.0 (TID 42) in 95788 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 14) (90/150) 2022-06-03 16:52:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:45 YarnAllocator: INFO: Driver requested a total number of 15 executor(s) for resource profile id: 0. 2022-06-03 16:52:45 TaskSetManager: INFO: Finished task 78.0 in stage 0.0 (TID 78) in 95404 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 35) (91/150) 2022-06-03 16:52:46 TaskSetManager: INFO: Finished task 119.0 in stage 0.0 (TID 119) in 92825 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 17) (92/150) 2022-06-03 16:52:47 TaskSetManager: INFO: Finished task 2.0 in stage 0.0 (TID 2) in 106641 ms on kc1-w-9.c.broad-mpg-gnomad.internal (executor 1) (93/150) 2022-06-03 16:52:48 TaskSetManager: INFO: Finished task 40.0 in stage 0.0 (TID 40) in 98357 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 14) (94/150) 2022-06-03 16:52:48 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:48 YarnAllocator: INFO: Driver requested a total number of 14 executor(s) for resource profile id: 0. 2022-06-03 16:52:48 TaskSetManager: INFO: Finished task 16.0 in stage 0.0 (TID 16) in 100570 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 8) (95/150) 2022-06-03 16:52:50 TaskSetManager: INFO: Finished task 145.0 in stage 0.0 (TID 145) in 96331 ms on kc1-w-45.c.broad-mpg-gnomad.internal (executor 32) (96/150) 2022-06-03 16:52:51 TaskSetManager: INFO: Finished task 109.0 in stage 0.0 (TID 109) in 98174 ms on kc1-w-4.c.broad-mpg-gnomad.internal (executor 16) (97/150) 2022-06-03 16:52:51 TaskSetManager: INFO: Finished task 126.0 in stage 0.0 (TID 126) in 98020 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 13) (98/150) 2022-06-03 16:52:51 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:51 YarnAllocator: INFO: Driver requested a total number of 13 executor(s) for resource profile id: 0. 2022-06-03 16:52:53 TaskSetManager: INFO: Finished task 108.0 in stage 0.0 (TID 108) in 100209 ms on kc1-w-4.c.broad-mpg-gnomad.internal (executor 16) (99/150) 2022-06-03 16:52:53 TaskSetManager: INFO: Finished task 104.0 in stage 0.0 (TID 104) in 100632 ms on kc1-w-10.c.broad-mpg-gnomad.internal (executor 18) (100/150) 2022-06-03 16:52:54 TaskSetManager: INFO: Finished task 70.0 in stage 0.0 (TID 70) in 103993 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 20) (101/150) 2022-06-03 16:52:54 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 22 2022-06-03 16:52:54 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 22 2022-06-03 16:52:54 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 22. 2022-06-03 16:52:54 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:54 ExecutorAllocationManager: INFO: Executors 22 removed due to idle timeout. 2022-06-03 16:52:54 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 22. 2022-06-03 16:52:54 DAGScheduler: INFO: Executor lost: 22 (epoch 0) 2022-06-03 16:52:54 BlockManagerMasterEndpoint: INFO: Trying to remove executor 22 from BlockManagerMaster. 2022-06-03 16:52:54 YarnScheduler: INFO: Executor 22 on kc1-w-3.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:52:54 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(22, kc1-w-3.c.broad-mpg-gnomad.internal, 40029, None) 2022-06-03 16:52:54 BlockManagerMaster: INFO: Removed 22 successfully in removeExecutor 2022-06-03 16:52:54 TaskSetManager: INFO: Finished task 87.0 in stage 0.0 (TID 87) in 103944 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 38) (102/150) 2022-06-03 16:52:54 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:54 YarnAllocator: INFO: Driver requested a total number of 12 executor(s) for resource profile id: 0. 2022-06-03 16:52:55 TaskSetManager: INFO: Finished task 110.0 in stage 0.0 (TID 110) in 102094 ms on kc1-w-4.c.broad-mpg-gnomad.internal (executor 16) (103/150) 2022-06-03 16:52:56 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 39 2022-06-03 16:52:56 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 39 2022-06-03 16:52:56 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 39. 2022-06-03 16:52:56 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:56 ExecutorAllocationManager: INFO: Executors 39 removed due to idle timeout. 2022-06-03 16:52:56 TaskSetManager: INFO: Finished task 58.0 in stage 0.0 (TID 58) in 106370 ms on kc1-w-23.c.broad-mpg-gnomad.internal (executor 21) (104/150) 2022-06-03 16:52:57 TaskSetManager: INFO: Finished task 129.0 in stage 0.0 (TID 129) in 103467 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 28) (105/150) 2022-06-03 16:52:57 TaskSetManager: INFO: Finished task 22.0 in stage 0.0 (TID 22) in 109816 ms on kc1-w-40.c.broad-mpg-gnomad.internal (executor 6) (106/150) 2022-06-03 16:52:57 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:52:57 YarnAllocator: INFO: Driver requested a total number of 11 executor(s) for resource profile id: 0. 2022-06-03 16:52:57 TaskSetManager: INFO: Finished task 136.0 in stage 0.0 (TID 136) in 103722 ms on kc1-w-49.c.broad-mpg-gnomad.internal (executor 31) (107/150) 2022-06-03 16:52:58 TaskSetManager: INFO: Finished task 68.0 in stage 0.0 (TID 68) in 107589 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 20) (108/150) 2022-06-03 16:52:58 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 39. 2022-06-03 16:52:58 DAGScheduler: INFO: Executor lost: 39 (epoch 0) 2022-06-03 16:52:58 BlockManagerMasterEndpoint: INFO: Trying to remove executor 39 from BlockManagerMaster. 2022-06-03 16:52:58 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(39, kc1-w-21.c.broad-mpg-gnomad.internal, 37871, None) 2022-06-03 16:52:58 BlockManagerMaster: INFO: Removed 39 successfully in removeExecutor 2022-06-03 16:52:58 YarnScheduler: INFO: Executor 39 on kc1-w-21.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:53:01 TaskSetManager: INFO: Finished task 131.0 in stage 0.0 (TID 131) in 107587 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 28) (109/150) 2022-06-03 16:53:01 TaskSetManager: INFO: Finished task 94.0 in stage 0.0 (TID 94) in 110712 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 37) (110/150) 2022-06-03 16:53:02 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:02 YarnAllocator: INFO: Driver requested a total number of 10 executor(s) for resource profile id: 0. 2022-06-03 16:53:02 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 7 2022-06-03 16:53:02 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 7 2022-06-03 16:53:02 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 7. 2022-06-03 16:53:02 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:02 ExecutorAllocationManager: INFO: Executors 7 removed due to idle timeout. 2022-06-03 16:53:03 TaskSetManager: INFO: Finished task 76.0 in stage 0.0 (TID 76) in 112772 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 35) (111/150) 2022-06-03 16:53:05 TaskSetManager: INFO: Finished task 85.0 in stage 0.0 (TID 85) in 114045 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 38) (112/150) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 52 in stage 0.0 (on kc1-w-13.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(1 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 48 in stage 0.0 (on kc1-w-16.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(2 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 71 in stage 0.0 (on kc1-w-39.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(3 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 73 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(4 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 88 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(5 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 142 in stage 0.0 (on kc1-w-15.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(6 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 5 in stage 0.0 (on kc1-w-22.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(7 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 28 in stage 0.0 (on kc1-w-14.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(8 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 9 in stage 0.0 (on kc1-w-33.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(9 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 74 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(10 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 49 in stage 0.0 (on kc1-w-16.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(11 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 93 in stage 0.0 (on kc1-w-27.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(12 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 147 in stage 0.0 (on kc1-w-45.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(13 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 118 in stage 0.0 (on kc1-w-7.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(14 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 89 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(15 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 120 in stage 0.0 (on kc1-w-7.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(16 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 79 in stage 0.0 (on kc1-w-39.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(17 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 81 in stage 0.0 (on kc1-w-27.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(18 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 75 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(19 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 4 in stage 0.0 (on kc1-w-22.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(20 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 100 in stage 0.0 (on kc1-w-28.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(21 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 50 in stage 0.0 (on kc1-w-16.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(22 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 46 in stage 0.0 (on kc1-w-18.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(23 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 90 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(24 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 84 in stage 0.0 (on kc1-w-8.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(25 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 86 in stage 0.0 (on kc1-w-8.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(26 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 130 in stage 0.0 (on kc1-w-24.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(27 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 80 in stage 0.0 (on kc1-w-27.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(28 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 51 in stage 0.0 (on kc1-w-16.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(29 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 105 in stage 0.0 (on kc1-w-10.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(30 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 47 in stage 0.0 (on kc1-w-18.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(31 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 72 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(32 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 45 in stage 0.0 (on kc1-w-18.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(33 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 112 in stage 0.0 (on kc1-w-36.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(34 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 91 in stage 0.0 (on kc1-w-20.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(35 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 62 in stage 0.0 (on kc1-w-8.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(36 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 41 in stage 0.0 (on kc1-w-18.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(37 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Marking task 6 in stage 0.0 (on kc1-w-22.c.broad-mpg-gnomad.internal) as speculatable because it ran more than 98418 ms(38 speculatable tasks in this taskset now) 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 6.1 in stage 0.0 (TID 150) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 6, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 41.1 in stage 0.0 (TID 151) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 41, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 62.1 in stage 0.0 (TID 152) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 62, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 91.1 in stage 0.0 (TID 153) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 91, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 112.1 in stage 0.0 (TID 154) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 112, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 45.1 in stage 0.0 (TID 155) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 45, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 72.1 in stage 0.0 (TID 156) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 11, partition 72, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 47.1 in stage 0.0 (TID 157) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 37, partition 47, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 105.1 in stage 0.0 (TID 158) (kc1-w-23.c.broad-mpg-gnomad.internal, executor 21, partition 105, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 51.1 in stage 0.0 (TID 159) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 30, partition 51, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 80.1 in stage 0.0 (TID 160) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 6, partition 80, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 130.1 in stage 0.0 (TID 161) (kc1-w-45.c.broad-mpg-gnomad.internal, executor 32, partition 130, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 86.1 in stage 0.0 (TID 162) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 8, partition 86, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 84.1 in stage 0.0 (TID 163) (kc1-w-9.c.broad-mpg-gnomad.internal, executor 1, partition 84, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 90.1 in stage 0.0 (TID 164) (kc1-w-17.c.broad-mpg-gnomad.internal, executor 9, partition 90, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 46.1 in stage 0.0 (TID 165) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 46, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 50.1 in stage 0.0 (TID 166) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 50, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 100.1 in stage 0.0 (TID 167) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 19, partition 100, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 4.1 in stage 0.0 (TID 168) (kc1-w-4.c.broad-mpg-gnomad.internal, executor 16, partition 4, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 75.1 in stage 0.0 (TID 169) (kc1-w-49.c.broad-mpg-gnomad.internal, executor 31, partition 75, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 81.1 in stage 0.0 (TID 170) (kc1-w-22.c.broad-mpg-gnomad.internal, executor 4, partition 81, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 79.1 in stage 0.0 (TID 171) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 14, partition 79, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 120.1 in stage 0.0 (TID 172) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 13, partition 120, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 89.1 in stage 0.0 (TID 173) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 89, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 118.1 in stage 0.0 (TID 174) (kc1-w-30.c.broad-mpg-gnomad.internal, executor 3, partition 118, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 147.1 in stage 0.0 (TID 175) (kc1-w-33.c.broad-mpg-gnomad.internal, executor 5, partition 147, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 93.1 in stage 0.0 (TID 176) (kc1-w-15.c.broad-mpg-gnomad.internal, executor 33, partition 93, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 49.1 in stage 0.0 (TID 177) (kc1-w-36.c.broad-mpg-gnomad.internal, executor 12, partition 49, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 74.1 in stage 0.0 (TID 178) (kc1-w-24.c.broad-mpg-gnomad.internal, executor 28, partition 74, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 9.1 in stage 0.0 (TID 179) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 20, partition 9, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 28.1 in stage 0.0 (TID 180) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 28, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 5.1 in stage 0.0 (TID 181) (kc1-w-40.c.broad-mpg-gnomad.internal, executor 10, partition 5, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 142.1 in stage 0.0 (TID 182) (kc1-w-39.c.broad-mpg-gnomad.internal, executor 35, partition 142, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 88.1 in stage 0.0 (TID 183) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 26, partition 88, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 73.1 in stage 0.0 (TID 184) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 73, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 71.1 in stage 0.0 (TID 185) (kc1-w-27.c.broad-mpg-gnomad.internal, executor 29, partition 71, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 48.1 in stage 0.0 (TID 186) (kc1-w-14.c.broad-mpg-gnomad.internal, executor 25, partition 48, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 TaskSetManager: INFO: Starting task 52.1 in stage 0.0 (TID 187) (kc1-w-7.c.broad-mpg-gnomad.internal, executor 17, partition 52, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:53:05 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 7. 2022-06-03 16:53:05 DAGScheduler: INFO: Executor lost: 7 (epoch 0) 2022-06-03 16:53:05 BlockManagerMasterEndpoint: INFO: Trying to remove executor 7 from BlockManagerMaster. 2022-06-03 16:53:05 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(7, kc1-w-33.c.broad-mpg-gnomad.internal, 44711, None) 2022-06-03 16:53:05 BlockManagerMaster: INFO: Removed 7 successfully in removeExecutor 2022-06-03 16:53:05 YarnScheduler: INFO: Executor 7 on kc1-w-33.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:53:05 TaskSetManager: INFO: Killing attempt 1 for task 89.1 in stage 0.0 (TID 173) on kc1-w-18.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:53:05 TaskSetManager: INFO: Finished task 89.0 in stage 0.0 (TID 89) in 114667 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 36) (113/150) 2022-06-03 16:53:06 TaskSetManager: WARN: Lost task 89.1 in stage 0.0 (TID 173) (kc1-w-18.c.broad-mpg-gnomad.internal executor 23): TaskKilled (another attempt succeeded) 2022-06-03 16:53:06 TaskSetManager: INFO: task 89.1 in stage 0.0 (TID 173) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:07 TaskSetManager: INFO: Killing attempt 1 for task 118.1 in stage 0.0 (TID 174) on kc1-w-30.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-7.c.broad-mpg-gnomad.internal 2022-06-03 16:53:07 TaskSetManager: INFO: Finished task 118.0 in stage 0.0 (TID 118) in 114608 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 17) (114/150) 2022-06-03 16:53:07 TaskSetManager: WARN: Lost task 118.1 in stage 0.0 (TID 174) (kc1-w-30.c.broad-mpg-gnomad.internal executor 3): TaskKilled (another attempt succeeded) 2022-06-03 16:53:07 TaskSetManager: INFO: task 118.1 in stage 0.0 (TID 174) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:13 TaskSetManager: INFO: Killing attempt 1 for task 5.1 in stage 0.0 (TID 181) on kc1-w-40.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-22.c.broad-mpg-gnomad.internal 2022-06-03 16:53:13 TaskSetManager: INFO: Finished task 5.0 in stage 0.0 (TID 5) in 126262 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 4) (115/150) 2022-06-03 16:53:13 TaskSetManager: WARN: Lost task 5.1 in stage 0.0 (TID 181) (kc1-w-40.c.broad-mpg-gnomad.internal executor 10): TaskKilled (another attempt succeeded) 2022-06-03 16:53:13 TaskSetManager: INFO: task 5.1 in stage 0.0 (TID 181) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:14 TaskSetManager: INFO: Killing attempt 1 for task 120.1 in stage 0.0 (TID 172) on kc1-w-24.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-7.c.broad-mpg-gnomad.internal 2022-06-03 16:53:14 TaskSetManager: INFO: Finished task 120.0 in stage 0.0 (TID 120) in 121049 ms on kc1-w-7.c.broad-mpg-gnomad.internal (executor 26) (116/150) 2022-06-03 16:53:14 TaskSetManager: WARN: Lost task 120.1 in stage 0.0 (TID 172) (kc1-w-24.c.broad-mpg-gnomad.internal executor 13): TaskKilled (another attempt succeeded) 2022-06-03 16:53:14 TaskSetManager: INFO: task 120.1 in stage 0.0 (TID 172) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:15 TaskSetManager: INFO: Killing attempt 1 for task 71.1 in stage 0.0 (TID 185) on kc1-w-27.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-39.c.broad-mpg-gnomad.internal 2022-06-03 16:53:15 TaskSetManager: INFO: Finished task 71.0 in stage 0.0 (TID 71) in 125195 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 20) (117/150) 2022-06-03 16:53:15 TaskSetManager: WARN: Lost task 71.1 in stage 0.0 (TID 185) (kc1-w-27.c.broad-mpg-gnomad.internal executor 29): TaskKilled (another attempt succeeded) 2022-06-03 16:53:15 TaskSetManager: INFO: task 71.1 in stage 0.0 (TID 185) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:15 TaskSetManager: INFO: Killing attempt 1 for task 90.1 in stage 0.0 (TID 164) on kc1-w-17.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:53:15 TaskSetManager: INFO: Finished task 90.0 in stage 0.0 (TID 90) in 124753 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 36) (118/150) 2022-06-03 16:53:15 TaskSetManager: WARN: Lost task 90.1 in stage 0.0 (TID 164) (kc1-w-17.c.broad-mpg-gnomad.internal executor 9): TaskKilled (another attempt succeeded) 2022-06-03 16:53:15 TaskSetManager: INFO: task 90.1 in stage 0.0 (TID 164) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:18 TaskSetManager: INFO: Killing attempt 1 for task 47.1 in stage 0.0 (TID 157) on kc1-w-27.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-18.c.broad-mpg-gnomad.internal 2022-06-03 16:53:18 TaskSetManager: INFO: Finished task 47.0 in stage 0.0 (TID 47) in 128276 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 23) (119/150) 2022-06-03 16:53:18 TaskSetManager: WARN: Lost task 47.1 in stage 0.0 (TID 157) (kc1-w-27.c.broad-mpg-gnomad.internal executor 37): TaskKilled (another attempt succeeded) 2022-06-03 16:53:18 TaskSetManager: INFO: task 47.1 in stage 0.0 (TID 157) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:22 TaskSetManager: INFO: Killing attempt 1 for task 79.1 in stage 0.0 (TID 171) on kc1-w-18.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-39.c.broad-mpg-gnomad.internal 2022-06-03 16:53:22 TaskSetManager: INFO: Finished task 79.0 in stage 0.0 (TID 79) in 132027 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 35) (120/150) 2022-06-03 16:53:22 TaskSetManager: INFO: Killing attempt 1 for task 4.1 in stage 0.0 (TID 168) on kc1-w-4.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-22.c.broad-mpg-gnomad.internal 2022-06-03 16:53:22 TaskSetManager: INFO: Finished task 4.0 in stage 0.0 (TID 4) in 135738 ms on kc1-w-22.c.broad-mpg-gnomad.internal (executor 4) (121/150) 2022-06-03 16:53:22 TaskSetManager: WARN: Lost task 79.1 in stage 0.0 (TID 171) (kc1-w-18.c.broad-mpg-gnomad.internal executor 14): TaskKilled (another attempt succeeded) 2022-06-03 16:53:22 TaskSetManager: INFO: task 79.1 in stage 0.0 (TID 171) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:23 TaskSetManager: WARN: Lost task 4.1 in stage 0.0 (TID 168) (kc1-w-4.c.broad-mpg-gnomad.internal executor 16): TaskKilled (another attempt succeeded) 2022-06-03 16:53:23 TaskSetManager: INFO: task 4.1 in stage 0.0 (TID 168) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:24 TaskSetManager: INFO: Killing attempt 1 for task 45.1 in stage 0.0 (TID 155) on kc1-w-7.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-18.c.broad-mpg-gnomad.internal 2022-06-03 16:53:24 TaskSetManager: INFO: Finished task 45.0 in stage 0.0 (TID 45) in 134788 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 23) (122/150) 2022-06-03 16:53:24 TaskSetManager: WARN: Lost task 45.1 in stage 0.0 (TID 155) (kc1-w-7.c.broad-mpg-gnomad.internal executor 17): TaskKilled (another attempt succeeded) 2022-06-03 16:53:24 TaskSetManager: INFO: task 45.1 in stage 0.0 (TID 155) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:25 TaskSetManager: INFO: Killing attempt 1 for task 147.1 in stage 0.0 (TID 175) on kc1-w-33.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-45.c.broad-mpg-gnomad.internal 2022-06-03 16:53:25 TaskSetManager: INFO: Finished task 147.0 in stage 0.0 (TID 147) in 130756 ms on kc1-w-45.c.broad-mpg-gnomad.internal (executor 32) (123/150) 2022-06-03 16:53:25 TaskSetManager: WARN: Lost task 147.1 in stage 0.0 (TID 175) (kc1-w-33.c.broad-mpg-gnomad.internal executor 5): TaskKilled (another attempt succeeded) 2022-06-03 16:53:25 TaskSetManager: INFO: task 147.1 in stage 0.0 (TID 175) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:26 TaskSetManager: INFO: Killing attempt 1 for task 93.1 in stage 0.0 (TID 176) on kc1-w-15.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-27.c.broad-mpg-gnomad.internal 2022-06-03 16:53:26 TaskSetManager: INFO: Finished task 93.0 in stage 0.0 (TID 93) in 135181 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 37) (124/150) 2022-06-03 16:53:26 TaskSetManager: WARN: Lost task 93.1 in stage 0.0 (TID 176) (kc1-w-15.c.broad-mpg-gnomad.internal executor 33): TaskKilled (another attempt succeeded) 2022-06-03 16:53:26 TaskSetManager: INFO: task 93.1 in stage 0.0 (TID 176) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:30 TaskSetManager: INFO: Killing attempt 1 for task 73.1 in stage 0.0 (TID 184) on kc1-w-13.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:53:30 TaskSetManager: INFO: Finished task 73.0 in stage 0.0 (TID 73) in 139932 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 27) (125/150) 2022-06-03 16:53:30 TaskSetManager: WARN: Lost task 73.1 in stage 0.0 (TID 184) (kc1-w-13.c.broad-mpg-gnomad.internal executor 34): TaskKilled (another attempt succeeded) 2022-06-03 16:53:30 TaskSetManager: INFO: task 73.1 in stage 0.0 (TID 184) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:30 TaskSetManager: INFO: Killing attempt 1 for task 74.1 in stage 0.0 (TID 178) on kc1-w-24.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:53:30 TaskSetManager: INFO: Finished task 74.0 in stage 0.0 (TID 74) in 140074 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 27) (126/150) 2022-06-03 16:53:30 TaskSetManager: WARN: Lost task 74.1 in stage 0.0 (TID 178) (kc1-w-24.c.broad-mpg-gnomad.internal executor 28): TaskKilled (another attempt succeeded) 2022-06-03 16:53:30 TaskSetManager: INFO: task 74.1 in stage 0.0 (TID 178) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:32 TaskSetManager: INFO: Killing attempt 1 for task 142.1 in stage 0.0 (TID 182) on kc1-w-39.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-15.c.broad-mpg-gnomad.internal 2022-06-03 16:53:32 TaskSetManager: INFO: Finished task 142.0 in stage 0.0 (TID 142) in 138095 ms on kc1-w-15.c.broad-mpg-gnomad.internal (executor 33) (127/150) 2022-06-03 16:53:32 TaskSetManager: WARN: Lost task 142.1 in stage 0.0 (TID 182) (kc1-w-39.c.broad-mpg-gnomad.internal executor 35): TaskKilled (another attempt succeeded) 2022-06-03 16:53:32 TaskSetManager: INFO: task 142.1 in stage 0.0 (TID 182) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:35 TaskSetManager: INFO: Killing attempt 1 for task 72.1 in stage 0.0 (TID 156) on kc1-w-14.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:53:35 TaskSetManager: INFO: Finished task 72.0 in stage 0.0 (TID 72) in 145027 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 27) (128/150) 2022-06-03 16:53:35 TaskSetManager: WARN: Lost task 72.1 in stage 0.0 (TID 156) (kc1-w-14.c.broad-mpg-gnomad.internal executor 11): TaskKilled (another attempt succeeded) 2022-06-03 16:53:35 TaskSetManager: INFO: task 72.1 in stage 0.0 (TID 156) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:36 TaskSetManager: INFO: Killing attempt 1 for task 130.1 in stage 0.0 (TID 161) on kc1-w-45.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-24.c.broad-mpg-gnomad.internal 2022-06-03 16:53:36 TaskSetManager: INFO: Finished task 130.0 in stage 0.0 (TID 130) in 143241 ms on kc1-w-24.c.broad-mpg-gnomad.internal (executor 28) (129/150) 2022-06-03 16:53:36 TaskSetManager: WARN: Lost task 130.1 in stage 0.0 (TID 161) (kc1-w-45.c.broad-mpg-gnomad.internal executor 32): TaskKilled (another attempt succeeded) 2022-06-03 16:53:36 TaskSetManager: INFO: task 130.1 in stage 0.0 (TID 161) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:39 TaskSetManager: INFO: Killing attempt 1 for task 80.1 in stage 0.0 (TID 160) on kc1-w-40.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-27.c.broad-mpg-gnomad.internal 2022-06-03 16:53:39 TaskSetManager: INFO: Finished task 80.0 in stage 0.0 (TID 80) in 149104 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 29) (130/150) 2022-06-03 16:53:39 TaskSetManager: WARN: Lost task 80.1 in stage 0.0 (TID 160) (kc1-w-40.c.broad-mpg-gnomad.internal executor 6): TaskKilled (another attempt succeeded) 2022-06-03 16:53:39 TaskSetManager: INFO: task 80.1 in stage 0.0 (TID 160) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:42 TaskSetManager: INFO: Killing attempt 1 for task 81.1 in stage 0.0 (TID 170) on kc1-w-22.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-27.c.broad-mpg-gnomad.internal 2022-06-03 16:53:42 TaskSetManager: INFO: Finished task 81.0 in stage 0.0 (TID 81) in 152020 ms on kc1-w-27.c.broad-mpg-gnomad.internal (executor 29) (131/150) 2022-06-03 16:53:42 TaskSetManager: WARN: Lost task 81.1 in stage 0.0 (TID 170) (kc1-w-22.c.broad-mpg-gnomad.internal executor 4): TaskKilled (another attempt succeeded) 2022-06-03 16:53:42 TaskSetManager: INFO: task 81.1 in stage 0.0 (TID 170) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:42 TaskSetManager: INFO: Killing attempt 1 for task 100.1 in stage 0.0 (TID 167) on kc1-w-13.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-28.c.broad-mpg-gnomad.internal 2022-06-03 16:53:42 TaskSetManager: INFO: Finished task 100.0 in stage 0.0 (TID 100) in 150055 ms on kc1-w-28.c.broad-mpg-gnomad.internal (executor 15) (132/150) 2022-06-03 16:53:42 TaskSetManager: WARN: Lost task 100.1 in stage 0.0 (TID 167) (kc1-w-13.c.broad-mpg-gnomad.internal executor 19): TaskKilled (another attempt succeeded) 2022-06-03 16:53:42 TaskSetManager: INFO: task 100.1 in stage 0.0 (TID 167) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:43 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:43 YarnAllocator: INFO: Driver requested a total number of 9 executor(s) for resource profile id: 0. 2022-06-03 16:53:43 TaskSetManager: INFO: Killing attempt 1 for task 28.1 in stage 0.0 (TID 180) on kc1-w-8.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-14.c.broad-mpg-gnomad.internal 2022-06-03 16:53:43 TaskSetManager: INFO: Finished task 28.0 in stage 0.0 (TID 28) in 153836 ms on kc1-w-14.c.broad-mpg-gnomad.internal (executor 25) (133/150) 2022-06-03 16:53:43 TaskSetManager: WARN: Lost task 28.1 in stage 0.0 (TID 180) (kc1-w-8.c.broad-mpg-gnomad.internal executor 38): TaskKilled (another attempt succeeded) 2022-06-03 16:53:43 TaskSetManager: INFO: task 28.1 in stage 0.0 (TID 180) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:44 TaskSetManager: INFO: Killing attempt 1 for task 105.1 in stage 0.0 (TID 158) on kc1-w-23.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-10.c.broad-mpg-gnomad.internal 2022-06-03 16:53:44 TaskSetManager: INFO: Finished task 105.0 in stage 0.0 (TID 105) in 151347 ms on kc1-w-10.c.broad-mpg-gnomad.internal (executor 18) (134/150) 2022-06-03 16:53:45 TaskSetManager: WARN: Lost task 105.1 in stage 0.0 (TID 158) (kc1-w-23.c.broad-mpg-gnomad.internal executor 21): TaskKilled (another attempt succeeded) 2022-06-03 16:53:45 TaskSetManager: INFO: task 105.1 in stage 0.0 (TID 158) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:45 YarnAllocator: INFO: Driver requested a total number of 8 executor(s) for resource profile id: 0. 2022-06-03 16:53:45 TaskSetManager: INFO: Killing attempt 1 for task 9.1 in stage 0.0 (TID 179) on kc1-w-39.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-33.c.broad-mpg-gnomad.internal 2022-06-03 16:53:45 TaskSetManager: INFO: Finished task 9.0 in stage 0.0 (TID 9) in 158994 ms on kc1-w-33.c.broad-mpg-gnomad.internal (executor 5) (135/150) 2022-06-03 16:53:46 TaskSetManager: WARN: Lost task 9.1 in stage 0.0 (TID 179) (kc1-w-39.c.broad-mpg-gnomad.internal executor 20): TaskKilled (another attempt succeeded) 2022-06-03 16:53:46 TaskSetManager: INFO: task 9.1 in stage 0.0 (TID 179) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:53 TaskSetManager: INFO: Killing attempt 1 for task 51.1 in stage 0.0 (TID 159) on kc1-w-8.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-16.c.broad-mpg-gnomad.internal 2022-06-03 16:53:53 TaskSetManager: INFO: Finished task 51.0 in stage 0.0 (TID 51) in 163211 ms on kc1-w-16.c.broad-mpg-gnomad.internal (executor 24) (136/150) 2022-06-03 16:53:53 TaskSetManager: WARN: Lost task 51.1 in stage 0.0 (TID 159) (kc1-w-8.c.broad-mpg-gnomad.internal executor 30): TaskKilled (another attempt succeeded) 2022-06-03 16:53:53 TaskSetManager: INFO: task 51.1 in stage 0.0 (TID 159) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:53 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:53 YarnAllocator: INFO: Driver requested a total number of 7 executor(s) for resource profile id: 0. 2022-06-03 16:53:55 TaskSetManager: INFO: Killing attempt 1 for task 41.1 in stage 0.0 (TID 151) on kc1-w-7.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-18.c.broad-mpg-gnomad.internal 2022-06-03 16:53:55 TaskSetManager: INFO: Finished task 41.0 in stage 0.0 (TID 41) in 165955 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 14) (137/150) 2022-06-03 16:53:55 TaskSetManager: WARN: Lost task 41.1 in stage 0.0 (TID 151) (kc1-w-7.c.broad-mpg-gnomad.internal executor 26): TaskKilled (another attempt succeeded) 2022-06-03 16:53:55 TaskSetManager: INFO: task 41.1 in stage 0.0 (TID 151) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:55 TaskSetManager: INFO: Killing attempt 1 for task 52.1 in stage 0.0 (TID 187) on kc1-w-7.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-13.c.broad-mpg-gnomad.internal 2022-06-03 16:53:55 TaskSetManager: INFO: Finished task 52.0 in stage 0.0 (TID 52) in 165525 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 19) (138/150) 2022-06-03 16:53:55 TaskSetManager: WARN: Lost task 52.1 in stage 0.0 (TID 187) (kc1-w-7.c.broad-mpg-gnomad.internal executor 17): TaskKilled (another attempt succeeded) 2022-06-03 16:53:55 TaskSetManager: INFO: task 52.1 in stage 0.0 (TID 187) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:53:55 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:53:55 YarnAllocator: INFO: Driver requested a total number of 6 executor(s) for resource profile id: 0. 2022-06-03 16:54:01 TaskSetManager: INFO: Killing attempt 1 for task 88.1 in stage 0.0 (TID 183) on kc1-w-7.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:54:01 TaskSetManager: INFO: Finished task 88.0 in stage 0.0 (TID 88) in 170405 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 36) (139/150) 2022-06-03 16:54:01 TaskSetManager: WARN: Lost task 88.1 in stage 0.0 (TID 183) (kc1-w-7.c.broad-mpg-gnomad.internal executor 26): TaskKilled (another attempt succeeded) 2022-06-03 16:54:01 TaskSetManager: INFO: task 88.1 in stage 0.0 (TID 183) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:01 TaskSetManager: INFO: Killing attempt 1 for task 91.1 in stage 0.0 (TID 153) on kc1-w-27.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:54:01 TaskSetManager: INFO: Finished task 91.0 in stage 0.0 (TID 91) in 170726 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 36) (140/150) 2022-06-03 16:54:01 TaskSetManager: WARN: Lost task 91.1 in stage 0.0 (TID 153) (kc1-w-27.c.broad-mpg-gnomad.internal executor 29): TaskKilled (another attempt succeeded) 2022-06-03 16:54:01 TaskSetManager: INFO: task 91.1 in stage 0.0 (TID 153) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:01 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:01 YarnAllocator: INFO: Driver requested a total number of 5 executor(s) for resource profile id: 0. 2022-06-03 16:54:02 TaskSetManager: INFO: Killing attempt 1 for task 84.1 in stage 0.0 (TID 163) on kc1-w-9.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-8.c.broad-mpg-gnomad.internal 2022-06-03 16:54:02 TaskSetManager: INFO: Finished task 84.0 in stage 0.0 (TID 84) in 171982 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 38) (141/150) 2022-06-03 16:54:03 TaskSetManager: WARN: Lost task 84.1 in stage 0.0 (TID 163) (kc1-w-9.c.broad-mpg-gnomad.internal executor 1): TaskKilled (another attempt succeeded) 2022-06-03 16:54:03 TaskSetManager: INFO: task 84.1 in stage 0.0 (TID 163) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:08 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 3 2022-06-03 16:54:08 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 3 2022-06-03 16:54:08 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 3. 2022-06-03 16:54:08 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:08 ExecutorAllocationManager: INFO: Executors 3 removed due to idle timeout. 2022-06-03 16:54:10 TaskSetManager: INFO: Killing attempt 0 for task 62.0 in stage 0.0 (TID 62) on kc1-w-8.c.broad-mpg-gnomad.internal as the attempt 1 succeeded on kc1-w-13.c.broad-mpg-gnomad.internal 2022-06-03 16:54:10 TaskSetManager: INFO: Finished task 62.1 in stage 0.0 (TID 152) in 65039 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (142/150) 2022-06-03 16:54:10 TaskSetManager: WARN: Lost task 62.0 in stage 0.0 (TID 62) (kc1-w-8.c.broad-mpg-gnomad.internal executor 30): TaskKilled (another attempt succeeded) 2022-06-03 16:54:10 TaskSetManager: INFO: task 62.0 in stage 0.0 (TID 62) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:10 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:10 YarnAllocator: INFO: Driver requested a total number of 4 executor(s) for resource profile id: 0. 2022-06-03 16:54:11 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 3. 2022-06-03 16:54:11 DAGScheduler: INFO: Executor lost: 3 (epoch 0) 2022-06-03 16:54:11 BlockManagerMasterEndpoint: INFO: Trying to remove executor 3 from BlockManagerMaster. 2022-06-03 16:54:11 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(3, kc1-w-30.c.broad-mpg-gnomad.internal, 41593, None) 2022-06-03 16:54:11 BlockManagerMaster: INFO: Removed 3 successfully in removeExecutor 2022-06-03 16:54:11 YarnScheduler: INFO: Executor 3 on kc1-w-30.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:13 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 10 2022-06-03 16:54:13 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 10 2022-06-03 16:54:13 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 10. 2022-06-03 16:54:13 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:13 ExecutorAllocationManager: INFO: Executors 10 removed due to idle timeout. 2022-06-03 16:54:14 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 13 2022-06-03 16:54:14 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 13 2022-06-03 16:54:14 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 13. 2022-06-03 16:54:14 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:14 ExecutorAllocationManager: INFO: Executors 13 removed due to idle timeout. 2022-06-03 16:54:15 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 10. 2022-06-03 16:54:15 DAGScheduler: INFO: Executor lost: 10 (epoch 0) 2022-06-03 16:54:15 BlockManagerMasterEndpoint: INFO: Trying to remove executor 10 from BlockManagerMaster. 2022-06-03 16:54:15 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(10, kc1-w-40.c.broad-mpg-gnomad.internal, 41683, None) 2022-06-03 16:54:15 BlockManagerMaster: INFO: Removed 10 successfully in removeExecutor 2022-06-03 16:54:15 YarnScheduler: INFO: Executor 10 on kc1-w-40.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:15 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 13. 2022-06-03 16:54:15 DAGScheduler: INFO: Executor lost: 13 (epoch 0) 2022-06-03 16:54:15 BlockManagerMasterEndpoint: INFO: Trying to remove executor 13 from BlockManagerMaster. 2022-06-03 16:54:15 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(13, kc1-w-24.c.broad-mpg-gnomad.internal, 39175, None) 2022-06-03 16:54:15 BlockManagerMaster: INFO: Removed 13 successfully in removeExecutor 2022-06-03 16:54:15 YarnScheduler: INFO: Executor 13 on kc1-w-24.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:16 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 9 2022-06-03 16:54:16 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 9 2022-06-03 16:54:16 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 9. 2022-06-03 16:54:16 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:16 ExecutorAllocationManager: INFO: Executors 9 removed due to idle timeout. 2022-06-03 16:54:19 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 9. 2022-06-03 16:54:19 DAGScheduler: INFO: Executor lost: 9 (epoch 0) 2022-06-03 16:54:19 BlockManagerMasterEndpoint: INFO: Trying to remove executor 9 from BlockManagerMaster. 2022-06-03 16:54:19 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(9, kc1-w-17.c.broad-mpg-gnomad.internal, 37281, None) 2022-06-03 16:54:19 BlockManagerMaster: INFO: Removed 9 successfully in removeExecutor 2022-06-03 16:54:19 YarnScheduler: INFO: Executor 9 on kc1-w-17.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:19 TaskSetManager: INFO: Killing attempt 1 for task 50.1 in stage 0.0 (TID 166) on kc1-w-28.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-16.c.broad-mpg-gnomad.internal 2022-06-03 16:54:19 TaskSetManager: INFO: Finished task 50.0 in stage 0.0 (TID 50) in 189300 ms on kc1-w-16.c.broad-mpg-gnomad.internal (executor 24) (143/150) 2022-06-03 16:54:19 TaskSetManager: INFO: Killing attempt 1 for task 48.1 in stage 0.0 (TID 186) on kc1-w-14.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-16.c.broad-mpg-gnomad.internal 2022-06-03 16:54:19 TaskSetManager: INFO: Finished task 48.0 in stage 0.0 (TID 48) in 189312 ms on kc1-w-16.c.broad-mpg-gnomad.internal (executor 24) (144/150) 2022-06-03 16:54:19 TaskSetManager: WARN: Lost task 50.1 in stage 0.0 (TID 166) (kc1-w-28.c.broad-mpg-gnomad.internal executor 15): TaskKilled (another attempt succeeded) 2022-06-03 16:54:19 TaskSetManager: INFO: task 50.1 in stage 0.0 (TID 166) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:19 TaskSetManager: WARN: Lost task 48.1 in stage 0.0 (TID 186) (kc1-w-14.c.broad-mpg-gnomad.internal executor 25): TaskKilled (another attempt succeeded) 2022-06-03 16:54:19 TaskSetManager: INFO: task 48.1 in stage 0.0 (TID 186) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:19 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:19 YarnAllocator: INFO: Driver requested a total number of 3 executor(s) for resource profile id: 0. 2022-06-03 16:54:19 TaskSetManager: INFO: Killing attempt 1 for task 49.1 in stage 0.0 (TID 177) on kc1-w-36.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-16.c.broad-mpg-gnomad.internal 2022-06-03 16:54:19 TaskSetManager: INFO: Finished task 49.0 in stage 0.0 (TID 49) in 189641 ms on kc1-w-16.c.broad-mpg-gnomad.internal (executor 24) (145/150) 2022-06-03 16:54:19 TaskSetManager: WARN: Lost task 49.1 in stage 0.0 (TID 177) (kc1-w-36.c.broad-mpg-gnomad.internal executor 12): TaskKilled (another attempt succeeded) 2022-06-03 16:54:19 TaskSetManager: INFO: task 49.1 in stage 0.0 (TID 177) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:22 TaskSetManager: INFO: Killing attempt 1 for task 75.1 in stage 0.0 (TID 169) on kc1-w-49.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-20.c.broad-mpg-gnomad.internal 2022-06-03 16:54:22 TaskSetManager: INFO: Finished task 75.0 in stage 0.0 (TID 75) in 192418 ms on kc1-w-20.c.broad-mpg-gnomad.internal (executor 27) (146/150) 2022-06-03 16:54:22 TaskSetManager: WARN: Lost task 75.1 in stage 0.0 (TID 169) (kc1-w-49.c.broad-mpg-gnomad.internal executor 31): TaskKilled (another attempt succeeded) 2022-06-03 16:54:22 TaskSetManager: INFO: task 75.1 in stage 0.0 (TID 169) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:23 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:23 YarnAllocator: INFO: Driver requested a total number of 2 executor(s) for resource profile id: 0. 2022-06-03 16:54:23 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 16 2022-06-03 16:54:23 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 16 2022-06-03 16:54:23 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 16. 2022-06-03 16:54:23 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:23 ExecutorAllocationManager: INFO: Executors 16 removed due to idle timeout. 2022-06-03 16:54:25 TaskSetManager: INFO: Killing attempt 1 for task 86.1 in stage 0.0 (TID 162) on kc1-w-22.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-8.c.broad-mpg-gnomad.internal 2022-06-03 16:54:25 TaskSetManager: INFO: Finished task 86.0 in stage 0.0 (TID 86) in 194964 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 38) (147/150) 2022-06-03 16:54:26 TaskSetManager: WARN: Lost task 86.1 in stage 0.0 (TID 162) (kc1-w-22.c.broad-mpg-gnomad.internal executor 8): TaskKilled (another attempt succeeded) 2022-06-03 16:54:26 TaskSetManager: INFO: task 86.1 in stage 0.0 (TID 162) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:26 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 37 2022-06-03 16:54:26 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 37 2022-06-03 16:54:26 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 37. 2022-06-03 16:54:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:26 ExecutorAllocationManager: INFO: Executors 37 removed due to idle timeout. 2022-06-03 16:54:26 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 16. 2022-06-03 16:54:26 DAGScheduler: INFO: Executor lost: 16 (epoch 0) 2022-06-03 16:54:26 BlockManagerMasterEndpoint: INFO: Trying to remove executor 16 from BlockManagerMaster. 2022-06-03 16:54:26 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(16, kc1-w-4.c.broad-mpg-gnomad.internal, 37077, None) 2022-06-03 16:54:26 BlockManagerMaster: INFO: Removed 16 successfully in removeExecutor 2022-06-03 16:54:26 YarnScheduler: INFO: Executor 16 on kc1-w-4.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:27 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 37. 2022-06-03 16:54:27 DAGScheduler: INFO: Executor lost: 37 (epoch 0) 2022-06-03 16:54:27 BlockManagerMasterEndpoint: INFO: Trying to remove executor 37 from BlockManagerMaster. 2022-06-03 16:54:27 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(37, kc1-w-27.c.broad-mpg-gnomad.internal, 43697, None) 2022-06-03 16:54:27 BlockManagerMaster: INFO: Removed 37 successfully in removeExecutor 2022-06-03 16:54:27 YarnScheduler: INFO: Executor 37 on kc1-w-27.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:32 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 33 2022-06-03 16:54:32 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 33 2022-06-03 16:54:32 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 33. 2022-06-03 16:54:32 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:32 ExecutorAllocationManager: INFO: Executors 33 removed due to idle timeout. 2022-06-03 16:54:34 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 33. 2022-06-03 16:54:34 DAGScheduler: INFO: Executor lost: 33 (epoch 0) 2022-06-03 16:54:34 BlockManagerMasterEndpoint: INFO: Trying to remove executor 33 from BlockManagerMaster. 2022-06-03 16:54:34 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(33, kc1-w-15.c.broad-mpg-gnomad.internal, 44743, None) 2022-06-03 16:54:34 BlockManagerMaster: INFO: Removed 33 successfully in removeExecutor 2022-06-03 16:54:34 YarnScheduler: INFO: Executor 33 on kc1-w-15.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:35 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 11 2022-06-03 16:54:35 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 11 2022-06-03 16:54:35 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 11. 2022-06-03 16:54:35 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:35 ExecutorAllocationManager: INFO: Executors 11 removed due to idle timeout. 2022-06-03 16:54:36 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 28 2022-06-03 16:54:36 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 28 2022-06-03 16:54:36 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 28. 2022-06-03 16:54:36 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:36 ExecutorAllocationManager: INFO: Executors 28 removed due to idle timeout. 2022-06-03 16:54:36 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 32 2022-06-03 16:54:36 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 32 2022-06-03 16:54:36 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 32. 2022-06-03 16:54:36 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:36 ExecutorAllocationManager: INFO: Executors 32 removed due to idle timeout. 2022-06-03 16:54:37 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 28. 2022-06-03 16:54:37 DAGScheduler: INFO: Executor lost: 28 (epoch 0) 2022-06-03 16:54:37 BlockManagerMasterEndpoint: INFO: Trying to remove executor 28 from BlockManagerMaster. 2022-06-03 16:54:37 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(28, kc1-w-24.c.broad-mpg-gnomad.internal, 35463, None) 2022-06-03 16:54:37 BlockManagerMaster: INFO: Removed 28 successfully in removeExecutor 2022-06-03 16:54:37 YarnScheduler: INFO: Executor 28 on kc1-w-24.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:38 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 11. 2022-06-03 16:54:38 DAGScheduler: INFO: Executor lost: 11 (epoch 0) 2022-06-03 16:54:38 BlockManagerMasterEndpoint: INFO: Trying to remove executor 11 from BlockManagerMaster. 2022-06-03 16:54:38 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(11, kc1-w-14.c.broad-mpg-gnomad.internal, 46275, None) 2022-06-03 16:54:38 BlockManagerMaster: INFO: Removed 11 successfully in removeExecutor 2022-06-03 16:54:38 YarnScheduler: INFO: Executor 11 on kc1-w-14.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:38 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 32. 2022-06-03 16:54:38 DAGScheduler: INFO: Executor lost: 32 (epoch 0) 2022-06-03 16:54:38 BlockManagerMasterEndpoint: INFO: Trying to remove executor 32 from BlockManagerMaster. 2022-06-03 16:54:38 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(32, kc1-w-45.c.broad-mpg-gnomad.internal, 41707, None) 2022-06-03 16:54:38 BlockManagerMaster: INFO: Removed 32 successfully in removeExecutor 2022-06-03 16:54:38 YarnScheduler: INFO: Executor 32 on kc1-w-45.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:39 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 6 2022-06-03 16:54:39 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 6 2022-06-03 16:54:39 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 6. 2022-06-03 16:54:39 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:39 ExecutorAllocationManager: INFO: Executors 6 removed due to idle timeout. 2022-06-03 16:54:42 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 6. 2022-06-03 16:54:42 DAGScheduler: INFO: Executor lost: 6 (epoch 0) 2022-06-03 16:54:42 BlockManagerMasterEndpoint: INFO: Trying to remove executor 6 from BlockManagerMaster. 2022-06-03 16:54:42 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(6, kc1-w-40.c.broad-mpg-gnomad.internal, 37615, None) 2022-06-03 16:54:42 BlockManagerMaster: INFO: Removed 6 successfully in removeExecutor 2022-06-03 16:54:42 YarnScheduler: INFO: Executor 6 on kc1-w-40.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:45 TaskSetManager: INFO: Killing attempt 0 for task 6.0 in stage 0.0 (TID 6) on kc1-w-22.c.broad-mpg-gnomad.internal as the attempt 1 succeeded on kc1-w-39.c.broad-mpg-gnomad.internal 2022-06-03 16:54:45 TaskSetManager: INFO: Finished task 6.1 in stage 0.0 (TID 150) in 99902 ms on kc1-w-39.c.broad-mpg-gnomad.internal (executor 35) (148/150) 2022-06-03 16:54:45 TaskSetManager: WARN: Lost task 6.0 in stage 0.0 (TID 6) (kc1-w-22.c.broad-mpg-gnomad.internal executor 4): TaskKilled (another attempt succeeded) 2022-06-03 16:54:45 TaskSetManager: INFO: task 6.0 in stage 0.0 (TID 6) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:45 YarnAllocator: INFO: Driver requested a total number of 1 executor(s) for resource profile id: 0. 2022-06-03 16:54:45 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 21 2022-06-03 16:54:45 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 21 2022-06-03 16:54:45 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 21. 2022-06-03 16:54:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:45 ExecutorAllocationManager: INFO: Executors 21 removed due to idle timeout. 2022-06-03 16:54:45 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 5 2022-06-03 16:54:45 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 5 2022-06-03 16:54:45 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 5. 2022-06-03 16:54:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:45 ExecutorAllocationManager: INFO: Executors 5 removed due to idle timeout. 2022-06-03 16:54:46 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 20 2022-06-03 16:54:46 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 20 2022-06-03 16:54:46 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 20. 2022-06-03 16:54:46 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:46 ExecutorAllocationManager: INFO: Executors 20 removed due to idle timeout. 2022-06-03 16:54:48 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 21. 2022-06-03 16:54:48 DAGScheduler: INFO: Executor lost: 21 (epoch 0) 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Trying to remove executor 21 from BlockManagerMaster. 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(21, kc1-w-23.c.broad-mpg-gnomad.internal, 33523, None) 2022-06-03 16:54:48 BlockManagerMaster: INFO: Removed 21 successfully in removeExecutor 2022-06-03 16:54:48 YarnScheduler: INFO: Executor 21 on kc1-w-23.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:48 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 20. 2022-06-03 16:54:48 DAGScheduler: INFO: Executor lost: 20 (epoch 0) 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Trying to remove executor 20 from BlockManagerMaster. 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(20, kc1-w-39.c.broad-mpg-gnomad.internal, 37865, None) 2022-06-03 16:54:48 BlockManagerMaster: INFO: Removed 20 successfully in removeExecutor 2022-06-03 16:54:48 YarnScheduler: INFO: Executor 20 on kc1-w-39.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:48 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 5. 2022-06-03 16:54:48 DAGScheduler: INFO: Executor lost: 5 (epoch 0) 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Trying to remove executor 5 from BlockManagerMaster. 2022-06-03 16:54:48 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(5, kc1-w-33.c.broad-mpg-gnomad.internal, 34997, None) 2022-06-03 16:54:48 BlockManagerMaster: INFO: Removed 5 successfully in removeExecutor 2022-06-03 16:54:48 YarnScheduler: INFO: Executor 5 on kc1-w-33.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:52 TaskSetManager: INFO: Killing attempt 1 for task 112.1 in stage 0.0 (TID 154) on kc1-w-14.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-36.c.broad-mpg-gnomad.internal 2022-06-03 16:54:52 TaskSetManager: INFO: Finished task 112.0 in stage 0.0 (TID 112) in 219179 ms on kc1-w-36.c.broad-mpg-gnomad.internal (executor 12) (149/150) 2022-06-03 16:54:52 TaskSetManager: WARN: Lost task 112.1 in stage 0.0 (TID 154) (kc1-w-14.c.broad-mpg-gnomad.internal executor 25): TaskKilled (another attempt succeeded) 2022-06-03 16:54:52 TaskSetManager: INFO: task 112.1 in stage 0.0 (TID 154) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:55 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 14 2022-06-03 16:54:55 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 14 2022-06-03 16:54:55 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 14. 2022-06-03 16:54:55 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:55 ExecutorAllocationManager: INFO: Executors 14 removed due to idle timeout. 2022-06-03 16:54:55 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 17, 19 2022-06-03 16:54:55 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 17, 19 2022-06-03 16:54:55 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 17, 19. 2022-06-03 16:54:55 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:54:55 ExecutorAllocationManager: INFO: Executors 17,19 removed due to idle timeout. 2022-06-03 16:54:55 TaskSetManager: INFO: Killing attempt 1 for task 46.1 in stage 0.0 (TID 165) on kc1-w-10.c.broad-mpg-gnomad.internal as the attempt 0 succeeded on kc1-w-18.c.broad-mpg-gnomad.internal 2022-06-03 16:54:55 TaskSetManager: INFO: Finished task 46.0 in stage 0.0 (TID 46) in 226096 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 23) (150/150) 2022-06-03 16:54:55 DAGScheduler: INFO: ResultStage 0 (collect at SparkBackend.scala:321) finished in 235.268 s 2022-06-03 16:54:55 DAGScheduler: INFO: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 2022-06-03 16:54:55 YarnScheduler: INFO: Killing all running tasks in stage 0: Stage finished 2022-06-03 16:54:55 DAGScheduler: INFO: Job 0 finished: collect at SparkBackend.scala:321, took 235.383345 s 2022-06-03 16:54:55 TaskSetManager: WARN: Lost task 46.1 in stage 0.0 (TID 165) (kc1-w-10.c.broad-mpg-gnomad.internal executor 18): TaskKilled (Stage finished) 2022-06-03 16:54:55 TaskSetManager: INFO: task 46.1 in stage 0.0 (TID 165) failed, but the task will not be re-executed (either because the task failed with a shuffle data fetch failure, so the previous stage needs to be re-run, or because a different copy of the task has already succeeded). 2022-06-03 16:54:55 YarnScheduler: INFO: Removed TaskSet 0.0, whose tasks have all completed, from pool default 2022-06-03 16:54:56 root: INFO: RegionPool: REPORT_THRESHOLD: 256.0K allocated (128.0K blocks / 128.0K chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:56 root: INFO: encoder cache miss (0 hits, 3 misses, 0.000) 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C699HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C699HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C700FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C700FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C701etypeEncode. 2022-06-03 16:54:56 root: INFO: instruction count: 19: __C701etypeEncode.apply 2022-06-03 16:54:56 root: INFO: instruction count: 53: __C701etypeEncode.__m702ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND 2022-06-03 16:54:56 root: INFO: instruction count: 41: __C701etypeEncode.__m703ENCODE_SIntervalPointer_TO_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND 2022-06-03 16:54:56 root: INFO: instruction count: 1: __C701etypeEncode.__m704ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:54:56 root: INFO: instruction count: 4: __C701etypeEncode.__m705ENCODE_SBoolean$_TO_r_bool 2022-06-03 16:54:56 root: INFO: took 4m1.4s 2022-06-03 16:54:56 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:54:56 root: INFO: lowering result: EncodedLiteral 2022-06-03 16:54:56 root: INFO: compiling and evaluating result: EncodedLiteral 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 2: (MakeTuple (0) (EncodedLiteral Array[Interval[Struct{}]])) 2022-06-03 16:54:56 root: INFO: encoder cache miss (0 hits, 4 misses, 0.000) 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C721HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C721HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C722FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C722FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C723etypeEncode. 2022-06-03 16:54:56 root: INFO: instruction count: 7: __C723etypeEncode.apply 2022-06-03 16:54:56 root: INFO: instruction count: 1: __C723etypeEncode.__m724ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:54:56 MemoryStore: INFO: Block broadcast_6 stored as values in memory (estimated size 176.0 B, free 21.7 GiB) 2022-06-03 16:54:56 MemoryStore: INFO: Block broadcast_6_piece0 stored as bytes in memory (estimated size 74.0 B, free 21.7 GiB) 2022-06-03 16:54:56 BlockManagerInfo: INFO: Added broadcast_6_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 74.0 B, free: 21.7 GiB) 2022-06-03 16:54:56 SparkContext: INFO: Created broadcast 6 from broadcast at SparkBackend.scala:313 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C706HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C706HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C707FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C707FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C708Compiled. 2022-06-03 16:54:56 root: INFO: instruction count: 26: __C708Compiled.apply 2022-06-03 16:54:56 root: INFO: instruction count: 9: __C708Compiled.setPartitionIndex 2022-06-03 16:54:56 root: INFO: instruction count: 4: __C708Compiled.addPartitionRegion 2022-06-03 16:54:56 root: INFO: instruction count: 4: __C708Compiled.setPool 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C708Compiled.addHailClassLoader 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C708Compiled.addFS 2022-06-03 16:54:56 root: INFO: instruction count: 67: __C708Compiled.addLiterals 2022-06-03 16:54:56 root: INFO: instruction count: 18: __C708Compiled.__m715DECODE_r_struct_of_END_TO_SBaseStructPointer 2022-06-03 16:54:56 root: INFO: instruction count: 62: __C708Compiled.__m716DECODE_r_array_of_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND_TO_SIndexablePointer 2022-06-03 16:54:56 root: INFO: instruction count: 44: __C708Compiled.__m718INPLACE_DECODE_r_struct_of_r_struct_of_ENDANDr_struct_of_ENDANDr_boolANDr_boolEND_TO_r_interval_of_r_struct_of_END 2022-06-03 16:54:56 root: INFO: instruction count: 8: __C708Compiled.__m719INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:54:56 root: INFO: instruction count: 10: __C708Compiled.__m720INPLACE_DECODE_r_bool_TO_r_bool 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C717Tuple3. 2022-06-03 16:54:56 root: INFO: instruction count: 12: __C717Tuple3. 2022-06-03 16:54:56 root: INFO: encoder cache hit 2022-06-03 16:54:56 root: INFO: took 130.309ms 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (EncodedLiteral Array[Interval[Struct{}]]) 2022-06-03 16:54:56 root: INFO: encoder cache miss (1 hits, 5 misses, 0.167) 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C725HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C725HailClassLoaderContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C726FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C726FSContainer. 2022-06-03 16:54:56 root: INFO: instruction count: 3: __C727etypeEncode. 2022-06-03 16:54:56 root: INFO: instruction count: 19: __C727etypeEncode.apply 2022-06-03 16:54:56 root: INFO: instruction count: 92: __C727etypeEncode.__m728ENCODE_SIndexablePointer_TO_o_array_of_o_struct_of_o_struct_of_ENDANDo_struct_of_ENDANDo_boolANDo_boolEND 2022-06-03 16:54:56 root: INFO: instruction count: 61: __C727etypeEncode.__m729ENCODE_SIntervalPointer_TO_o_struct_of_o_struct_of_ENDANDo_struct_of_ENDANDo_boolANDo_boolEND 2022-06-03 16:54:56 root: INFO: instruction count: 1: __C727etypeEncode.__m730ENCODE_SBaseStructPointer_TO_o_struct_of_END 2022-06-03 16:54:56 root: INFO: instruction count: 4: __C727etypeEncode.__m731ENCODE_SBoolean$_TO_o_bool 2022-06-03 16:54:56 root: INFO: finished execution of query hail_query_1 2022-06-03 16:54:56 root: INFO: RegionPool: FREE: 320.0K allocated (192.0K blocks / 128.0K chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode total 4m2.0s self 162.891ms children 4m1.8s %children 99.93% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 270.649ms self 1.099ms children 269.550ms %children 99.59% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.773ms self 0.773ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 268.723ms self 172.992ms children 95.731ms %children 35.62% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 95.731ms self 2.913ms children 92.818ms %children 96.96% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 17.269ms self 17.269ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 12.387ms self 12.387ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 26.857ms self 26.857ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 22.861ms self 22.861ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 2.623ms self 2.623ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 10.823ms self 10.823ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 15.097ms self 0.021ms children 15.076ms %children 99.86% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 15.005ms self 15.005ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 5.228ms self 0.029ms children 5.199ms %children 99.45% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 5.161ms self 2.623ms children 2.538ms %children 49.17% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 2.538ms self 0.136ms children 2.402ms %children 94.65% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.147ms self 0.147ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.118ms self 0.118ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.171ms self 0.171ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.935ms self 0.935ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.957ms self 0.957ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 4.213ms self 0.016ms children 4.198ms %children 99.63% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 4.120ms self 4.120ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 4m1.5s self 0.040ms children 4m1.5s %children 100.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 4m1.5s self 882.401ms children 4m0.6s %children 99.63% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 362.632ms self 0.083ms children 362.549ms %children 99.98% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 1.110ms self 1.110ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 360.990ms self 111.165ms children 249.825ms %children 69.21% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 249.825ms self 0.581ms children 249.244ms %children 99.77% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 34.349ms self 34.349ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 5.689ms self 5.689ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 8.735ms self 8.735ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 53.826ms self 53.826ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 4.382ms self 4.382ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 40.331ms self 40.331ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 4.826ms self 4.826ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 5.204ms self 5.204ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 3.538ms self 3.538ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 24.803ms self 24.803ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.907ms self 1.907ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 11.512ms self 11.512ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 3.441ms self 3.441ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.937ms self 0.937ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 3.496ms self 3.496ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 28.149ms self 28.149ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.301ms self 1.301ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 12.818ms self 12.818ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.449ms self 0.449ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 1.464ms self 0.030ms children 1.433ms %children 97.92% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.339ms self 0.339ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.659ms self 0.659ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.435ms self 0.435ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 184.690ms self 0.069ms children 184.621ms %children 99.96% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.578ms self 0.578ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 183.580ms self 32.433ms children 151.147ms %children 82.33% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 151.147ms self 0.511ms children 150.636ms %children 99.66% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 2.110ms self 2.110ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.970ms self 0.970ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 2.656ms self 2.656ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 29.702ms self 29.702ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 1.535ms self 1.535ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 14.443ms self 14.443ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 2.830ms self 2.830ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.276ms self 1.276ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 12.854ms self 12.854ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 23.750ms self 23.750ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 1.052ms self 1.052ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 14.418ms self 14.418ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 2.486ms self 2.486ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 4.542ms self 4.542ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 2.562ms self 2.562ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 19.328ms self 19.328ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.856ms self 0.856ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 13.265ms self 13.265ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.463ms self 0.463ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 1.755ms self 0.027ms children 1.728ms %children 98.46% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.571ms self 0.571ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.721ms self 0.721ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.435ms self 0.435ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 3.365ms self 0.037ms children 3.328ms %children 98.90% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.542ms self 0.542ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.902ms self 0.902ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 1.884ms self 1.884ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 175.093ms self 0.047ms children 175.046ms %children 99.97% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.388ms self 0.388ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 174.146ms self 39.457ms children 134.688ms %children 77.34% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 134.688ms self 0.383ms children 134.306ms %children 99.72% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 3.176ms self 3.176ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.154ms self 1.154ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 4.514ms self 4.514ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 23.141ms self 23.141ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 1.781ms self 1.781ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 14.878ms self 14.878ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 3.272ms self 3.272ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.189ms self 1.189ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 4.987ms self 4.987ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 18.735ms self 18.735ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.801ms self 0.801ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 11.621ms self 11.621ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 3.745ms self 3.745ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.146ms self 1.146ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 4.506ms self 4.506ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 24.449ms self 24.449ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.813ms self 0.813ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 10.398ms self 10.398ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.512ms self 0.512ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 3.411s self 2.564s children 846.388ms %children 24.82% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 319.696ms self 0.091ms children 319.605ms %children 99.97% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 1.278ms self 1.278ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 318.037ms self 200.799ms children 117.238ms %children 36.86% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 117.238ms self 0.648ms children 116.590ms %children 99.45% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 4.750ms self 4.750ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.420ms self 1.420ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 3.428ms self 3.428ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 25.757ms self 25.757ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.480ms self 1.480ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 16.700ms self 16.700ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 2.702ms self 2.702ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.964ms self 0.964ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.442ms self 2.442ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 19.082ms self 19.082ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.155ms self 1.155ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 9.864ms self 9.864ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.457ms self 1.457ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.367ms self 1.367ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.935ms self 2.935ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 11.228ms self 11.228ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.675ms self 0.675ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 9.184ms self 9.184ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.289ms self 0.289ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 2.373ms self 0.036ms children 2.337ms %children 98.49% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.494ms self 0.494ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 1.477ms self 1.477ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.366ms self 0.366ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 82.456ms self 0.038ms children 82.418ms %children 99.95% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 82.169ms self 16.279ms children 65.891ms %children 80.19% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 65.891ms self 0.331ms children 65.560ms %children 99.50% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.765ms self 0.765ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.224ms self 1.224ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 2.298ms self 2.298ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 11.783ms self 11.783ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.601ms self 0.601ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 8.002ms self 8.002ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.971ms self 0.971ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.531ms self 0.531ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.609ms self 1.609ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 10.968ms self 10.968ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.599ms self 0.599ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 7.605ms self 7.605ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.856ms self 0.856ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.586ms self 0.586ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.409ms self 1.409ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 6.940ms self 6.940ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.864ms self 0.864ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 7.945ms self 7.945ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.130ms self 0.130ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 252.120ms self 0.049ms children 252.070ms %children 99.98% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.246ms self 0.246ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 250.038ms self 250.038ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 1.787ms self 1.787ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 146.097ms self 0.064ms children 146.033ms %children 99.96% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.143ms self 0.143ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 145.716ms self 58.347ms children 87.369ms %children 59.96% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 87.369ms self 0.368ms children 87.001ms %children 99.58% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.244ms self 1.244ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 3.519ms self 3.519ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 3.346ms self 3.346ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 18.433ms self 18.433ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.629ms self 0.629ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 11.685ms self 11.685ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.156ms self 1.156ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.434ms self 0.434ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.541ms self 1.541ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 10.618ms self 10.618ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.559ms self 0.559ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 8.450ms self 8.450ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.233ms self 1.233ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.590ms self 0.590ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.738ms self 1.738ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 12.915ms self 12.915ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.561ms self 0.561ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 8.349ms self 8.349ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.174ms self 0.174ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 43.647ms self 43.647ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 27.712ms self 27.712ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 3m56.3s self 3m56.3s children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 1.650ms self 0.023ms children 1.628ms %children 98.63% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 1.606ms self 0.746ms children 0.860ms %children 53.57% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.860ms self 0.074ms children 0.787ms %children 91.45% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.129ms self 0.129ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.343ms self 0.343ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.078ms self 0.078ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.114ms self 0.114ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.093ms self 0.042ms children 0.051ms %children 54.54% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.747ms self 0.012ms children 0.735ms %children 98.39% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.729ms self 0.330ms children 0.398ms %children 54.66% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.398ms self 0.090ms children 0.308ms %children 77.39% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.061ms self 0.007ms children 0.054ms %children 88.68% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.060ms self 0.030ms children 0.030ms %children 49.71% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.832ms self 0.012ms children 0.820ms %children 98.60% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.814ms self 0.382ms children 0.432ms %children 53.06% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.432ms self 0.035ms children 0.397ms %children 91.90% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.183ms self 0.183ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 107.142ms self 97.212ms children 9.930ms %children 9.27% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 2.686ms self 0.045ms children 2.641ms %children 98.33% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 2.620ms self 1.052ms children 1.569ms %children 59.87% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 1.569ms self 0.087ms children 1.482ms %children 94.46% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.237ms self 0.237ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.244ms self 0.244ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.501ms self 0.501ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.378ms self 0.378ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.128ms self 0.020ms children 0.108ms %children 84.20% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 1.975ms self 0.041ms children 1.934ms %children 97.93% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.913ms self 0.751ms children 1.162ms %children 60.73% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 1.162ms self 0.057ms children 1.105ms %children 95.08% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.089ms self 0.089ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.081ms self 0.081ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.501ms self 0.501ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.090ms self 0.090ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.260ms self 0.260ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 1.559ms self 0.044ms children 1.515ms %children 97.15% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 1.468ms self 1.468ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 2.211ms self 0.032ms children 2.180ms %children 98.57% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 2.165ms self 0.876ms children 1.289ms %children 59.56% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 1.289ms self 0.041ms children 1.249ms %children 96.85% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.100ms self 0.100ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.127ms self 0.127ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.547ms self 0.547ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.262ms self 0.262ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 1.371ms self 1.371ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 14.760ms self 14.760ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 1.804ms self 0.016ms children 1.788ms %children 99.10% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 1.774ms self 0.741ms children 1.033ms %children 58.24% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 1.033ms self 0.045ms children 0.988ms %children 95.61% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.237ms self 0.237ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.143ms self 0.143ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.305ms self 0.305ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.094ms self 0.094ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.160ms self 0.160ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/Compile total 3.990ms self 3.990ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 0.753ms self 0.753ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2022-06-03 16:54:56 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:57 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:57 root: INFO: timing SparkBackend.parse_table_ir total 325.374ms self 325.374ms children 0.000ms %children 0.00% 2022-06-03 16:54:57 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:57 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:57 root: INFO: timing SparkBackend.parse_value_ir total 35.089ms self 35.089ms children 0.000ms %children 0.00% 2022-06-03 16:54:57 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:57 root: INFO: starting execution of query hail_query_2 of initial size 93 2022-06-03 16:54:57 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 93: (GetField rows (TableCollect (TableHead 11 (TableMapRows (TableMapRows (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false)))) (Let __uid_3 (GetField sift (Ref row)) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen sift codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (GetField locus (Ref row))) (alleles (GetField alleles (Ref row))) (polyphen (GetField polyphen (Ref row))) (`sift.prediction` (GetField prediction (Ref __uid_3))) (`sift.score` (GetField score (Ref __uid_3))) (codons (GetField codons (Ref row))) (most_severe_consequence (GetField most_severe_consequence (Ref row))) (transcript (GetField transcript (Ref row))) (ref (GetField ref (Ref row))) (alt (GetField alt (Ref row))) (interval (GetField interval (Ref row))) (oe (GetField oe (Ref row))) (misbad (GetField misbad (Ref row))) (fitted_score (GetField fitted_score (Ref row))) (gnomad_scores (GetField gnomad_scores (Ref row)))))) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen `sift.prediction` `sift.score` codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (ApplySpecial 1 showStr () String (GetField locus (Ref row)))) (alleles (ApplySpecial 2 showStr () String (GetField alleles (Ref row)))) (polyphen (ApplySpecial 3 showStr () String (GetField polyphen (Ref row)))) (`sift.prediction` (ApplySpecial 4 showStr () String (GetField `sift.prediction` (Ref row)))) (`sift.score` (ApplySpecial 5 showStr () String (GetField `sift.score` (Ref row)))) (codons (ApplySpecial 6 showStr () String (GetField codons (Ref row)))) (most_severe_consequence (ApplySpecial 7 showStr () String (GetField most_severe_consequence (Ref row)))) (transcript (ApplySpecial 8 showStr () String (GetField transcript (Ref row)))) (ref (ApplySpecial 9 showStr () String (GetField ref (Ref row)))) (alt (ApplySpecial 10 showStr () String (GetField alt (Ref row)))) (interval (ApplySpecial 11 showStr () String (GetField interval (Ref row)))) (oe (ApplySpecial 12 showStr () String (GetField oe (Ref row)))) (misbad (ApplySpecial 13 showStr () String (GetField misbad (Ref row)))) (fitted_score (ApplySpecial 14 showStr () String (GetField fitted_score (Ref row)))) (gnomad_scores (ApplySpecial 15 showStr () String (GetField gnomad_scores (Ref row))))))))) 2022-06-03 16:54:57 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_311 (GetField sift (Ref row)) (Let __iruid_312 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_312)) None (locus (ApplySpecial 1 showStr () String (GetField locus (Ref __iruid_312)))) (alleles (ApplySpecial 2 showStr () String (GetField alleles (Ref __iruid_312)))) (polyphen (ApplySpecial 3 showStr () String (GetField polyphen (Ref __iruid_312)))) (`sift.prediction` (ApplySpecial 4 showStr () String (GetField prediction (Ref __iruid_311)))) (`sift.score` (ApplySpecial 5 showStr () String (GetField score (Ref __iruid_311)))) (codons (ApplySpecial 6 showStr () String (GetField codons (Ref __iruid_312)))) (most_severe_consequence (ApplySpecial 7 showStr () String (GetField most_severe_consequence (Ref __iruid_312)))) (transcript (ApplySpecial 8 showStr () String (GetField transcript (Ref __iruid_312)))) (ref (ApplySpecial 9 showStr () String (GetField ref (Ref __iruid_312)))) (alt (ApplySpecial 10 showStr () String (GetField alt (Ref __iruid_312)))) (interval (ApplySpecial 11 showStr () String (GetField interval (Ref __iruid_312)))) (oe (ApplySpecial 12 showStr () String (GetField oe (Ref __iruid_312)))) (misbad (ApplySpecial 13 showStr () String (GetField misbad (Ref __iruid_312)))) (fitted_score (ApplySpecial 14 showStr () String (GetField fitted_score (Ref __iruid_312)))) (gnomad_scores (ApplySpecial 15 showStr () String (GetField gnomad_scores (Ref __iruid_312)))))))))) 2022-06-03 16:54:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_311 (GetField sift (Ref row)) (Let __iruid_312 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_312)) None (locus (ApplySpecial 1 showStr () String (GetField locus (Ref __iruid_312)))) (alleles (ApplySpecial 2 showStr () String (GetField alleles (Ref __iruid_312)))) (polyphen (ApplySpecial 3 showStr () String (GetField polyphen (Ref __iruid_312)))) (`sift.prediction` (ApplySpecial 4 showStr () String (GetField prediction (Ref __iruid_311)))) (`sift.score` (ApplySpecial 5 showStr () String (GetField score (Ref __iruid_311)))) (codons (ApplySpecial 6 showStr () String (GetField codons (Ref __iruid_312)))) (most_severe_consequence (ApplySpecial 7 showStr () String (GetField most_severe_consequence (Ref __iruid_312)))) (transcript (ApplySpecial 8 showStr () String (GetField transcript (Ref __iruid_312)))) (ref (ApplySpecial 9 showStr () String (GetField ref (Ref __iruid_312)))) (alt (ApplySpecial 10 showStr () String (GetField alt (Ref __iruid_312)))) (interval (ApplySpecial 11 showStr () String (GetField interval (Ref __iruid_312)))) (oe (ApplySpecial 12 showStr () String (GetField oe (Ref __iruid_312)))) (misbad (ApplySpecial 13 showStr () String (GetField misbad (Ref __iruid_312)))) (fitted_score (ApplySpecial 14 showStr () String (GetField fitted_score (Ref __iruid_312)))) (gnomad_scores (ApplySpecial 15 showStr () String (GetField gnomad_scores (Ref __iruid_312)))))))))) 2022-06-03 16:54:57 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 14. 2022-06-03 16:54:57 DAGScheduler: INFO: Executor lost: 14 (epoch 0) 2022-06-03 16:54:57 BlockManagerMasterEndpoint: INFO: Trying to remove executor 14 from BlockManagerMaster. 2022-06-03 16:54:57 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(14, kc1-w-18.c.broad-mpg-gnomad.internal, 36891, None) 2022-06-03 16:54:57 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 17. 2022-06-03 16:54:57 BlockManagerMaster: INFO: Removed 14 successfully in removeExecutor 2022-06-03 16:54:57 DAGScheduler: INFO: Executor lost: 17 (epoch 0) 2022-06-03 16:54:57 BlockManagerMasterEndpoint: INFO: Trying to remove executor 17 from BlockManagerMaster. 2022-06-03 16:54:57 YarnScheduler: INFO: Executor 14 on kc1-w-18.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:57 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(17, kc1-w-7.c.broad-mpg-gnomad.internal, 40761, None) 2022-06-03 16:54:57 YarnScheduler: INFO: Executor 17 on kc1-w-7.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:57 BlockManagerMaster: INFO: Removed 17 successfully in removeExecutor 2022-06-03 16:54:57 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_317 (GetField sift (Ref row)) (Let __iruid_318 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_318)) None (locus (ApplySpecial 1 showStr () String (GetField locus (Ref __iruid_318)))) (alleles (ApplySpecial 2 showStr () String (GetField alleles (Ref __iruid_318)))) (polyphen (ApplySpecial 3 showStr () String (GetField polyphen (Ref __iruid_318)))) (`sift.prediction` (ApplySpecial 4 showStr () String (GetField prediction (Ref __iruid_317)))) (`sift.score` (ApplySpecial 5 showStr () String (GetField score (Ref __iruid_317)))) (codons (ApplySpecial 6 showStr () String (GetField codons (Ref __iruid_318)))) (most_severe_consequence (ApplySpecial 7 showStr () String (GetField most_severe_consequence (Ref __iruid_318)))) (transcript (ApplySpecial 8 showStr () String (GetField transcript (Ref __iruid_318)))) (ref (ApplySpecial 9 showStr () String (GetField ref (Ref __iruid_318)))) (alt (ApplySpecial 10 showStr () String (GetField alt (Ref __iruid_318)))) (interval (ApplySpecial 11 showStr () String (GetField interval (Ref __iruid_318)))) (oe (ApplySpecial 12 showStr () String (GetField oe (Ref __iruid_318)))) (misbad (ApplySpecial 13 showStr () String (GetField misbad (Ref __iruid_318)))) (fitted_score (ApplySpecial 14 showStr () String (GetField fitted_score (Ref __iruid_318)))) (gnomad_scores (ApplySpecial 15 showStr () String (GetField gnomad_scores (Ref __iruid_318)))))))))) 2022-06-03 16:54:57 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:54:57 root: INFO: lowering result: TableCollect 2022-06-03 16:54:57 root: ERROR: error while applying lowering 'LowerOrInterpretNonCompilable' 2022-06-03 16:54:57 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:57 root: ERROR: AssertionError: assertion failed From java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:208) at is.hail.rvd.IndexedRVDSpec2.readTableStage(AbstractRVDSpec.scala:542) at is.hail.expr.ir.TableNativeReader.lower(TableIR.scala:1039) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:706) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1551) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:944) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1142) at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:458) at is.hail.expr.ir.lowering.LowerTableIR$.apply(LowerTableIR.scala:537) at is.hail.expr.ir.lowering.LowerToCDA$.lower(LowerToCDA.scala:69) at is.hail.expr.ir.lowering.LowerToCDA$.apply(LowerToCDA.scala:18) at is.hail.expr.ir.lowering.LowerToDistributedArrayPass.transform(LoweringPass.scala:77) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:27) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:67) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:53) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:72) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:69) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14) at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:64) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:15) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:13) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:13) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:47) at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:416) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$2(SparkBackend.scala:452) at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:69) at is.hail.utils.package$.using(package.scala:640) at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:69) at is.hail.utils.package$.using(package.scala:640) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:58) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:310) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$1(SparkBackend.scala:449) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeEncode(SparkBackend.scala:448) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2022-06-03 16:54:58 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 19. 2022-06-03 16:54:58 DAGScheduler: INFO: Executor lost: 19 (epoch 0) 2022-06-03 16:54:58 BlockManagerMasterEndpoint: INFO: Trying to remove executor 19 from BlockManagerMaster. 2022-06-03 16:54:58 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(19, kc1-w-13.c.broad-mpg-gnomad.internal, 43623, None) 2022-06-03 16:54:58 BlockManagerMaster: INFO: Removed 19 successfully in removeExecutor 2022-06-03 16:54:58 YarnScheduler: INFO: Executor 19 on kc1-w-13.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:54:58 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:58 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:58 root: INFO: timing SparkBackend.parse_value_ir total 13.899ms self 13.899ms children 0.000ms %children 0.00% 2022-06-03 16:54:58 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:58 root: INFO: starting execution of query hail_query_3 of initial size 93 2022-06-03 16:54:58 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 93: (GetField rows (TableCollect (TableHead 11 (TableMapRows (TableMapRows (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false)))) (Let __uid_4 (GetField sift (Ref row)) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen sift codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (GetField locus (Ref row))) (alleles (GetField alleles (Ref row))) (polyphen (GetField polyphen (Ref row))) (`sift.prediction` (GetField prediction (Ref __uid_4))) (`sift.score` (GetField score (Ref __uid_4))) (codons (GetField codons (Ref row))) (most_severe_consequence (GetField most_severe_consequence (Ref row))) (transcript (GetField transcript (Ref row))) (ref (GetField ref (Ref row))) (alt (GetField alt (Ref row))) (interval (GetField interval (Ref row))) (oe (GetField oe (Ref row))) (misbad (GetField misbad (Ref row))) (fitted_score (GetField fitted_score (Ref row))) (gnomad_scores (GetField gnomad_scores (Ref row)))))) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen `sift.prediction` `sift.score` codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (ApplySpecial 16 showStr () String (GetField locus (Ref row)))) (alleles (ApplySpecial 17 showStr () String (GetField alleles (Ref row)))) (polyphen (ApplySpecial 18 showStr () String (GetField polyphen (Ref row)))) (`sift.prediction` (ApplySpecial 19 showStr () String (GetField `sift.prediction` (Ref row)))) (`sift.score` (ApplySpecial 20 showStr () String (GetField `sift.score` (Ref row)))) (codons (ApplySpecial 21 showStr () String (GetField codons (Ref row)))) (most_severe_consequence (ApplySpecial 22 showStr () String (GetField most_severe_consequence (Ref row)))) (transcript (ApplySpecial 23 showStr () String (GetField transcript (Ref row)))) (ref (ApplySpecial 24 showStr () String (GetField ref (Ref row)))) (alt (ApplySpecial 25 showStr () String (GetField alt (Ref row)))) (interval (ApplySpecial 26 showStr () String (GetField interval (Ref row)))) (oe (ApplySpecial 27 showStr () String (GetField oe (Ref row)))) (misbad (ApplySpecial 28 showStr () String (GetField misbad (Ref row)))) (fitted_score (ApplySpecial 29 showStr () String (GetField fitted_score (Ref row)))) (gnomad_scores (ApplySpecial 30 showStr () String (GetField gnomad_scores (Ref row))))))))) 2022-06-03 16:54:58 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_331 (GetField sift (Ref row)) (Let __iruid_332 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_332)) None (locus (ApplySpecial 16 showStr () String (GetField locus (Ref __iruid_332)))) (alleles (ApplySpecial 17 showStr () String (GetField alleles (Ref __iruid_332)))) (polyphen (ApplySpecial 18 showStr () String (GetField polyphen (Ref __iruid_332)))) (`sift.prediction` (ApplySpecial 19 showStr () String (GetField prediction (Ref __iruid_331)))) (`sift.score` (ApplySpecial 20 showStr () String (GetField score (Ref __iruid_331)))) (codons (ApplySpecial 21 showStr () String (GetField codons (Ref __iruid_332)))) (most_severe_consequence (ApplySpecial 22 showStr () String (GetField most_severe_consequence (Ref __iruid_332)))) (transcript (ApplySpecial 23 showStr () String (GetField transcript (Ref __iruid_332)))) (ref (ApplySpecial 24 showStr () String (GetField ref (Ref __iruid_332)))) (alt (ApplySpecial 25 showStr () String (GetField alt (Ref __iruid_332)))) (interval (ApplySpecial 26 showStr () String (GetField interval (Ref __iruid_332)))) (oe (ApplySpecial 27 showStr () String (GetField oe (Ref __iruid_332)))) (misbad (ApplySpecial 28 showStr () String (GetField misbad (Ref __iruid_332)))) (fitted_score (ApplySpecial 29 showStr () String (GetField fitted_score (Ref __iruid_332)))) (gnomad_scores (ApplySpecial 30 showStr () String (GetField gnomad_scores (Ref __iruid_332)))))))))) 2022-06-03 16:54:58 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_331 (GetField sift (Ref row)) (Let __iruid_332 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_332)) None (locus (ApplySpecial 16 showStr () String (GetField locus (Ref __iruid_332)))) (alleles (ApplySpecial 17 showStr () String (GetField alleles (Ref __iruid_332)))) (polyphen (ApplySpecial 18 showStr () String (GetField polyphen (Ref __iruid_332)))) (`sift.prediction` (ApplySpecial 19 showStr () String (GetField prediction (Ref __iruid_331)))) (`sift.score` (ApplySpecial 20 showStr () String (GetField score (Ref __iruid_331)))) (codons (ApplySpecial 21 showStr () String (GetField codons (Ref __iruid_332)))) (most_severe_consequence (ApplySpecial 22 showStr () String (GetField most_severe_consequence (Ref __iruid_332)))) (transcript (ApplySpecial 23 showStr () String (GetField transcript (Ref __iruid_332)))) (ref (ApplySpecial 24 showStr () String (GetField ref (Ref __iruid_332)))) (alt (ApplySpecial 25 showStr () String (GetField alt (Ref __iruid_332)))) (interval (ApplySpecial 26 showStr () String (GetField interval (Ref __iruid_332)))) (oe (ApplySpecial 27 showStr () String (GetField oe (Ref __iruid_332)))) (misbad (ApplySpecial 28 showStr () String (GetField misbad (Ref __iruid_332)))) (fitted_score (ApplySpecial 29 showStr () String (GetField fitted_score (Ref __iruid_332)))) (gnomad_scores (ApplySpecial 30 showStr () String (GetField gnomad_scores (Ref __iruid_332)))))))))) 2022-06-03 16:54:58 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht (IntervalRead: 1 intervals, filter=false))))) (Let __iruid_337 (GetField sift (Ref row)) (Let __iruid_338 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_338)) None (locus (ApplySpecial 16 showStr () String (GetField locus (Ref __iruid_338)))) (alleles (ApplySpecial 17 showStr () String (GetField alleles (Ref __iruid_338)))) (polyphen (ApplySpecial 18 showStr () String (GetField polyphen (Ref __iruid_338)))) (`sift.prediction` (ApplySpecial 19 showStr () String (GetField prediction (Ref __iruid_337)))) (`sift.score` (ApplySpecial 20 showStr () String (GetField score (Ref __iruid_337)))) (codons (ApplySpecial 21 showStr () String (GetField codons (Ref __iruid_338)))) (most_severe_consequence (ApplySpecial 22 showStr () String (GetField most_severe_consequence (Ref __iruid_338)))) (transcript (ApplySpecial 23 showStr () String (GetField transcript (Ref __iruid_338)))) (ref (ApplySpecial 24 showStr () String (GetField ref (Ref __iruid_338)))) (alt (ApplySpecial 25 showStr () String (GetField alt (Ref __iruid_338)))) (interval (ApplySpecial 26 showStr () String (GetField interval (Ref __iruid_338)))) (oe (ApplySpecial 27 showStr () String (GetField oe (Ref __iruid_338)))) (misbad (ApplySpecial 28 showStr () String (GetField misbad (Ref __iruid_338)))) (fitted_score (ApplySpecial 29 showStr () String (GetField fitted_score (Ref __iruid_338)))) (gnomad_scores (ApplySpecial 30 showStr () String (GetField gnomad_scores (Ref __iruid_338)))))))))) 2022-06-03 16:54:58 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:54:58 root: INFO: lowering result: TableCollect 2022-06-03 16:54:58 root: ERROR: error while applying lowering 'LowerOrInterpretNonCompilable' 2022-06-03 16:54:58 root: INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:58 root: ERROR: AssertionError: assertion failed From java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:208) at is.hail.rvd.IndexedRVDSpec2.readTableStage(AbstractRVDSpec.scala:542) at is.hail.expr.ir.TableNativeReader.lower(TableIR.scala:1039) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:706) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1551) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:944) at is.hail.expr.ir.lowering.LowerTableIR$.lower$2(LowerTableIR.scala:686) at is.hail.expr.ir.lowering.LowerTableIR$.applyTable(LowerTableIR.scala:1142) at is.hail.expr.ir.lowering.LowerTableIR$.lower$1(LowerTableIR.scala:458) at is.hail.expr.ir.lowering.LowerTableIR$.apply(LowerTableIR.scala:537) at is.hail.expr.ir.lowering.LowerToCDA$.lower(LowerToCDA.scala:69) at is.hail.expr.ir.lowering.LowerToCDA$.apply(LowerToCDA.scala:18) at is.hail.expr.ir.lowering.LowerToDistributedArrayPass.transform(LoweringPass.scala:77) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.evaluate$1(LowerOrInterpretNonCompilable.scala:27) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:67) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.rewrite$1(LowerOrInterpretNonCompilable.scala:53) at is.hail.expr.ir.LowerOrInterpretNonCompilable$.apply(LowerOrInterpretNonCompilable.scala:72) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.transform(LoweringPass.scala:69) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$3(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.$anonfun$apply$1(LoweringPass.scala:16) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass.apply(LoweringPass.scala:14) at is.hail.expr.ir.lowering.LoweringPass.apply$(LoweringPass.scala:13) at is.hail.expr.ir.lowering.LowerOrInterpretNonCompilablePass$.apply(LoweringPass.scala:64) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1(LoweringPipeline.scala:15) at is.hail.expr.ir.lowering.LoweringPipeline.$anonfun$apply$1$adapted(LoweringPipeline.scala:13) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:13) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:47) at is.hail.backend.spark.SparkBackend._execute(SparkBackend.scala:416) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$2(SparkBackend.scala:452) at is.hail.backend.ExecuteContext$.$anonfun$scoped$3(ExecuteContext.scala:69) at is.hail.utils.package$.using(package.scala:640) at is.hail.backend.ExecuteContext$.$anonfun$scoped$2(ExecuteContext.scala:69) at is.hail.utils.package$.using(package.scala:640) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:58) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:310) at is.hail.backend.spark.SparkBackend.$anonfun$executeEncode$1(SparkBackend.scala:449) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeEncode(SparkBackend.scala:448) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2022-06-03 16:54:58 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:59 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:59 root: INFO: timing SparkBackend.parse_table_ir total 320.784ms self 320.784ms children 0.000ms %children 0.00% 2022-06-03 16:54:59 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:59 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 16: Thread-4 2022-06-03 16:54:59 root: INFO: timing SparkBackend.parse_value_ir total 7.037ms self 7.037ms children 0.000ms %children 0.00% 2022-06-03 16:54:59 root: INFO: RegionPool: initialized for thread 16: Thread-4 2022-06-03 16:54:59 root: INFO: starting execution of query hail_query_4 of initial size 93 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 93: (GetField rows (TableCollect (TableHead 11 (TableMapRows (TableMapRows (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht ))) (Let __uid_5 (GetField sift (Ref row)) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen sift codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (GetField locus (Ref row))) (alleles (GetField alleles (Ref row))) (polyphen (GetField polyphen (Ref row))) (`sift.prediction` (GetField prediction (Ref __uid_5))) (`sift.score` (GetField score (Ref __uid_5))) (codons (GetField codons (Ref row))) (most_severe_consequence (GetField most_severe_consequence (Ref row))) (transcript (GetField transcript (Ref row))) (ref (GetField ref (Ref row))) (alt (GetField alt (Ref row))) (interval (GetField interval (Ref row))) (oe (GetField oe (Ref row))) (misbad (GetField misbad (Ref row))) (fitted_score (GetField fitted_score (Ref row))) (gnomad_scores (GetField gnomad_scores (Ref row)))))) (InsertFields (SelectFields () (SelectFields (locus alleles polyphen `sift.prediction` `sift.score` codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row))) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref row)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref row)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref row)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField `sift.prediction` (Ref row)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField `sift.score` (Ref row)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref row)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref row)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref row)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref row)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref row)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref row)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref row)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref row)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref row)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref row))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht )))) (Let __iruid_351 (GetField sift (Ref row)) (Let __iruid_352 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_352)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_352)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_352)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_352)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_351)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_351)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_352)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_352)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_352)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_352)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_352)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_352)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_352)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_352)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_352)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_352)))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht )))) (Let __iruid_351 (GetField sift (Ref row)) (Let __iruid_352 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_352)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_352)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_352)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_352)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_351)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_351)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_352)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_352)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_352)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_352)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_352)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_352)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_352)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_352)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_352)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_352)))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 60: (GetField rows (TableCollect (TableMapRows (TableHead 11 (TableOrderBy () (TableRead None False (TableNativeReader gs://regional_missense_constraint/temp/mpc_temp_gnomad.ht )))) (Let __iruid_357 (GetField sift (Ref row)) (Let __iruid_358 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_358)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_358)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_358)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_358)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_357)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_357)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_358)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_358)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_358)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_358)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_358)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_358)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_358)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_358)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_358)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_358)))))))))) 2022-06-03 16:54:59 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:54:59 root: INFO: lowering result: TableCollect 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 137: (ToArray (Let __iruid_360 (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)) (Let __iruid_362 (ToArray (ToStream False (Literal Array[Struct{path:String}] ))) (Let __iruid_372 (TailLoop __iruid_363 (__iruid_364) (I32 4) (Let __iruid_369 (Let __iruid_360 (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)) (Let __iruid_368 (CollectDistributedArray __iruid_365 __iruid_367 (StreamTake (ToStream False (Ref __iruid_362)) (Ref __iruid_364)) (MakeStruct (__iruid_360 (Ref __iruid_360))) (Let __iruid_360 (GetField __iruid_360 (Ref __iruid_367)) (StreamLen (StreamTake (Let __iruid_366 (Ref __iruid_365) (Let __iruid_361 (Ref __iruid_366) (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_361))))) (I32 11))))) (Ref __iruid_368))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_371 __iruid_370 (ToStream False (Ref __iruid_369)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_371) (Ref __iruid_370)))) (I64 11)) (ApplyComparisonOp LTEQ (ArrayLen (Ref __iruid_362)) (ArrayLen (Ref __iruid_369)))) (Ref __iruid_369) (Recur __iruid_363 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_364) (I32 4)))))) (Let __iruid_377 (Let __iruid_373 (ArrayLen (Ref __iruid_372)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_373) (I32 0)) (MakeTuple (0 1) (I32 0) (I64 0)) (TailLoop __iruid_374 (__iruid_375 __iruid_376) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_375) (ApplyBinaryPrimOp Subtract (Ref __iruid_373) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_376) (Cast Int64 (ArrayRef -1 (Ref __iruid_372) (Ref __iruid_375)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_375) (I32 1)) (Ref __iruid_376)) (Recur __iruid_374 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_375) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_376) (Cast Int64 (ArrayRef -1 (Ref __iruid_372) (Ref __iruid_375))))))))) (StreamZip -1 AssumeSameLength (part howMany) (StreamTake (ToStream False (Ref __iruid_362)) (GetTupleElement 0 (Ref __iruid_377))) (StreamMap __iruid_378 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_377)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_378) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_377)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_377))) (ArrayRef -1 (Ref __iruid_372) (Ref __iruid_378)))) (MakeStruct (numberToTake (Ref howMany)) (old (Ref part))))))))) 2022-06-03 16:54:59 root: INFO: Prune: MakeStruct: eliminating field '__iruid_360' 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 108: (ToArray (Let __iruid_418 (TailLoop __iruid_419 (__iruid_420) (I32 4) (Let __iruid_421 (CollectDistributedArray __iruid_422 __iruid_423 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_420)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_422))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_424 __iruid_425 (ToStream False (Ref __iruid_421)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_424) (Ref __iruid_425)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_421)))) (Ref __iruid_421) (Recur __iruid_419 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_420) (I32 4)))))) (Let __iruid_426 (ArrayLen (Ref __iruid_418)) (Let __iruid_427 (If (ApplyComparisonOp EQWithNA (Ref __iruid_426) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_428 (__iruid_429 __iruid_430) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_429) (ApplyBinaryPrimOp Subtract (Ref __iruid_426) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_430) (Cast Int64 (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_429)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_429) (I32 1)) (Ref __iruid_430)) (Recur __iruid_428 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_429) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_430) (Cast Int64 (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_429)))))))) (StreamZip -1 AssumeSameLength (__iruid_431 __iruid_432) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_427))) (StreamMap __iruid_433 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_427)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_433) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_427)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_427))) (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_433)))) (MakeStruct (numberToTake (Ref __iruid_432)) (old (Ref __iruid_431)))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 108: (ToArray (Let __iruid_418 (TailLoop __iruid_419 (__iruid_420) (I32 4) (Let __iruid_421 (CollectDistributedArray __iruid_422 __iruid_423 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_420)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_422))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_424 __iruid_425 (ToStream False (Ref __iruid_421)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_424) (Ref __iruid_425)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_421)))) (Ref __iruid_421) (Recur __iruid_419 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_420) (I32 4)))))) (Let __iruid_426 (ArrayLen (Ref __iruid_418)) (Let __iruid_427 (If (ApplyComparisonOp EQWithNA (Ref __iruid_426) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_428 (__iruid_429 __iruid_430) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_429) (ApplyBinaryPrimOp Subtract (Ref __iruid_426) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_430) (Cast Int64 (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_429)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_429) (I32 1)) (Ref __iruid_430)) (Recur __iruid_428 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_429) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_430) (Cast Int64 (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_429)))))))) (StreamZip -1 AssumeSameLength (__iruid_431 __iruid_432) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_427))) (StreamMap __iruid_433 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_427)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_433) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_427)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_427))) (ArrayRef -1 (Ref __iruid_418) (Ref __iruid_433)))) (MakeStruct (numberToTake (Ref __iruid_432)) (old (Ref __iruid_431)))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 108: (ToArray (Let __iruid_466 (TailLoop __iruid_467 (__iruid_468) (I32 4) (Let __iruid_469 (CollectDistributedArray __iruid_470 __iruid_471 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_468)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_470))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_472 __iruid_473 (ToStream False (Ref __iruid_469)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_472) (Ref __iruid_473)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_469)))) (Ref __iruid_469) (Recur __iruid_467 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_468) (I32 4)))))) (Let __iruid_474 (ArrayLen (Ref __iruid_466)) (Let __iruid_475 (If (ApplyComparisonOp EQWithNA (Ref __iruid_474) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_476 (__iruid_477 __iruid_478) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_477) (ApplyBinaryPrimOp Subtract (Ref __iruid_474) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_478) (Cast Int64 (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_477)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_477) (I32 1)) (Ref __iruid_478)) (Recur __iruid_476 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_477) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_478) (Cast Int64 (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_477)))))))) (StreamZip -1 AssumeSameLength (__iruid_479 __iruid_480) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_475))) (StreamMap __iruid_481 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_475)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_481) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_475)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_475))) (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_481)))) (MakeStruct (numberToTake (Ref __iruid_480)) (old (Ref __iruid_479)))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 108: (ToArray (Let __iruid_466 (TailLoop __iruid_467 (__iruid_468) (I32 4) (Let __iruid_469 (CollectDistributedArray __iruid_470 __iruid_471 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_468)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_470))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_472 __iruid_473 (ToStream False (Ref __iruid_469)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_472) (Ref __iruid_473)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_469)))) (Ref __iruid_469) (Recur __iruid_467 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_468) (I32 4)))))) (Let __iruid_474 (ArrayLen (Ref __iruid_466)) (Let __iruid_475 (If (ApplyComparisonOp EQWithNA (Ref __iruid_474) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_476 (__iruid_477 __iruid_478) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_477) (ApplyBinaryPrimOp Subtract (Ref __iruid_474) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_478) (Cast Int64 (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_477)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_477) (I32 1)) (Ref __iruid_478)) (Recur __iruid_476 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_477) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_478) (Cast Int64 (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_477)))))))) (StreamZip -1 AssumeSameLength (__iruid_479 __iruid_480) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_475))) (StreamMap __iruid_481 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_475)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_481) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_475)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_475))) (ArrayRef -1 (Ref __iruid_466) (Ref __iruid_481)))) (MakeStruct (numberToTake (Ref __iruid_480)) (old (Ref __iruid_479)))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 108: (ToArray (Let __iruid_514 (TailLoop __iruid_515 (__iruid_516) (I32 4) (Let __iruid_517 (CollectDistributedArray __iruid_518 __iruid_519 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_516)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_518))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_520 __iruid_521 (ToStream False (Ref __iruid_517)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_520) (Ref __iruid_521)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_517)))) (Ref __iruid_517) (Recur __iruid_515 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_516) (I32 4)))))) (Let __iruid_522 (ArrayLen (Ref __iruid_514)) (Let __iruid_523 (If (ApplyComparisonOp EQWithNA (Ref __iruid_522) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_524 (__iruid_525 __iruid_526) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_525) (ApplyBinaryPrimOp Subtract (Ref __iruid_522) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_526) (Cast Int64 (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_525)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_525) (I32 1)) (Ref __iruid_526)) (Recur __iruid_524 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_525) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_526) (Cast Int64 (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_525)))))))) (StreamZip -1 AssumeSameLength (__iruid_527 __iruid_528) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_523))) (StreamMap __iruid_529 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_523)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_529) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_523)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_523))) (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_529)))) (MakeStruct (numberToTake (Ref __iruid_528)) (old (Ref __iruid_527)))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_514 (TailLoop __iruid_515 (__iruid_516) (I32 4) (Let __iruid_517 (CollectDistributedArray __iruid_518 __iruid_519 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_516)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_518))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_520 __iruid_521 (ToStream False (Ref __iruid_517)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_520) (Ref __iruid_521)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_517)))) (Ref __iruid_517) (Recur __iruid_515 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_516) (I32 4)))))) (Let __iruid_522 (ArrayLen (Ref __iruid_514)) (Let __iruid_523 (If (ApplyComparisonOp EQWithNA (Ref __iruid_522) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_524 (__iruid_525 __iruid_526) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_525) (ApplyBinaryPrimOp Subtract (Ref __iruid_522) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_526) (Cast Int64 (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_525)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_525) (I32 1)) (Ref __iruid_526)) (Recur __iruid_524 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_525) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_526) (Cast Int64 (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_525)))))))) (StreamZip -1 AssumeSameLength (__iruid_527 __iruid_528) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_523))) (StreamMap __iruid_529 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_523)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_529) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_523)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_523))) (ArrayRef -1 (Ref __iruid_514) (Ref __iruid_529)))) (MakeStruct (numberToTake (Ref __iruid_528)) (old (Ref __iruid_527))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_562 (TailLoop __iruid_563 (__iruid_564) (I32 4) (Let __iruid_565 (CollectDistributedArray __iruid_566 __iruid_567 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_564)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_566))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_568 __iruid_569 (ToStream False (Ref __iruid_565)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_568) (Ref __iruid_569)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_565)))) (Ref __iruid_565) (Recur __iruid_563 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_564) (I32 4)))))) (Let __iruid_570 (ArrayLen (Ref __iruid_562)) (Let __iruid_571 (If (ApplyComparisonOp EQWithNA (Ref __iruid_570) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_572 (__iruid_573 __iruid_574) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_573) (ApplyBinaryPrimOp Subtract (Ref __iruid_570) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_574) (Cast Int64 (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_573)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_573) (I32 1)) (Ref __iruid_574)) (Recur __iruid_572 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_573) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_574) (Cast Int64 (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_573)))))))) (StreamZip -1 AssumeSameLength (__iruid_575 __iruid_576) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_571))) (StreamMap __iruid_577 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_571)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_577) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_571)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_571))) (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_577)))) (MakeStruct (numberToTake (Ref __iruid_576)) (old (Ref __iruid_575))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_562 (TailLoop __iruid_563 (__iruid_564) (I32 4) (Let __iruid_565 (CollectDistributedArray __iruid_566 __iruid_567 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_564)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_566))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_568 __iruid_569 (ToStream False (Ref __iruid_565)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_568) (Ref __iruid_569)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_565)))) (Ref __iruid_565) (Recur __iruid_563 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_564) (I32 4)))))) (Let __iruid_570 (ArrayLen (Ref __iruid_562)) (Let __iruid_571 (If (ApplyComparisonOp EQWithNA (Ref __iruid_570) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_572 (__iruid_573 __iruid_574) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_573) (ApplyBinaryPrimOp Subtract (Ref __iruid_570) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_574) (Cast Int64 (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_573)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_573) (I32 1)) (Ref __iruid_574)) (Recur __iruid_572 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_573) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_574) (Cast Int64 (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_573)))))))) (StreamZip -1 AssumeSameLength (__iruid_575 __iruid_576) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_571))) (StreamMap __iruid_577 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_571)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_577) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_571)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_571))) (ArrayRef -1 (Ref __iruid_562) (Ref __iruid_577)))) (MakeStruct (numberToTake (Ref __iruid_576)) (old (Ref __iruid_575))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_610 (TailLoop __iruid_611 (__iruid_612) (I32 4) (Let __iruid_613 (CollectDistributedArray __iruid_614 __iruid_615 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_612)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_614))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_616 __iruid_617 (ToStream False (Ref __iruid_613)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_616) (Ref __iruid_617)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_613)))) (Ref __iruid_613) (Recur __iruid_611 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_612) (I32 4)))))) (Let __iruid_618 (ArrayLen (Ref __iruid_610)) (Let __iruid_619 (If (ApplyComparisonOp EQWithNA (Ref __iruid_618) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_620 (__iruid_621 __iruid_622) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_621) (ApplyBinaryPrimOp Subtract (Ref __iruid_618) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_622) (Cast Int64 (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_621)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_621) (I32 1)) (Ref __iruid_622)) (Recur __iruid_620 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_621) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_622) (Cast Int64 (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_621)))))))) (StreamZip -1 AssumeSameLength (__iruid_623 __iruid_624) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_619))) (StreamMap __iruid_625 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_619)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_625) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_619)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_619))) (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_625)))) (MakeStruct (numberToTake (Ref __iruid_624)) (old (Ref __iruid_623))))))))) 2022-06-03 16:54:59 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_610 (TailLoop __iruid_611 (__iruid_612) (I32 4) (Let __iruid_613 (CollectDistributedArray __iruid_614 __iruid_615 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_612)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_614))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_616 __iruid_617 (ToStream False (Ref __iruid_613)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_616) (Ref __iruid_617)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_613)))) (Ref __iruid_613) (Recur __iruid_611 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_612) (I32 4)))))) (Let __iruid_618 (ArrayLen (Ref __iruid_610)) (Let __iruid_619 (If (ApplyComparisonOp EQWithNA (Ref __iruid_618) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_620 (__iruid_621 __iruid_622) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_621) (ApplyBinaryPrimOp Subtract (Ref __iruid_618) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_622) (Cast Int64 (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_621)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_621) (I32 1)) (Ref __iruid_622)) (Recur __iruid_620 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_621) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_622) (Cast Int64 (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_621)))))))) (StreamZip -1 AssumeSameLength (__iruid_623 __iruid_624) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_619))) (StreamMap __iruid_625 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_619)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_625) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_619)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_619))) (ArrayRef -1 (Ref __iruid_610) (Ref __iruid_625)))) (MakeStruct (numberToTake (Ref __iruid_624)) (old (Ref __iruid_623))))))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 109: (MakeTuple (0) (ToArray (Let __iruid_658 (TailLoop __iruid_659 (__iruid_660) (I32 4) (Let __iruid_661 (CollectDistributedArray __iruid_662 __iruid_663 (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (Ref __iruid_660)) (Literal Struct{} ) (StreamLen (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_662))) (I32 11)))) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp GTEQ (Cast Int64 (StreamFold __iruid_664 __iruid_665 (ToStream False (Ref __iruid_661)) (I32 0) (ApplyBinaryPrimOp Add (Ref __iruid_664) (Ref __iruid_665)))) (I64 11)) (ApplyComparisonOp LTEQ (I32 150) (ArrayLen (Ref __iruid_661)))) (Ref __iruid_661) (Recur __iruid_659 Array[Int32] (ApplyBinaryPrimOp Multiply (Ref __iruid_660) (I32 4)))))) (Let __iruid_666 (ArrayLen (Ref __iruid_658)) (Let __iruid_667 (If (ApplyComparisonOp EQWithNA (Ref __iruid_666) (I32 0)) (Literal Tuple[Int32,Int64] ) (TailLoop __iruid_668 (__iruid_669 __iruid_670) (I32 0) (I64 11) (If (ApplySpecial -1 lor () Boolean (ApplyComparisonOp EQWithNA (Ref __iruid_669) (ApplyBinaryPrimOp Subtract (Ref __iruid_666) (I32 1))) (ApplyComparisonOp LTEQ (ApplyBinaryPrimOp Subtract (Ref __iruid_670) (Cast Int64 (ArrayRef -1 (Ref __iruid_658) (Ref __iruid_669)))) (I64 0))) (MakeTuple (0 1) (ApplyBinaryPrimOp Add (Ref __iruid_669) (I32 1)) (Ref __iruid_670)) (Recur __iruid_668 Tuple[Int32,Int64] (ApplyBinaryPrimOp Add (Ref __iruid_669) (I32 1)) (ApplyBinaryPrimOp Subtract (Ref __iruid_670) (Cast Int64 (ArrayRef -1 (Ref __iruid_658) (Ref __iruid_669)))))))) (StreamZip -1 AssumeSameLength (__iruid_671 __iruid_672) (StreamTake (ToStream False (Literal Array[Struct{path:String}] )) (GetTupleElement 0 (Ref __iruid_667))) (StreamMap __iruid_673 (StreamRange -1 False (I32 0) (GetTupleElement 0 (Ref __iruid_667)) (I32 1)) (If (ApplyComparisonOp EQWithNA (Ref __iruid_673) (ApplyBinaryPrimOp Subtract (GetTupleElement 0 (Ref __iruid_667)) (I32 1))) (Cast Int32 (GetTupleElement 1 (Ref __iruid_667))) (ArrayRef -1 (Ref __iruid_658) (Ref __iruid_673)))) (MakeStruct (numberToTake (Ref __iruid_672)) (old (Ref __iruid_671))))))))) 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C752HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C752HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C753FSContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C753FSContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C754collect_distributed_array. 2022-06-03 16:55:00 root: INFO: instruction count: 111: __C754collect_distributed_array.apply 2022-06-03 16:55:00 root: INFO: instruction count: 17: __C754collect_distributed_array.apply 2022-06-03 16:55:00 root: INFO: instruction count: 27: __C754collect_distributed_array.__m755DECODE_r_struct_of_r_struct_of_r_binaryENDEND_TO_SBaseStructPointer 2022-06-03 16:55:00 root: INFO: instruction count: 17: __C754collect_distributed_array.__m756INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:55:00 root: INFO: instruction count: 31: __C754collect_distributed_array.__m757INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:00 root: INFO: instruction count: 27: __C754collect_distributed_array.__m759DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:55:00 root: INFO: instruction count: 8: __C754collect_distributed_array.__m760INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:00 root: INFO: instruction count: 156: __C754collect_distributed_array.__m762split_StreamLen 2022-06-03 16:55:00 root: INFO: instruction count: 382: __C754collect_distributed_array.__m767DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryANDo_float64ANDr_struct_of_o_binaryANDo_float64ENDANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolENDANDo_float64ANDo_float64ANDo_float64ANDr_array_of_r_float64END_TO_SBaseStructPointer 2022-06-03 16:55:00 root: INFO: instruction count: 26: __C754collect_distributed_array.__m768INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C754collect_distributed_array.__m769INPLACE_DECODE_r_int32_TO_r_int32 2022-06-03 16:55:00 root: INFO: instruction count: 114: __C754collect_distributed_array.__m770INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string 2022-06-03 16:55:00 root: INFO: instruction count: 31: __C754collect_distributed_array.__m771INPLACE_DECODE_o_binary_TO_o_string 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C754collect_distributed_array.__m772INPLACE_DECODE_o_float64_TO_o_float64 2022-06-03 16:55:00 root: INFO: instruction count: 75: __C754collect_distributed_array.__m773INPLACE_DECODE_r_struct_of_o_binaryANDo_float64END_TO_r_struct_of_o_stringANDo_float64END 2022-06-03 16:55:00 root: INFO: instruction count: 93: __C754collect_distributed_array.__m774INPLACE_DECODE_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_o_interval_of_o_locus 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C754collect_distributed_array.__m775INPLACE_DECODE_r_bool_TO_r_bool 2022-06-03 16:55:00 root: INFO: instruction count: 58: __C754collect_distributed_array.__m776INPLACE_DECODE_r_array_of_r_float64_TO_r_array_of_r_float64 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C754collect_distributed_array.__m777INPLACE_DECODE_r_float64_TO_r_float64 2022-06-03 16:55:00 root: INFO: instruction count: 13: __C754collect_distributed_array.__m781ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32END 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C754collect_distributed_array.__m782ENCODE_SInt32$_TO_r_int32 2022-06-03 16:55:00 root: INFO: instruction count: 9: __C754collect_distributed_array.setPartitionIndex 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C754collect_distributed_array.addPartitionRegion 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C754collect_distributed_array.setPool 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C754collect_distributed_array.addHailClassLoader 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C754collect_distributed_array.addFS 2022-06-03 16:55:00 root: INFO: encoder cache miss (1 hits, 6 misses, 0.143) 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C865HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C865HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C866FSContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C866FSContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C867etypeEncode. 2022-06-03 16:55:00 root: INFO: instruction count: 7: __C867etypeEncode.apply 2022-06-03 16:55:00 root: INFO: instruction count: 41: __C867etypeEncode.__m868ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryENDANDr_struct_of_ENDANDr_struct_of_r_int32ANDr_int64ENDEND 2022-06-03 16:55:00 root: INFO: instruction count: 35: __C867etypeEncode.__m869ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryEND 2022-06-03 16:55:00 root: INFO: instruction count: 13: __C867etypeEncode.__m870ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:55:00 root: INFO: instruction count: 16: __C867etypeEncode.__m871ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:00 root: INFO: instruction count: 1: __C867etypeEncode.__m872ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:55:00 root: INFO: instruction count: 25: __C867etypeEncode.__m873ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32ANDr_int64END 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C867etypeEncode.__m874ENCODE_SInt32$_TO_r_int32 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C867etypeEncode.__m875ENCODE_SInt64$_TO_r_int64 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-22.c.broad-mpg-gnomad.internal:33423 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-49.c.broad-mpg-gnomad.internal:46035 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-7.c.broad-mpg-gnomad.internal:33731 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_7 stored as values in memory (estimated size 17.4 KiB, free 21.7 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-28.c.broad-mpg-gnomad.internal:38115 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-10.c.broad-mpg-gnomad.internal:34193 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-8.c.broad-mpg-gnomad.internal:36667 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-14.c.broad-mpg-gnomad.internal:44979 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-39.c.broad-mpg-gnomad.internal:43981 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-13.c.broad-mpg-gnomad.internal:43607 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-20.c.broad-mpg-gnomad.internal:45189 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_7_piece0 stored as bytes in memory (estimated size 6.2 KiB, free 21.7 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-22.c.broad-mpg-gnomad.internal:39851 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-18.c.broad-mpg-gnomad.internal:35929 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-8.c.broad-mpg-gnomad.internal:44645 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-36.c.broad-mpg-gnomad.internal:40831 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 49.0 B, free: 21.7 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-20.c.broad-mpg-gnomad.internal:40647 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-27.c.broad-mpg-gnomad.internal:35819 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_7_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 6.2 KiB, free: 21.7 GiB) 2022-06-03 16:55:00 SparkContext: INFO: Created broadcast 7 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C732HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C732HailClassLoaderContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C733FSContainer. 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C733FSContainer. 2022-06-03 16:55:00 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-9.c.broad-mpg-gnomad.internal:43647 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C734Compiled. 2022-06-03 16:55:00 root: INFO: instruction count: 31: __C734Compiled.apply 2022-06-03 16:55:00 root: INFO: instruction count: 415: __C734Compiled.__m735split_ToArray 2022-06-03 16:55:00 root: INFO: instruction count: 466: __C734Compiled.__m737split_TailLoop 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C734Compiled.setBackend 2022-06-03 16:55:00 root: INFO: instruction count: 9: __C734Compiled.__m791ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryENDEND 2022-06-03 16:55:00 root: INFO: instruction count: 13: __C734Compiled.__m792ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:55:00 root: INFO: instruction count: 16: __C734Compiled.__m793ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:00 root: INFO: instruction count: 9: __C734Compiled.__m794ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-06-03 16:55:00 root: INFO: instruction count: 1: __C734Compiled.__m795ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C734Compiled.setObjects 2022-06-03 16:55:00 root: INFO: instruction count: 27: __C734Compiled.__m797DECODE_r_struct_of_r_int32END_TO_SBaseStructPointer 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C734Compiled.__m798INPLACE_DECODE_r_int32_TO_r_int32 2022-06-03 16:55:00 root: INFO: instruction count: 70: __C734Compiled.__m802split_StreamFold 2022-06-03 16:55:00 root: INFO: instruction count: 11: __C734Compiled.__m812ord_gteq 2022-06-03 16:55:00 root: INFO: instruction count: 16: __C734Compiled.__m813ord_gteqNonnull 2022-06-03 16:55:00 root: INFO: instruction count: 11: __C734Compiled.__m814ord_lteq 2022-06-03 16:55:00 root: INFO: instruction count: 14: __C734Compiled.__m815ord_lteqNonnull 2022-06-03 16:55:00 root: INFO: instruction count: 11: __C734Compiled.__m823ord_equiv 2022-06-03 16:55:00 root: INFO: instruction count: 14: __C734Compiled.__m824ord_equivNonnull 2022-06-03 16:55:00 root: INFO: instruction count: 225: __C734Compiled.__m826split_TailLoop 2022-06-03 16:55:00 root: INFO: instruction count: 35: __C734Compiled.__m832arrayref_bounds_check 2022-06-03 16:55:00 root: INFO: instruction count: 11: __C734Compiled.__m833ord_lteq 2022-06-03 16:55:00 root: INFO: instruction count: 16: __C734Compiled.__m834ord_lteqNonnull 2022-06-03 16:55:00 root: INFO: instruction count: 9: __C734Compiled.setPartitionIndex 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C734Compiled.addPartitionRegion 2022-06-03 16:55:00 root: INFO: instruction count: 4: __C734Compiled.setPool 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C734Compiled.addHailClassLoader 2022-06-03 16:55:00 root: INFO: instruction count: 3: __C734Compiled.addFS 2022-06-03 16:55:00 root: INFO: instruction count: 69: __C734Compiled.addLiterals 2022-06-03 16:55:00 root: INFO: instruction count: 45: __C734Compiled.__m858DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryENDANDr_struct_of_ENDANDr_struct_of_r_int32ANDr_int64ENDEND_TO_SBaseStructPointer 2022-06-03 16:55:00 root: INFO: instruction count: 58: __C734Compiled.__m859INPLACE_DECODE_r_array_of_r_struct_of_r_binaryEND_TO_r_array_of_r_struct_of_r_stringEND 2022-06-03 16:55:00 root: INFO: instruction count: 17: __C734Compiled.__m860INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:55:00 root: INFO: instruction count: 31: __C734Compiled.__m861INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:00 root: INFO: instruction count: 8: __C734Compiled.__m862INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:00 root: INFO: instruction count: 26: __C734Compiled.__m863INPLACE_DECODE_r_struct_of_r_int32ANDr_int64END_TO_r_tuple_of_r_int32ANDr_int64END 2022-06-03 16:55:00 root: INFO: instruction count: 10: __C734Compiled.__m864INPLACE_DECODE_r_int64_TO_r_int64 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_8 stored as values in memory (estimated size 64.0 B, free 21.7 GiB) 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_8_piece0 stored as bytes in memory (estimated size 49.0 B, free 21.7 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 49.0 B, free: 21.7 GiB) 2022-06-03 16:55:00 SparkContext: INFO: Created broadcast 8 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:00 SparkContext: INFO: Starting job: collect at SparkBackend.scala:321 2022-06-03 16:55:00 DAGScheduler: INFO: Got job 1 (collect at SparkBackend.scala:321) with 4 output partitions 2022-06-03 16:55:00 DAGScheduler: INFO: Final stage: ResultStage 1 (collect at SparkBackend.scala:321) 2022-06-03 16:55:00 DAGScheduler: INFO: Parents of final stage: List() 2022-06-03 16:55:00 DAGScheduler: INFO: Missing parents: List() 2022-06-03 16:55:00 DAGScheduler: INFO: Submitting ResultStage 1 (SparkBackendComputeRDD[1] at RDD at SparkBackend.scala:753), which has no missing parents 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_9 stored as values in memory (estimated size 16.4 KiB, free 21.7 GiB) 2022-06-03 16:55:00 MemoryStore: INFO: Block broadcast_9_piece0 stored as bytes in memory (estimated size 9.2 KiB, free 21.7 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 9.2 KiB, free: 21.7 GiB) 2022-06-03 16:55:00 SparkContext: INFO: Created broadcast 9 from broadcast at DAGScheduler.scala:1388 2022-06-03 16:55:00 DAGScheduler: INFO: Submitting 4 missing tasks from ResultStage 1 (SparkBackendComputeRDD[1] at RDD at SparkBackend.scala:753) (first 15 tasks are for partitions Vector(0, 1, 2, 3)) 2022-06-03 16:55:00 YarnScheduler: INFO: Adding task set 1.0 with 4 tasks resource profile 0 2022-06-03 16:55:00 FairSchedulableBuilder: INFO: Added task set TaskSet_1.0 tasks to pool default 2022-06-03 16:55:00 TaskSetManager: INFO: Starting task 0.0 in stage 1.0 (TID 188) (kc1-w-10.c.broad-mpg-gnomad.internal, executor 18, partition 0, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:55:00 TaskSetManager: INFO: Starting task 1.0 in stage 1.0 (TID 189) (kc1-w-28.c.broad-mpg-gnomad.internal, executor 15, partition 1, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:55:00 TaskSetManager: INFO: Starting task 2.0 in stage 1.0 (TID 190) (kc1-w-13.c.broad-mpg-gnomad.internal, executor 34, partition 2, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:55:00 TaskSetManager: INFO: Starting task 3.0 in stage 1.0 (TID 191) (kc1-w-8.c.broad-mpg-gnomad.internal, executor 38, partition 3, PROCESS_LOCAL, 4428 bytes) taskResourceAssignments Map() 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 9.2 KiB, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 9.2 KiB, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 9.2 KiB, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_9_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 9.2 KiB, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc1-w-28.c.broad-mpg-gnomad.internal:38115 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc1-w-8.c.broad-mpg-gnomad.internal:44645 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc1-w-13.c.broad-mpg-gnomad.internal:43607 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 BlockManagerInfo: INFO: Added broadcast_8_piece0 in memory on kc1-w-10.c.broad-mpg-gnomad.internal:34193 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:00 TaskSetManager: INFO: Finished task 1.0 in stage 1.0 (TID 189) in 135 ms on kc1-w-28.c.broad-mpg-gnomad.internal (executor 15) (1/4) 2022-06-03 16:55:00 TaskSetManager: INFO: Finished task 2.0 in stage 1.0 (TID 190) in 159 ms on kc1-w-13.c.broad-mpg-gnomad.internal (executor 34) (2/4) 2022-06-03 16:55:00 TaskSetManager: INFO: Finished task 0.0 in stage 1.0 (TID 188) in 168 ms on kc1-w-10.c.broad-mpg-gnomad.internal (executor 18) (3/4) 2022-06-03 16:55:00 TaskSetManager: INFO: Finished task 3.0 in stage 1.0 (TID 191) in 204 ms on kc1-w-8.c.broad-mpg-gnomad.internal (executor 38) (4/4) 2022-06-03 16:55:00 YarnScheduler: INFO: Removed TaskSet 1.0, whose tasks have all completed, from pool default 2022-06-03 16:55:00 DAGScheduler: INFO: ResultStage 1 (collect at SparkBackend.scala:321) finished in 0.229 s 2022-06-03 16:55:00 DAGScheduler: INFO: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 2022-06-03 16:55:00 YarnScheduler: INFO: Killing all running tasks in stage 1: Stage finished 2022-06-03 16:55:00 DAGScheduler: INFO: Job 1 finished: collect at SparkBackend.scala:321, took 0.234503 s 2022-06-03 16:55:00 root: INFO: RegionPool: REPORT_THRESHOLD: 256.0K allocated (192.0K blocks / 64.0K chunks), regions.size = 3, 0 current java objects, thread 16: Thread-4 2022-06-03 16:55:00 root: INFO: compiling and evaluating result: TableCollect 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 92: (Let __iruid_360 (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)) (Let __iruid_678 (CollectDistributedArray __iruid_675 __iruid_677 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (MakeStruct (__iruid_360 (Ref __iruid_360))) (Let __iruid_360 (GetField __iruid_360 (Ref __iruid_677)) (ToArray (Let global (Ref __iruid_360) (StreamMap __iruid_676 (StreamTake (Let __iruid_361 (GetField old (Ref __iruid_675)) (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (Ref __iruid_361)))) (GetField numberToTake (Ref __iruid_675))) (Let row (Ref __iruid_676) (Let __iruid_357 (GetField sift (Ref row)) (Let __iruid_358 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref row)) (InsertFields (SelectFields () (Ref __iruid_358)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_358)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_358)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_358)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_357)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_357)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_358)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_358)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_358)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_358)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_358)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_358)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_358)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_358)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_358)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_358))))))))))))) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_679 (ToStream False (Ref __iruid_678)) (ToStream False (Ref __iruid_679))))) (global (Ref __iruid_360))))) 2022-06-03 16:55:00 root: INFO: Prune: MakeStruct: eliminating field '__iruid_360' 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 78: (MakeStruct (rows (ToArray (StreamFlatMap __iruid_699 (ToStream False (CollectDistributedArray __iruid_700 __iruid_701 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_702 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_700)))) (GetField numberToTake (Ref __iruid_700))) (Let __iruid_703 (GetField sift (Ref __iruid_702)) (Let __iruid_704 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_702)) (InsertFields (SelectFields () (Ref __iruid_704)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_704)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_704)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_704)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_703)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_703)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_704)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_704)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_704)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_704)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_704)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_704)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_704)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_704)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_704)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_704))))))))))) (ToStream False (Ref __iruid_699))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)))) 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 78: (MakeStruct (rows (ToArray (StreamFlatMap __iruid_699 (ToStream False (CollectDistributedArray __iruid_700 __iruid_701 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_702 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_700)))) (GetField numberToTake (Ref __iruid_700))) (Let __iruid_703 (GetField sift (Ref __iruid_702)) (Let __iruid_704 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_702)) (InsertFields (SelectFields () (Ref __iruid_704)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_704)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_704)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_704)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_703)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_703)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_704)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_704)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_704)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_704)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_704)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_704)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_704)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_704)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_704)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_704))))))))))) (ToStream False (Ref __iruid_699))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)))) 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 78: (MakeStruct (rows (ToArray (StreamFlatMap __iruid_717 (ToStream False (CollectDistributedArray __iruid_718 __iruid_719 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_720 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_718)))) (GetField numberToTake (Ref __iruid_718))) (Let __iruid_721 (GetField sift (Ref __iruid_720)) (Let __iruid_722 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_720)) (InsertFields (SelectFields () (Ref __iruid_722)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_722)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_722)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_722)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_721)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_721)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_722)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_722)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_722)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_722)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_722)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_722)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_722)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_722)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_722)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_722))))))))))) (ToStream False (Ref __iruid_717))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)))) 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 78: (MakeStruct (rows (ToArray (StreamFlatMap __iruid_717 (ToStream False (CollectDistributedArray __iruid_718 __iruid_719 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_720 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_718)))) (GetField numberToTake (Ref __iruid_718))) (Let __iruid_721 (GetField sift (Ref __iruid_720)) (Let __iruid_722 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_720)) (InsertFields (SelectFields () (Ref __iruid_722)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_722)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_722)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_722)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_721)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_721)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_722)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_722)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_722)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_722)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_722)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_722)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_722)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_722)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_722)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_722))))))))))) (ToStream False (Ref __iruid_717))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)))) 2022-06-03 16:55:00 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 78: (MakeStruct (rows (ToArray (StreamFlatMap __iruid_735 (ToStream False (CollectDistributedArray __iruid_736 __iruid_737 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_738 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_736)))) (GetField numberToTake (Ref __iruid_736))) (Let __iruid_739 (GetField sift (Ref __iruid_738)) (Let __iruid_740 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_738)) (InsertFields (SelectFields () (Ref __iruid_740)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_740)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_740)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_740)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_739)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_739)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_740)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_740)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_740)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_740)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_740)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_740)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_740)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_740)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_740)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_740))))))))))) (ToStream False (Ref __iruid_735))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0)))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_735 (ToStream False (CollectDistributedArray __iruid_736 __iruid_737 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_738 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_736)))) (GetField numberToTake (Ref __iruid_736))) (Let __iruid_739 (GetField sift (Ref __iruid_738)) (Let __iruid_740 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_738)) (InsertFields (SelectFields () (Ref __iruid_740)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_740)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_740)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_740)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_739)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_739)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_740)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_740)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_740)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_740)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_740)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_740)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_740)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_740)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_740)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_740))))))))))) (ToStream False (Ref __iruid_735))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_753 (ToStream False (CollectDistributedArray __iruid_754 __iruid_755 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_756 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_754)))) (GetField numberToTake (Ref __iruid_754))) (Let __iruid_757 (GetField sift (Ref __iruid_756)) (Let __iruid_758 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_756)) (InsertFields (SelectFields () (Ref __iruid_758)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_758)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_758)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_758)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_757)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_757)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_758)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_758)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_758)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_758)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_758)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_758)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_758)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_758)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_758)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_758))))))))))) (ToStream False (Ref __iruid_753))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_753 (ToStream False (CollectDistributedArray __iruid_754 __iruid_755 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_756 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_754)))) (GetField numberToTake (Ref __iruid_754))) (Let __iruid_757 (GetField sift (Ref __iruid_756)) (Let __iruid_758 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_756)) (InsertFields (SelectFields () (Ref __iruid_758)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_758)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_758)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_758)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_757)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_757)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_758)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_758)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_758)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_758)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_758)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_758)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_758)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_758)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_758)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_758))))))))))) (ToStream False (Ref __iruid_753))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_771 (ToStream False (CollectDistributedArray __iruid_772 __iruid_773 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_774 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_772)))) (GetField numberToTake (Ref __iruid_772))) (Let __iruid_775 (GetField sift (Ref __iruid_774)) (Let __iruid_776 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_774)) (InsertFields (SelectFields () (Ref __iruid_776)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_776)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_776)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_776)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_775)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_775)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_776)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_776)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_776)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_776)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_776)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_776)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_776)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_776)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_776)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_776))))))))))) (ToStream False (Ref __iruid_771))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_771 (ToStream False (CollectDistributedArray __iruid_772 __iruid_773 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_774 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_772)))) (GetField numberToTake (Ref __iruid_772))) (Let __iruid_775 (GetField sift (Ref __iruid_774)) (Let __iruid_776 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_774)) (InsertFields (SelectFields () (Ref __iruid_776)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_776)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_776)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_776)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_775)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_775)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_776)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_776)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_776)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_776)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_776)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_776)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_776)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_776)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_776)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_776))))))))))) (ToStream False (Ref __iruid_771))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:00 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 79: (MakeTuple (0) (MakeStruct (rows (ToArray (StreamFlatMap __iruid_789 (ToStream False (CollectDistributedArray __iruid_790 __iruid_791 (ToStream False (Literal Array[Struct{numberToTake:Int32,old:Struct{path:String}}] )) (Literal Struct{} ) (ToArray (StreamMap __iruid_792 (StreamTake (ReadPartition Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{locus:EBaseStruct{contig:+EBinary,position:+EInt32},alleles:EArray[EBinary],polyphen:EFloat64,sift:+EBaseStruct{prediction:EBinary,score:EFloat64},codons:EBinary,most_severe_consequence:EBinary,transcript:EBinary,ref:EBinary,alt:EBinary,interval:EBaseStruct{start:EBaseStruct{contig:+EBinary,position:+EInt32},end:EBaseStruct{contig:+EBinary,position:+EInt32},includesStart:+EBoolean,includesEnd:+EBoolean},oe:EFloat64,misbad:EFloat64,fitted_score:EFloat64,gnomad_scores:+EArray[+EFloat64]}\",\"_vType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],polyphen:Float64,sift:Struct{prediction:String,score:Float64},codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:Interval[Locus(GRCh37)],oe:Float64,misbad:Float64,fitted_score:Float64,gnomad_scores:Array[Float64]}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (GetField path (GetField old (Ref __iruid_790)))) (GetField numberToTake (Ref __iruid_790))) (Let __iruid_793 (GetField sift (Ref __iruid_792)) (Let __iruid_794 (SelectFields (locus alleles polyphen codons most_severe_consequence transcript ref alt interval oe misbad fitted_score gnomad_scores) (Ref __iruid_792)) (InsertFields (SelectFields () (Ref __iruid_794)) None (locus (ApplySpecial 31 showStr () String (GetField locus (Ref __iruid_794)))) (alleles (ApplySpecial 32 showStr () String (GetField alleles (Ref __iruid_794)))) (polyphen (ApplySpecial 33 showStr () String (GetField polyphen (Ref __iruid_794)))) (`sift.prediction` (ApplySpecial 34 showStr () String (GetField prediction (Ref __iruid_793)))) (`sift.score` (ApplySpecial 35 showStr () String (GetField score (Ref __iruid_793)))) (codons (ApplySpecial 36 showStr () String (GetField codons (Ref __iruid_794)))) (most_severe_consequence (ApplySpecial 37 showStr () String (GetField most_severe_consequence (Ref __iruid_794)))) (transcript (ApplySpecial 38 showStr () String (GetField transcript (Ref __iruid_794)))) (ref (ApplySpecial 39 showStr () String (GetField ref (Ref __iruid_794)))) (alt (ApplySpecial 40 showStr () String (GetField alt (Ref __iruid_794)))) (interval (ApplySpecial 41 showStr () String (GetField interval (Ref __iruid_794)))) (oe (ApplySpecial 42 showStr () String (GetField oe (Ref __iruid_794)))) (misbad (ApplySpecial 43 showStr () String (GetField misbad (Ref __iruid_794)))) (fitted_score (ApplySpecial 44 showStr () String (GetField fitted_score (Ref __iruid_794)))) (gnomad_scores (ApplySpecial 45 showStr () String (GetField gnomad_scores (Ref __iruid_794))))))))))) (ToStream False (Ref __iruid_789))))) (global (ArrayRef -1 (ToArray (ReadPartition Struct{} "{\"name\":\"PartitionNativeReader\",\"spec\":{\"name\":\"TypedCodecSpec\",\"_eType\":\"+EBaseStruct{}\",\"_vType\":\"Struct{}\",\"_bufferSpec\":{\"name\":\"LEB128BufferSpec\",\"child\":{\"name\":\"BlockingBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"LZ4FastBlockBufferSpec\",\"blockSize\":32768,\"child\":{\"name\":\"StreamBlockBufferSpec\"}}}}}}" (Str "gs://regio..."))) (I32 0))))) 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C890HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C890HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C891FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C891FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C892collect_distributed_array. 2022-06-03 16:55:01 root: INFO: instruction count: 111: __C892collect_distributed_array.apply 2022-06-03 16:55:01 root: INFO: instruction count: 17: __C892collect_distributed_array.apply 2022-06-03 16:55:01 root: INFO: instruction count: 27: __C892collect_distributed_array.__m893DECODE_r_struct_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDENDEND_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 26: __C892collect_distributed_array.__m894INPLACE_DECODE_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND_TO_r_struct_of_r_int32ANDr_struct_of_r_stringENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 10: __C892collect_distributed_array.__m895INPLACE_DECODE_r_int32_TO_r_int32 2022-06-03 16:55:01 root: INFO: instruction count: 17: __C892collect_distributed_array.__m896INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:55:01 root: INFO: instruction count: 31: __C892collect_distributed_array.__m897INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:01 root: INFO: instruction count: 27: __C892collect_distributed_array.__m899DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 8: __C892collect_distributed_array.__m900INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:01 root: INFO: instruction count: 398: __C892collect_distributed_array.__m902split_ToArray 2022-06-03 16:55:01 root: INFO: instruction count: 382: __C892collect_distributed_array.__m907DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryANDo_float64ANDr_struct_of_o_binaryANDo_float64ENDANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolENDANDo_float64ANDo_float64ANDo_float64ANDr_array_of_r_float64END_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 26: __C892collect_distributed_array.__m908INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus 2022-06-03 16:55:01 root: INFO: instruction count: 114: __C892collect_distributed_array.__m909INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string 2022-06-03 16:55:01 root: INFO: instruction count: 31: __C892collect_distributed_array.__m910INPLACE_DECODE_o_binary_TO_o_string 2022-06-03 16:55:01 root: INFO: instruction count: 10: __C892collect_distributed_array.__m911INPLACE_DECODE_o_float64_TO_o_float64 2022-06-03 16:55:01 root: INFO: instruction count: 75: __C892collect_distributed_array.__m912INPLACE_DECODE_r_struct_of_o_binaryANDo_float64END_TO_r_struct_of_o_stringANDo_float64END 2022-06-03 16:55:01 root: INFO: instruction count: 93: __C892collect_distributed_array.__m913INPLACE_DECODE_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_o_interval_of_o_locus 2022-06-03 16:55:01 root: INFO: instruction count: 10: __C892collect_distributed_array.__m914INPLACE_DECODE_r_bool_TO_r_bool 2022-06-03 16:55:01 root: INFO: instruction count: 58: __C892collect_distributed_array.__m915INPLACE_DECODE_r_array_of_r_float64_TO_r_array_of_r_float64 2022-06-03 16:55:01 root: INFO: instruction count: 10: __C892collect_distributed_array.__m916INPLACE_DECODE_r_float64_TO_r_float64 2022-06-03 16:55:01 root: INFO: instruction count: 422: __C892collect_distributed_array.__m922split_InsertFields 2022-06-03 16:55:01 root: INFO: instruction count: 23: __C892collect_distributed_array.__m927setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m930setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m933setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m936setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m939setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 23: __C892collect_distributed_array.__m942setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 23: __C892collect_distributed_array.__m945setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m948setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m951setup_null 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C892collect_distributed_array.__m970setup_jab 2022-06-03 16:55:01 root: INFO: instruction count: 25: __C892collect_distributed_array.__m975ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 35: __C892collect_distributed_array.__m976ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 181: __C892collect_distributed_array.__m977ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 16: __C892collect_distributed_array.__m978ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:01 root: INFO: instruction count: 9: __C892collect_distributed_array.setPartitionIndex 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C892collect_distributed_array.addPartitionRegion 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C892collect_distributed_array.setPool 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C892collect_distributed_array.addHailClassLoader 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C892collect_distributed_array.addFS 2022-06-03 16:55:01 root: INFO: instruction count: 43: __C892collect_distributed_array.addReferenceGenomes 2022-06-03 16:55:01 root: INFO: instruction count: 521: __C892collect_distributed_array.__m902split_ToArray_region9_9 2022-06-03 16:55:01 root: INFO: instruction count: 94: __C892collect_distributed_array.__m922split_InsertFields_region0_6 2022-06-03 16:55:01 root: INFO: instruction count: 101: __C892collect_distributed_array.__m922split_InsertFields_region102_107 2022-06-03 16:55:01 root: INFO: instruction count: 85: __C892collect_distributed_array.__m922split_InsertFields_region7_15 2022-06-03 16:55:01 root: INFO: instruction count: 78: __C892collect_distributed_array.__m922split_InsertFields_region72_80 2022-06-03 16:55:01 root: INFO: instruction count: 67: __C892collect_distributed_array.__m922split_InsertFields_region65_71 2022-06-03 16:55:01 root: INFO: instruction count: 67: __C892collect_distributed_array.__m922split_InsertFields_region58_64 2022-06-03 16:55:01 root: INFO: instruction count: 67: __C892collect_distributed_array.__m922split_InsertFields_region51_57 2022-06-03 16:55:01 root: INFO: instruction count: 67: __C892collect_distributed_array.__m922split_InsertFields_region44_50 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C983__m902split_ToArraySpills. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C989__m922split_InsertFieldsSpills. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C924RGContainer_GRCh37. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C924RGContainer_GRCh37. 2022-06-03 16:55:01 root: INFO: encoder cache miss (1 hits, 7 misses, 0.125) 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1077HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1077HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1078FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1078FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1079etypeEncode. 2022-06-03 16:55:01 root: INFO: instruction count: 7: __C1079etypeEncode.apply 2022-06-03 16:55:01 root: INFO: instruction count: 45: __C1079etypeEncode.__m1080ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDENDANDr_struct_of_ENDANDr_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 35: __C1079etypeEncode.__m1081ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 21: __C1079etypeEncode.__m1082ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C1079etypeEncode.__m1083ENCODE_SInt32$_TO_r_int32 2022-06-03 16:55:01 root: INFO: instruction count: 13: __C1079etypeEncode.__m1084ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 16: __C1079etypeEncode.__m1085ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:01 root: INFO: instruction count: 1: __C1079etypeEncode.__m1086ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_10 stored as values in memory (estimated size 304.0 B, free 21.7 GiB) 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_10_piece0 stored as bytes in memory (estimated size 203.0 B, free 21.7 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Added broadcast_10_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 203.0 B, free: 21.7 GiB) 2022-06-03 16:55:01 SparkContext: INFO: Created broadcast 10 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C876HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C876HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C877FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C877FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C878Compiled. 2022-06-03 16:55:01 root: INFO: instruction count: 77: __C878Compiled.apply 2022-06-03 16:55:01 root: INFO: instruction count: 571: __C878Compiled.__m879split_ToArray 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C878Compiled.setBackend 2022-06-03 16:55:01 root: INFO: instruction count: 9: __C878Compiled.__m1018ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 21: __C878Compiled.__m1019ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C878Compiled.__m1020ENCODE_SInt32$_TO_r_int32 2022-06-03 16:55:01 root: INFO: instruction count: 13: __C878Compiled.__m1021ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 16: __C878Compiled.__m1022ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:01 root: INFO: instruction count: 9: __C878Compiled.__m1023ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 1: __C878Compiled.__m1024ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C878Compiled.setObjects 2022-06-03 16:55:01 root: INFO: instruction count: 27: __C878Compiled.__m1026DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryENDEND_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 58: __C878Compiled.__m1027INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:01 root: INFO: instruction count: 143: __C878Compiled.__m1028INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:01 root: INFO: instruction count: 31: __C878Compiled.__m1029INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C878Compiled.__m1048setup_jab 2022-06-03 16:55:01 root: INFO: instruction count: 264: __C878Compiled.__m1053split_ToArray 2022-06-03 16:55:01 root: INFO: instruction count: 18: __C878Compiled.__m1059DECODE_r_struct_of_END_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 12: __C878Compiled.__m1062setup_jab 2022-06-03 16:55:01 root: INFO: instruction count: 35: __C878Compiled.__m1067arrayref_bounds_check 2022-06-03 16:55:01 root: INFO: instruction count: 9: __C878Compiled.setPartitionIndex 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C878Compiled.addPartitionRegion 2022-06-03 16:55:01 root: INFO: instruction count: 4: __C878Compiled.setPool 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C878Compiled.addHailClassLoader 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C878Compiled.addFS 2022-06-03 16:55:01 root: INFO: instruction count: 73: __C878Compiled.addLiterals 2022-06-03 16:55:01 root: INFO: instruction count: 45: __C878Compiled.__m1071DECODE_r_struct_of_r_array_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDENDANDr_struct_of_ENDANDr_binaryEND_TO_SBaseStructPointer 2022-06-03 16:55:01 root: INFO: instruction count: 58: __C878Compiled.__m1072INPLACE_DECODE_r_array_of_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND_TO_r_array_of_r_struct_of_r_int32ANDr_struct_of_r_stringENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 26: __C878Compiled.__m1073INPLACE_DECODE_r_struct_of_r_int32ANDr_struct_of_r_binaryENDEND_TO_r_struct_of_r_int32ANDr_struct_of_r_stringENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 10: __C878Compiled.__m1074INPLACE_DECODE_r_int32_TO_r_int32 2022-06-03 16:55:01 root: INFO: instruction count: 17: __C878Compiled.__m1075INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_struct_of_r_stringEND 2022-06-03 16:55:01 root: INFO: instruction count: 8: __C878Compiled.__m1076INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1087__m879split_ToArraySpills. 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_11 stored as values in memory (estimated size 64.0 B, free 21.7 GiB) 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_11_piece0 stored as bytes in memory (estimated size 49.0 B, free 21.7 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Added broadcast_11_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 49.0 B, free: 21.7 GiB) 2022-06-03 16:55:01 SparkContext: INFO: Created broadcast 11 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on kc1-w-16.c.broad-mpg-gnomad.internal:34507 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:01 SparkContext: INFO: Starting job: collect at SparkBackend.scala:321 2022-06-03 16:55:01 DAGScheduler: INFO: Got job 2 (collect at SparkBackend.scala:321) with 1 output partitions 2022-06-03 16:55:01 DAGScheduler: INFO: Final stage: ResultStage 2 (collect at SparkBackend.scala:321) 2022-06-03 16:55:01 DAGScheduler: INFO: Parents of final stage: List() 2022-06-03 16:55:01 DAGScheduler: INFO: Missing parents: List() 2022-06-03 16:55:01 DAGScheduler: INFO: Submitting ResultStage 2 (SparkBackendComputeRDD[2] at RDD at SparkBackend.scala:753), which has no missing parents 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_12 stored as values in memory (estimated size 45.7 KiB, free 21.7 GiB) 2022-06-03 16:55:01 MemoryStore: INFO: Block broadcast_12_piece0 stored as bytes in memory (estimated size 24.1 KiB, free 21.7 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Added broadcast_12_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 24.1 KiB, free: 21.7 GiB) 2022-06-03 16:55:01 SparkContext: INFO: Created broadcast 12 from broadcast at DAGScheduler.scala:1388 2022-06-03 16:55:01 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 2 (SparkBackendComputeRDD[2] at RDD at SparkBackend.scala:753) (first 15 tasks are for partitions Vector(0)) 2022-06-03 16:55:01 YarnScheduler: INFO: Adding task set 2.0 with 1 tasks resource profile 0 2022-06-03 16:55:01 FairSchedulableBuilder: INFO: Added task set TaskSet_2.0 tasks to pool default 2022-06-03 16:55:01 TaskSetManager: INFO: Starting task 0.0 in stage 2.0 (TID 192) (kc1-w-18.c.broad-mpg-gnomad.internal, executor 23, partition 0, PROCESS_LOCAL, 4432 bytes) taskResourceAssignments Map() 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-28.c.broad-mpg-gnomad.internal:38115 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-18.c.broad-mpg-gnomad.internal:35929 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-22.c.broad-mpg-gnomad.internal:39851 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 77.9 KiB, free: 21.7 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-9.c.broad-mpg-gnomad.internal:43647 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-7.c.broad-mpg-gnomad.internal:33731 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-22.c.broad-mpg-gnomad.internal:33423 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-10.c.broad-mpg-gnomad.internal:34193 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-39.c.broad-mpg-gnomad.internal:43981 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-49.c.broad-mpg-gnomad.internal:46035 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-27.c.broad-mpg-gnomad.internal:35819 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-20.c.broad-mpg-gnomad.internal:45189 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-20.c.broad-mpg-gnomad.internal:40647 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-13.c.broad-mpg-gnomad.internal:43607 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-14.c.broad-mpg-gnomad.internal:44979 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-8.c.broad-mpg-gnomad.internal:36667 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-8.c.broad-mpg-gnomad.internal:44645 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-16.c.broad-mpg-gnomad.internal:34507 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Added broadcast_12_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 24.1 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on kc1-w-36.c.broad-mpg-gnomad.internal:40831 in memory (size: 77.9 KiB, free: 5.1 GiB) 2022-06-03 16:55:01 BlockManagerInfo: INFO: Added broadcast_11_piece0 in memory on kc1-w-18.c.broad-mpg-gnomad.internal:35929 (size: 49.0 B, free: 5.1 GiB) 2022-06-03 16:55:01 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 26 2022-06-03 16:55:01 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 26 2022-06-03 16:55:01 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 26. 2022-06-03 16:55:01 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:01 ExecutorAllocationManager: INFO: Executors 26 removed due to idle timeout. 2022-06-03 16:55:01 TaskSetManager: INFO: Finished task 0.0 in stage 2.0 (TID 192) in 604 ms on kc1-w-18.c.broad-mpg-gnomad.internal (executor 23) (1/1) 2022-06-03 16:55:01 YarnScheduler: INFO: Removed TaskSet 2.0, whose tasks have all completed, from pool default 2022-06-03 16:55:01 DAGScheduler: INFO: ResultStage 2 (collect at SparkBackend.scala:321) finished in 0.615 s 2022-06-03 16:55:01 DAGScheduler: INFO: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job 2022-06-03 16:55:01 YarnScheduler: INFO: Killing all running tasks in stage 2: Stage finished 2022-06-03 16:55:01 DAGScheduler: INFO: Job 2 finished: collect at SparkBackend.scala:321, took 0.619221 s 2022-06-03 16:55:01 root: INFO: RegionPool: REPORT_THRESHOLD: 512.0K allocated (384.0K blocks / 128.0K chunks), regions.size = 6, 0 current java objects, thread 16: Thread-4 2022-06-03 16:55:01 root: INFO: RegionPool: REPORT_THRESHOLD: 1.0M allocated (384.0K blocks / 640.0K chunks), regions.size = 6, 0 current java objects, thread 16: Thread-4 2022-06-03 16:55:01 root: INFO: encoder cache miss (1 hits, 8 misses, 0.111) 2022-06-03 16:55:01 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 36, 29 2022-06-03 16:55:01 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 36, 29 2022-06-03 16:55:01 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 36, 29. 2022-06-03 16:55:01 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:01 ExecutorAllocationManager: INFO: Executors 36,29 removed due to idle timeout. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1088HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1088HailClassLoaderContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1089FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1089FSContainer. 2022-06-03 16:55:01 root: INFO: instruction count: 3: __C1090etypeEncode. 2022-06-03 16:55:01 root: INFO: instruction count: 7: __C1090etypeEncode.apply 2022-06-03 16:55:01 root: INFO: instruction count: 33: __C1090etypeEncode.__m1091ENCODE_SBaseStructPointer_TO_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryENDANDr_struct_of_ENDEND 2022-06-03 16:55:01 root: INFO: instruction count: 35: __C1090etypeEncode.__m1092ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 181: __C1090etypeEncode.__m1093ENCODE_SBaseStructPointer_TO_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND 2022-06-03 16:55:01 root: INFO: instruction count: 16: __C1090etypeEncode.__m1094ENCODE_SStringPointer_TO_r_binary 2022-06-03 16:55:01 root: INFO: instruction count: 1: __C1090etypeEncode.__m1095ENCODE_SBaseStructPointer_TO_r_struct_of_END 2022-06-03 16:55:01 root: INFO: took 2.753s 2022-06-03 16:55:01 root: INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2022-06-03 16:55:01 root: INFO: lowering result: EncodedLiteral 2022-06-03 16:55:01 root: INFO: compiling and evaluating result: EncodedLiteral 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 2: (MakeTuple (0) (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:01 root: INFO: encoder cache hit 2022-06-03 16:55:02 MemoryStore: INFO: Block broadcast_13 stored as values in memory (estimated size 419.3 KiB, free 21.7 GiB) 2022-06-03 16:55:02 MemoryStore: INFO: Block broadcast_13_piece0 stored as bytes in memory (estimated size 121.0 KiB, free 21.7 GiB) 2022-06-03 16:55:02 BlockManagerInfo: INFO: Added broadcast_13_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 121.0 KiB, free: 21.7 GiB) 2022-06-03 16:55:02 SparkContext: INFO: Created broadcast 13 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1096HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1096HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1097FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1097FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1098Compiled. 2022-06-03 16:55:02 root: INFO: instruction count: 23: __C1098Compiled.apply 2022-06-03 16:55:02 root: INFO: instruction count: 9: __C1098Compiled.setPartitionIndex 2022-06-03 16:55:02 root: INFO: instruction count: 4: __C1098Compiled.addPartitionRegion 2022-06-03 16:55:02 root: INFO: instruction count: 4: __C1098Compiled.setPool 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1098Compiled.addHailClassLoader 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1098Compiled.addFS 2022-06-03 16:55:02 root: INFO: instruction count: 58: __C1098Compiled.addLiterals 2022-06-03 16:55:02 root: INFO: instruction count: 18: __C1098Compiled.__m1103DECODE_r_struct_of_END_TO_SBaseStructPointer 2022-06-03 16:55:02 root: INFO: instruction count: 36: __C1098Compiled.__m1104DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryENDANDr_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:55:02 root: INFO: instruction count: 58: __C1098Compiled.__m1105INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:02 root: INFO: instruction count: 143: __C1098Compiled.__m1106INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:02 root: INFO: instruction count: 31: __C1098Compiled.__m1107INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:02 root: INFO: instruction count: 8: __C1098Compiled.__m1108INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:02 root: INFO: encoder cache hit 2022-06-03 16:55:02 root: INFO: took 77.542ms 2022-06-03 16:55:02 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 2: (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:02 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 2: (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}})) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 3: (MakeTuple (0) (GetField rows (EncodedLiteral Struct{rows:Array[Struct{locus:String,alleles:String,polyphen:String,`sift.prediction`:String,`sift.score`:String,codons:String,most_severe_consequence:String,transcript:String,ref:String,alt:String,interval:String,oe:String,misbad:String,fitted_score:String,gnomad_scores:String}],global:Struct{}}))) 2022-06-03 16:55:02 root: INFO: encoder cache hit 2022-06-03 16:55:02 MemoryStore: INFO: Block broadcast_14 stored as values in memory (estimated size 419.3 KiB, free 21.7 GiB) 2022-06-03 16:55:02 MemoryStore: INFO: Block broadcast_14_piece0 stored as bytes in memory (estimated size 121.0 KiB, free 21.7 GiB) 2022-06-03 16:55:02 BlockManagerInfo: INFO: Added broadcast_14_piece0 in memory on kc1-m.c.broad-mpg-gnomad.internal:44073 (size: 121.0 KiB, free: 21.7 GiB) 2022-06-03 16:55:02 SparkContext: INFO: Created broadcast 14 from broadcast at SparkBackend.scala:313 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1109HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1109HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1110FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1110FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1111Compiled. 2022-06-03 16:55:02 root: INFO: instruction count: 42: __C1111Compiled.apply 2022-06-03 16:55:02 root: INFO: instruction count: 9: __C1111Compiled.setPartitionIndex 2022-06-03 16:55:02 root: INFO: instruction count: 4: __C1111Compiled.addPartitionRegion 2022-06-03 16:55:02 root: INFO: instruction count: 4: __C1111Compiled.setPool 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1111Compiled.addHailClassLoader 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1111Compiled.addFS 2022-06-03 16:55:02 root: INFO: instruction count: 58: __C1111Compiled.addLiterals 2022-06-03 16:55:02 root: INFO: instruction count: 18: __C1111Compiled.__m1116DECODE_r_struct_of_END_TO_SBaseStructPointer 2022-06-03 16:55:02 root: INFO: instruction count: 36: __C1111Compiled.__m1117DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryENDANDr_struct_of_ENDEND_TO_SBaseStructPointer 2022-06-03 16:55:02 root: INFO: instruction count: 58: __C1111Compiled.__m1118INPLACE_DECODE_r_array_of_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_array_of_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:02 root: INFO: instruction count: 143: __C1111Compiled.__m1119INPLACE_DECODE_r_struct_of_r_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryANDr_binaryEND_TO_r_struct_of_r_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringANDr_stringEND 2022-06-03 16:55:02 root: INFO: instruction count: 31: __C1111Compiled.__m1120INPLACE_DECODE_r_binary_TO_r_string 2022-06-03 16:55:02 root: INFO: instruction count: 8: __C1111Compiled.__m1121INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2022-06-03 16:55:02 root: INFO: RegionPool: REPORT_THRESHOLD: 2.0M allocated (448.0K blocks / 1.6M chunks), regions.size = 6, 0 current java objects, thread 16: Thread-4 2022-06-03 16:55:02 root: INFO: encoder cache miss (4 hits, 9 misses, 0.308) 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1122HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1122HailClassLoaderContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1123FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1123FSContainer. 2022-06-03 16:55:02 root: INFO: instruction count: 3: __C1124etypeEncode. 2022-06-03 16:55:02 root: INFO: instruction count: 19: __C1124etypeEncode.apply 2022-06-03 16:55:02 root: INFO: instruction count: 74: __C1124etypeEncode.__m1125ENCODE_SIndexablePointer_TO_o_array_of_o_struct_of_o_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryEND 2022-06-03 16:55:02 root: INFO: instruction count: 249: __C1124etypeEncode.__m1126ENCODE_SBaseStructPointer_TO_o_struct_of_o_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryANDo_binaryEND 2022-06-03 16:55:02 root: INFO: instruction count: 16: __C1124etypeEncode.__m1127ENCODE_SStringPointer_TO_o_binary 2022-06-03 16:55:02 root: INFO: finished execution of query hail_query_4 2022-06-03 16:55:02 root: INFO: RegionPool: FREE: 2.4M allocated (448.0K blocks / 1.9M chunks), regions.size = 6, 0 current java objects, thread 16: Thread-4 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode total 3.020s self 24.550ms children 2.995s %children 99.19% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR total 56.835ms self 0.021ms children 56.815ms %children 99.96% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation total 56.730ms self 7.245ms children 49.486ms %children 87.23% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 49.486ms self 0.207ms children 49.279ms %children 99.58% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.308ms self 0.308ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.397ms self 1.397ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 10.150ms self 10.150ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 4.037ms self 4.037ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.217ms self 0.217ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.235ms self 4.235ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.483ms self 0.483ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.859ms self 0.859ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.836ms self 0.836ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.470ms self 3.470ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.270ms self 0.270ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 6.208ms self 6.208ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.934ms self 0.934ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.792ms self 1.792ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.065ms self 1.065ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 7.702ms self 7.702ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.361ms self 0.361ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.955ms self 4.955ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, initial IR/Verify total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable total 0.414ms self 0.014ms children 0.400ms %children 96.59% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/LoweringTransformation total 0.303ms self 0.303ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerMatrixToTable/Verify total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable total 53.617ms self 0.024ms children 53.593ms %children 99.96% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 53.475ms self 8.910ms children 44.564ms %children 83.34% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 44.564ms self 0.258ms children 44.306ms %children 99.42% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.436ms self 0.436ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.461ms self 1.461ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.972ms self 0.972ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 4.413ms self 4.413ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.245ms self 0.245ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 5.514ms self 5.514ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 1.036ms self 1.036ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.246ms self 1.246ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.873ms self 0.873ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 4.622ms self 4.622ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.245ms self 0.245ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 5.815ms self 5.815ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.829ms self 0.829ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.487ms self 1.487ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.141ms self 1.141ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 7.379ms self 7.379ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.528ms self 0.528ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 6.065ms self 6.065ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets total 0.520ms self 0.011ms children 0.509ms %children 97.83% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.361ms self 0.361ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LiftRelationalValuesToRelationalLets/Verify total 0.087ms self 0.087ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable total 2.832s self 0.037ms children 2.832s %children 100.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation total 2.831s self 194.071ms children 2.637s %children 93.15% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate total 1.314s self 39.898ms children 1.274s %children 96.96% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR total 226.566ms self 0.059ms children 226.507ms %children 99.97% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/Verify total 0.206ms self 0.206ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation total 226.244ms self 37.489ms children 188.755ms %children 83.43% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 188.755ms self 0.508ms children 188.247ms %children 99.73% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 5.581ms self 5.581ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 6.901ms self 6.901ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 6.675ms self 6.675ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 70.801ms self 70.801ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.541ms self 1.541ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 36.931ms self 36.931ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 4.962ms self 4.962ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.328ms self 1.328ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.877ms self 2.877ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 18.731ms self 18.731ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.016ms self 1.016ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 7.534ms self 7.534ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 2.661ms self 2.661ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.925ms self 0.925ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.943ms self 1.943ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 13.092ms self 13.092ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.471ms self 0.471ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 4.278ms self 4.278ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, initial IR/Verify total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerMatrixToTable total 1.348ms self 0.018ms children 1.330ms %children 98.63% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerMatrixToTable/Verify total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerMatrixToTable/LoweringTransformation total 1.194ms self 1.194ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerMatrixToTable/Verify total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable total 71.518ms self 0.024ms children 71.494ms %children 99.97% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 71.342ms self 11.777ms children 59.565ms %children 83.49% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 59.565ms self 0.211ms children 59.354ms %children 99.65% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.890ms self 0.890ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.616ms self 0.616ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.359ms self 1.359ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 9.576ms self 9.576ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.439ms self 0.439ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 4.724ms self 4.724ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.737ms self 0.737ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.603ms self 0.603ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.488ms self 1.488ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 11.371ms self 11.371ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.458ms self 0.458ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 5.489ms self 5.489ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.927ms self 0.927ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.409ms self 0.409ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 1.378ms self 1.378ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 9.911ms self 9.911ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.873ms self 0.873ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 8.106ms self 8.106ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.111ms self 0.111ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LiftRelationalValuesToRelationalLets total 1.237ms self 0.013ms children 1.223ms %children 98.92% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LiftRelationalValuesToRelationalLets/Verify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LiftRelationalValuesToRelationalLets/LoweringTransformation total 1.095ms self 1.095ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LiftRelationalValuesToRelationalLets/Verify total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerOrInterpretNonCompilable total 0.566ms self 0.012ms children 0.555ms %children 97.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerOrInterpretNonCompilable/Verify total 0.078ms self 0.078ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerOrInterpretNonCompilable/LoweringTransformation total 0.340ms self 0.340ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/LowerOrInterpretNonCompilable/Verify total 0.137ms self 0.137ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable total 60.071ms self 0.031ms children 60.041ms %children 99.95% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.096ms self 0.096ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 59.819ms self 16.097ms children 43.722ms %children 73.09% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 43.722ms self 0.194ms children 43.527ms %children 99.56% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.493ms self 0.493ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.619ms self 0.619ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 1.213ms self 1.213ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 8.217ms self 8.217ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.447ms self 0.447ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 4.902ms self 4.902ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.773ms self 0.773ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.604ms self 0.604ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 1.125ms self 1.125ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 6.553ms self 6.553ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.461ms self 0.461ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 5.870ms self 5.870ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.780ms self 0.780ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.619ms self 0.619ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 1.837ms self 1.837ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 5.346ms self 5.346ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.308ms self 0.308ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 3.359ms self 3.359ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile total 647.743ms self 357.855ms children 289.888ms %children 44.75% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR total 84.580ms self 0.029ms children 84.551ms %children 99.97% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/Verify total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 84.476ms self 11.282ms children 73.194ms %children 86.64% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 73.194ms self 0.403ms children 72.791ms %children 99.45% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.642ms self 0.642ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.598ms self 0.598ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 1.143ms self 1.143ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 8.561ms self 8.561ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.429ms self 0.429ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.742ms self 3.742ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.655ms self 0.655ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.499ms self 0.499ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 15.492ms self 15.492ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 9.693ms self 9.693ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.648ms self 0.648ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 8.424ms self 8.424ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.329ms self 1.329ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.580ms self 0.580ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.844ms self 2.844ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 10.349ms self 10.349ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.727ms self 0.727ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 6.436ms self 6.436ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, initial IR/Verify total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/InlineApplyIR total 0.793ms self 0.024ms children 0.769ms %children 97.03% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/InlineApplyIR/Verify total 0.406ms self 0.406ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/InlineApplyIR/LoweringTransformation total 0.313ms self 0.313ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/InlineApplyIR/Verify total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR total 78.962ms self 0.058ms children 78.904ms %children 99.93% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 78.751ms self 16.171ms children 62.579ms %children 79.46% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 62.579ms self 0.372ms children 62.207ms %children 99.41% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.567ms self 0.567ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.633ms self 0.633ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.341ms self 1.341ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 9.680ms self 9.680ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.499ms self 0.499ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 6.999ms self 6.999ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 1.883ms self 1.883ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.545ms self 0.545ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.703ms self 1.703ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 9.212ms self 9.212ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.373ms self 0.373ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 5.721ms self 5.721ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 1.580ms self 1.580ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.607ms self 0.607ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 1.373ms self 1.373ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 11.267ms self 11.267ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.610ms self 0.610ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 7.614ms self 7.614ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.126ms self 0.126ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/LowerArrayAggsToRunAggs total 32.473ms self 0.051ms children 32.422ms %children 99.84% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/LowerArrayAggsToRunAggs/Verify total 0.230ms self 0.230ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 31.931ms self 31.931ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/LowerArrayAggsToRunAggs/Verify total 0.261ms self 0.261ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 85.042ms self 0.053ms children 84.989ms %children 99.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.162ms self 0.162ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 84.769ms self 18.325ms children 66.444ms %children 78.38% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 66.444ms self 0.360ms children 66.083ms %children 99.46% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.458ms self 1.458ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.811ms self 1.811ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 8.144ms self 8.144ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 8.402ms self 8.402ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.401ms self 0.401ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 6.452ms self 6.452ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.456ms self 1.456ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.813ms self 0.813ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.746ms self 1.746ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 8.876ms self 8.876ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.375ms self 0.375ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 6.149ms self 6.149ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 1.087ms self 1.087ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.762ms self 0.762ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 1.473ms self 1.473ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 11.240ms self 11.240ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.409ms self 0.409ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 5.030ms self 5.030ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/Compile/EmitContext.analyze total 8.038ms self 8.038ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/InitializeCompiledFunction total 6.186ms self 6.186ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/CompileAndEvaluate/RunCompiledFunction total 258.747ms self 258.747ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 51.701ms self 0.038ms children 51.663ms %children 99.93% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 51.602ms self 11.206ms children 40.397ms %children 78.28% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 40.397ms self 0.438ms children 39.958ms %children 98.92% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 1.399ms self 1.399ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 1.117ms self 1.117ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.434ms self 2.434ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 11.105ms self 11.105ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.649ms self 0.649ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 7.281ms self 7.281ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.723ms self 0.723ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.371ms self 0.371ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.569ms self 0.569ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 2.714ms self 2.714ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.210ms self 0.210ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.629ms self 3.629ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.464ms self 0.464ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.399ms self 0.399ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.613ms self 0.613ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 2.759ms self 2.759ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.114ms self 0.114ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.409ms self 3.409ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.584ms self 0.013ms children 0.572ms %children 97.84% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.533ms self 0.533ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 28.103ms self 0.016ms children 28.087ms %children 99.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 28.054ms self 7.701ms children 20.352ms %children 72.55% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 20.352ms self 0.197ms children 20.155ms %children 99.03% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.318ms self 0.318ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.431ms self 0.431ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.599ms self 0.599ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 2.651ms self 2.651ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.116ms self 0.116ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 3.170ms self 3.170ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.499ms self 0.499ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.292ms self 0.292ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.540ms self 0.540ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 2.246ms self 2.246ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.108ms self 0.108ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 2.760ms self 2.760ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.335ms self 0.335ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.301ms self 0.301ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.486ms self 0.486ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 2.212ms self 2.212ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.111ms self 0.111ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 2.980ms self 2.980ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.291ms self 0.008ms children 0.282ms %children 97.08% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.261ms self 0.261ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.369ms self 0.007ms children 0.362ms %children 98.05% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.250ms self 0.250ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.102ms self 0.102ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 28.436ms self 0.016ms children 28.419ms %children 99.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 28.386ms self 6.286ms children 22.099ms %children 77.85% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 22.099ms self 0.122ms children 21.977ms %children 99.45% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.391ms self 0.391ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.410ms self 0.410ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.569ms self 0.569ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 3.831ms self 3.831ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.101ms self 0.101ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 2.809ms self 2.809ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.392ms self 0.392ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.243ms self 0.243ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.503ms self 0.503ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 2.075ms self 2.075ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.097ms self 0.097ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 2.739ms self 2.739ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.343ms self 0.343ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.238ms self 0.238ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.478ms self 0.478ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 2.811ms self 2.811ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 3.828ms self 3.828ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 432.606ms self 325.201ms children 107.406ms %children 24.83% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 27.131ms self 0.014ms children 27.117ms %children 99.95% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 27.081ms self 6.285ms children 20.795ms %children 76.79% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 20.795ms self 0.103ms children 20.692ms %children 99.51% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.376ms self 0.376ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.376ms self 0.376ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.667ms self 0.667ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 2.572ms self 2.572ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.170ms self 0.170ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.718ms self 3.718ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.382ms self 0.382ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.273ms self 0.273ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.486ms self 0.486ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.950ms self 1.950ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.518ms self 2.518ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.278ms self 0.278ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.239ms self 0.239ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.421ms self 0.421ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 3.379ms self 3.379ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 2.705ms self 2.705ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.220ms self 0.008ms children 0.212ms %children 96.20% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.150ms self 0.150ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 30.023ms self 0.018ms children 30.005ms %children 99.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 29.973ms self 6.239ms children 23.734ms %children 79.18% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 23.734ms self 0.137ms children 23.597ms %children 99.42% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.219ms self 0.219ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.297ms self 0.297ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.491ms self 0.491ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 2.098ms self 2.098ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.630ms self 2.630ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.746ms self 0.746ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.452ms self 0.452ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.521ms self 0.521ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 5.697ms self 5.697ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.142ms self 0.142ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 3.418ms self 3.418ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.369ms self 0.369ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.279ms self 0.279ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.559ms self 0.559ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 2.499ms self 2.499ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.109ms self 0.109ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 2.976ms self 2.976ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 8.674ms self 0.018ms children 8.656ms %children 99.79% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 8.450ms self 8.450ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.146ms self 0.146ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 34.098ms self 0.018ms children 34.080ms %children 99.95% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 34.045ms self 7.277ms children 26.768ms %children 78.63% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 26.768ms self 0.150ms children 26.618ms %children 99.44% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.371ms self 0.371ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.408ms self 0.408ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.668ms self 0.668ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 2.784ms self 2.784ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 3.204ms self 3.204ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.389ms self 0.389ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.275ms self 0.275ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.503ms self 0.503ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 3.367ms self 3.367ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 5.402ms self 5.402ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.451ms self 0.451ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.317ms self 0.317ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.639ms self 0.639ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 3.167ms self 3.167ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.221ms self 0.221ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 4.137ms self 4.137ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 7.259ms self 7.259ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 5.397ms self 5.397ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 702.253ms self 702.253ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR total 1.470ms self 0.020ms children 1.450ms %children 98.64% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation total 1.433ms self 0.780ms children 0.653ms %children 45.56% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 0.653ms self 0.036ms children 0.617ms %children 94.42% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.152ms self 0.152ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.217ms self 0.217ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.102ms self 0.102ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, initial IR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable total 0.056ms self 0.008ms children 0.048ms %children 85.49% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/LoweringTransformation total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable total 0.760ms self 0.011ms children 0.749ms %children 98.51% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 0.743ms self 0.330ms children 0.413ms %children 55.63% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 0.413ms self 0.025ms children 0.388ms %children 93.95% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.194ms self 0.194ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets total 0.040ms self 0.007ms children 0.033ms %children 83.58% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/LoweringTransformation total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LiftRelationalValuesToRelationalLets/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable total 0.037ms self 0.006ms children 0.031ms %children 84.09% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/LoweringTransformation total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/LowerOrInterpretNonCompilable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable total 0.817ms self 0.012ms children 0.805ms %children 98.48% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.752ms self 0.379ms children 0.372ms %children 49.57% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.372ms self 0.023ms children 0.349ms %children 93.72% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.173ms self 0.173ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.026ms self 0.026ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile total 62.031ms self 46.318ms children 15.713ms %children 25.33% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR total 1.965ms self 0.014ms children 1.951ms %children 99.31% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 1.943ms self 0.718ms children 1.225ms %children 63.05% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 1.225ms self 0.040ms children 1.185ms %children 96.70% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.407ms self 0.407ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.358ms self 0.358ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.269ms self 0.269ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, initial IR/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR total 0.076ms self 0.016ms children 0.059ms %children 78.53% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/LoweringTransformation total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/InlineApplyIR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR total 2.509ms self 0.022ms children 2.487ms %children 99.11% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 2.473ms self 0.800ms children 1.673ms %children 67.63% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 1.673ms self 0.060ms children 1.613ms %children 96.44% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.721ms self 0.721ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.145ms self 0.145ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.633ms self 0.633ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs total 3.085ms self 0.053ms children 3.032ms %children 98.29% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 2.908ms self 2.908ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/LowerArrayAggsToRunAggs/Verify total 0.092ms self 0.092ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 5.460ms self 0.040ms children 5.420ms %children 99.28% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 5.384ms self 2.821ms children 2.563ms %children 47.61% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 2.563ms self 0.108ms children 2.456ms %children 95.80% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.359ms self 0.359ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.228ms self 0.228ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.244ms self 0.244ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.916ms self 0.916ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.115ms self 0.115ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.593ms self 0.593ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.029ms self 0.029ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/Compile/EmitContext.analyze total 2.619ms self 2.619ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/InitializeCompiledFunction total 8.508ms self 8.508ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/LoweringTransformation/RunCompiledFunction total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/LowerOrInterpretNonCompilable/Verify total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable total 4.150ms self 0.031ms children 4.119ms %children 99.26% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 4.105ms self 0.770ms children 3.336ms %children 81.25% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 3.336ms self 0.053ms children 3.282ms %children 98.40% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.134ms self 0.134ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.086ms self 0.086ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 2.578ms self 2.578ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.348ms self 0.348ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile total 39.383ms self 30.743ms children 8.641ms %children 21.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR total 2.179ms self 0.021ms children 2.158ms %children 99.04% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 2.144ms self 0.911ms children 1.233ms %children 57.53% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 1.233ms self 0.052ms children 1.181ms %children 95.79% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.212ms self 0.212ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.093ms self 0.093ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.371ms self 0.371ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.354ms self 0.354ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR total 0.102ms self 0.029ms children 0.074ms %children 72.02% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/LoweringTransformation total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/InlineApplyIR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR total 1.665ms self 0.012ms children 1.653ms %children 99.28% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 1.645ms self 0.633ms children 1.012ms %children 61.52% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 1.012ms self 0.041ms children 0.971ms %children 95.94% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.114ms self 0.114ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.082ms self 0.082ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.365ms self 0.365ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.320ms self 0.320ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs total 1.196ms self 0.010ms children 1.186ms %children 99.17% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 1.164ms self 1.164ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/LowerArrayAggsToRunAggs/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 2.160ms self 0.045ms children 2.114ms %children 97.90% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 2.104ms self 0.834ms children 1.270ms %children 60.34% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 1.270ms self 0.043ms children 1.226ms %children 96.57% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.102ms self 0.102ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.199ms self 0.199ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.439ms self 0.439ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.376ms self 0.376ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/Compile/EmitContext.analyze total 1.339ms self 1.339ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/InitializeCompiledFunction total 8.788ms self 8.788ms children 0.000ms %children 0.00% 2022-06-03 16:55:02 root: INFO: timing SparkBackend.executeEncode/RunCompiledFunction total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2022-06-03 16:55:03 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 1 2022-06-03 16:55:03 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 1 2022-06-03 16:55:03 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 1. 2022-06-03 16:55:03 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:03 ExecutorAllocationManager: INFO: Executors 1 removed due to idle timeout. 2022-06-03 16:55:04 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 26. 2022-06-03 16:55:04 DAGScheduler: INFO: Executor lost: 26 (epoch 0) 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Trying to remove executor 26 from BlockManagerMaster. 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(26, kc1-w-7.c.broad-mpg-gnomad.internal, 33731, None) 2022-06-03 16:55:04 BlockManagerMaster: INFO: Removed 26 successfully in removeExecutor 2022-06-03 16:55:04 YarnScheduler: INFO: Executor 26 on kc1-w-7.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:04 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 29. 2022-06-03 16:55:04 DAGScheduler: INFO: Executor lost: 29 (epoch 0) 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Trying to remove executor 29 from BlockManagerMaster. 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(29, kc1-w-27.c.broad-mpg-gnomad.internal, 35819, None) 2022-06-03 16:55:04 BlockManagerMaster: INFO: Removed 29 successfully in removeExecutor 2022-06-03 16:55:04 YarnScheduler: INFO: Executor 29 on kc1-w-27.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:04 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 36. 2022-06-03 16:55:04 DAGScheduler: INFO: Executor lost: 36 (epoch 0) 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Trying to remove executor 36 from BlockManagerMaster. 2022-06-03 16:55:04 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(36, kc1-w-20.c.broad-mpg-gnomad.internal, 45189, None) 2022-06-03 16:55:04 BlockManagerMaster: INFO: Removed 36 successfully in removeExecutor 2022-06-03 16:55:04 YarnScheduler: INFO: Executor 36 on kc1-w-20.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:05 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 1. 2022-06-03 16:55:05 DAGScheduler: INFO: Executor lost: 1 (epoch 0) 2022-06-03 16:55:05 BlockManagerMasterEndpoint: INFO: Trying to remove executor 1 from BlockManagerMaster. 2022-06-03 16:55:05 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(1, kc1-w-9.c.broad-mpg-gnomad.internal, 43647, None) 2022-06-03 16:55:05 YarnScheduler: INFO: Executor 1 on kc1-w-9.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:05 BlockManagerMaster: INFO: Removed 1 successfully in removeExecutor 2022-06-03 16:55:10 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 30 2022-06-03 16:55:10 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 30 2022-06-03 16:55:10 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 30. 2022-06-03 16:55:10 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:10 ExecutorAllocationManager: INFO: Executors 30 removed due to idle timeout. 2022-06-03 16:55:11 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 30. 2022-06-03 16:55:11 DAGScheduler: INFO: Executor lost: 30 (epoch 0) 2022-06-03 16:55:11 BlockManagerMasterEndpoint: INFO: Trying to remove executor 30 from BlockManagerMaster. 2022-06-03 16:55:11 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(30, kc1-w-8.c.broad-mpg-gnomad.internal, 36667, None) 2022-06-03 16:55:11 BlockManagerMaster: INFO: Removed 30 successfully in removeExecutor 2022-06-03 16:55:11 YarnScheduler: INFO: Executor 30 on kc1-w-8.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:19 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 24 2022-06-03 16:55:19 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 24 2022-06-03 16:55:19 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 24. 2022-06-03 16:55:19 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:19 ExecutorAllocationManager: INFO: Executors 24 removed due to idle timeout. 2022-06-03 16:55:22 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 27 2022-06-03 16:55:22 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 27 2022-06-03 16:55:22 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 27. 2022-06-03 16:55:22 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:22 ExecutorAllocationManager: INFO: Executors 27 removed due to idle timeout. 2022-06-03 16:55:23 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 31 2022-06-03 16:55:23 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 31 2022-06-03 16:55:23 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 31. 2022-06-03 16:55:23 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:23 ExecutorAllocationManager: INFO: Executors 31 removed due to idle timeout. 2022-06-03 16:55:23 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 24. 2022-06-03 16:55:23 DAGScheduler: INFO: Executor lost: 24 (epoch 0) 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Trying to remove executor 24 from BlockManagerMaster. 2022-06-03 16:55:23 YarnScheduler: INFO: Executor 24 on kc1-w-16.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(24, kc1-w-16.c.broad-mpg-gnomad.internal, 34507, None) 2022-06-03 16:55:23 BlockManagerMaster: INFO: Removed 24 successfully in removeExecutor 2022-06-03 16:55:23 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 31. 2022-06-03 16:55:23 DAGScheduler: INFO: Executor lost: 31 (epoch 0) 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Trying to remove executor 31 from BlockManagerMaster. 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(31, kc1-w-49.c.broad-mpg-gnomad.internal, 46035, None) 2022-06-03 16:55:23 BlockManagerMaster: INFO: Removed 31 successfully in removeExecutor 2022-06-03 16:55:23 YarnScheduler: INFO: Executor 31 on kc1-w-49.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:23 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 27. 2022-06-03 16:55:23 DAGScheduler: INFO: Executor lost: 27 (epoch 0) 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Trying to remove executor 27 from BlockManagerMaster. 2022-06-03 16:55:23 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(27, kc1-w-20.c.broad-mpg-gnomad.internal, 40647, None) 2022-06-03 16:55:23 YarnScheduler: INFO: Executor 27 on kc1-w-20.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:23 BlockManagerMaster: INFO: Removed 27 successfully in removeExecutor 2022-06-03 16:55:26 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 8 2022-06-03 16:55:26 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 8 2022-06-03 16:55:26 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 8. 2022-06-03 16:55:26 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:26 ExecutorAllocationManager: INFO: Executors 8 removed due to idle timeout. 2022-06-03 16:55:27 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 8. 2022-06-03 16:55:27 DAGScheduler: INFO: Executor lost: 8 (epoch 0) 2022-06-03 16:55:27 BlockManagerMasterEndpoint: INFO: Trying to remove executor 8 from BlockManagerMaster. 2022-06-03 16:55:27 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(8, kc1-w-22.c.broad-mpg-gnomad.internal, 33423, None) 2022-06-03 16:55:27 BlockManagerMaster: INFO: Removed 8 successfully in removeExecutor 2022-06-03 16:55:27 YarnScheduler: INFO: Executor 8 on kc1-w-22.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:45 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 35 2022-06-03 16:55:45 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 35 2022-06-03 16:55:45 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 35. 2022-06-03 16:55:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:45 ExecutorAllocationManager: INFO: Executors 35 removed due to idle timeout. 2022-06-03 16:55:45 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 4 2022-06-03 16:55:45 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 4 2022-06-03 16:55:45 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 4. 2022-06-03 16:55:45 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:45 ExecutorAllocationManager: INFO: Executors 4 removed due to idle timeout. 2022-06-03 16:55:46 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 35. 2022-06-03 16:55:46 DAGScheduler: INFO: Executor lost: 35 (epoch 0) 2022-06-03 16:55:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 35 from BlockManagerMaster. 2022-06-03 16:55:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(35, kc1-w-39.c.broad-mpg-gnomad.internal, 43981, None) 2022-06-03 16:55:46 BlockManagerMaster: INFO: Removed 35 successfully in removeExecutor 2022-06-03 16:55:46 YarnScheduler: INFO: Executor 35 on kc1-w-39.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:46 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 4. 2022-06-03 16:55:46 DAGScheduler: INFO: Executor lost: 4 (epoch 0) 2022-06-03 16:55:46 BlockManagerMasterEndpoint: INFO: Trying to remove executor 4 from BlockManagerMaster. 2022-06-03 16:55:46 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(4, kc1-w-22.c.broad-mpg-gnomad.internal, 39851, None) 2022-06-03 16:55:46 BlockManagerMaster: INFO: Removed 4 successfully in removeExecutor 2022-06-03 16:55:46 YarnScheduler: INFO: Executor 4 on kc1-w-22.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:52 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 12 2022-06-03 16:55:52 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 12 2022-06-03 16:55:52 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 12. 2022-06-03 16:55:52 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:52 ExecutorAllocationManager: INFO: Executors 12 removed due to idle timeout. 2022-06-03 16:55:52 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 25 2022-06-03 16:55:52 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 25 2022-06-03 16:55:52 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 25. 2022-06-03 16:55:52 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:55:52 ExecutorAllocationManager: INFO: Executors 25 removed due to idle timeout. 2022-06-03 16:55:52 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 25. 2022-06-03 16:55:52 DAGScheduler: INFO: Executor lost: 25 (epoch 0) 2022-06-03 16:55:52 BlockManagerMasterEndpoint: INFO: Trying to remove executor 25 from BlockManagerMaster. 2022-06-03 16:55:52 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(25, kc1-w-14.c.broad-mpg-gnomad.internal, 44979, None) 2022-06-03 16:55:52 YarnScheduler: INFO: Executor 25 on kc1-w-14.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:55:52 BlockManagerMaster: INFO: Removed 25 successfully in removeExecutor 2022-06-03 16:55:53 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 12. 2022-06-03 16:55:53 DAGScheduler: INFO: Executor lost: 12 (epoch 0) 2022-06-03 16:55:53 BlockManagerMasterEndpoint: INFO: Trying to remove executor 12 from BlockManagerMaster. 2022-06-03 16:55:53 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(12, kc1-w-36.c.broad-mpg-gnomad.internal, 40831, None) 2022-06-03 16:55:53 BlockManagerMaster: INFO: Removed 12 successfully in removeExecutor 2022-06-03 16:55:53 YarnScheduler: INFO: Executor 12 on kc1-w-36.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:56:00 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 15 2022-06-03 16:56:00 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 15 2022-06-03 16:56:00 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 15. 2022-06-03 16:56:00 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:56:00 ExecutorAllocationManager: INFO: Executors 15 removed due to idle timeout. 2022-06-03 16:56:00 YarnClientSchedulerBackend: INFO: Requesting to kill executor(s) 18, 38, 34 2022-06-03 16:56:00 YarnClientSchedulerBackend: INFO: Actual list of executor(s) to be killed is 18, 38, 34 2022-06-03 16:56:00 ApplicationMaster$AMEndpoint: INFO: Driver requested to kill executor(s) 18, 38, 34. 2022-06-03 16:56:00 YarnAllocator: INFO: Resource profile 0 doesn't exist, adding it 2022-06-03 16:56:00 ExecutorAllocationManager: INFO: Executors 18,38,34 removed due to idle timeout. 2022-06-03 16:56:03 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 34. 2022-06-03 16:56:03 DAGScheduler: INFO: Executor lost: 34 (epoch 0) 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Trying to remove executor 34 from BlockManagerMaster. 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(34, kc1-w-13.c.broad-mpg-gnomad.internal, 43607, None) 2022-06-03 16:56:03 BlockManagerMaster: INFO: Removed 34 successfully in removeExecutor 2022-06-03 16:56:03 YarnScheduler: INFO: Executor 34 on kc1-w-13.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:56:03 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 18. 2022-06-03 16:56:03 DAGScheduler: INFO: Executor lost: 18 (epoch 0) 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Trying to remove executor 18 from BlockManagerMaster. 2022-06-03 16:56:03 YarnScheduler: INFO: Executor 18 on kc1-w-10.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(18, kc1-w-10.c.broad-mpg-gnomad.internal, 34193, None) 2022-06-03 16:56:03 BlockManagerMaster: INFO: Removed 18 successfully in removeExecutor 2022-06-03 16:56:03 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 38. 2022-06-03 16:56:03 DAGScheduler: INFO: Executor lost: 38 (epoch 0) 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Trying to remove executor 38 from BlockManagerMaster. 2022-06-03 16:56:03 YarnScheduler: INFO: Executor 38 on kc1-w-8.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:56:03 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(38, kc1-w-8.c.broad-mpg-gnomad.internal, 44645, None) 2022-06-03 16:56:03 BlockManagerMaster: INFO: Removed 38 successfully in removeExecutor 2022-06-03 16:56:04 YarnSchedulerBackend$YarnDriverEndpoint: INFO: Disabling executor 15. 2022-06-03 16:56:04 DAGScheduler: INFO: Executor lost: 15 (epoch 0) 2022-06-03 16:56:04 BlockManagerMasterEndpoint: INFO: Trying to remove executor 15 from BlockManagerMaster. 2022-06-03 16:56:04 BlockManagerMasterEndpoint: INFO: Removing block manager BlockManagerId(15, kc1-w-28.c.broad-mpg-gnomad.internal, 38115, None) 2022-06-03 16:56:04 YarnScheduler: INFO: Executor 15 on kc1-w-28.c.broad-mpg-gnomad.internal killed by driver. 2022-06-03 16:56:04 BlockManagerMaster: INFO: Removed 15 successfully in removeExecutor 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_12_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 24.1 KiB, free: 21.7 GiB) 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_12_piece0 on kc1-w-18.c.broad-mpg-gnomad.internal:35929 in memory (size: 24.1 KiB, free: 5.1 GiB) 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_9_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 9.2 KiB, free: 21.7 GiB) 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_11_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 49.0 B, free: 21.7 GiB) 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_11_piece0 on kc1-w-18.c.broad-mpg-gnomad.internal:35929 in memory (size: 49.0 B, free: 5.1 GiB) 2022-06-03 17:07:25 BlockManagerInfo: INFO: Removed broadcast_8_piece0 on kc1-m.c.broad-mpg-gnomad.internal:44073 in memory (size: 49.0 B, free: 21.7 GiB) 2022-06-03 17:29:17 Hail: INFO: copying log to 'gs://gnomad-tmp/kc/repartition_test.log'...