2021-09-23 14:27:20 Hail: INFO: Running Hail version 0.2.61-3c86d3ba497a 2021-09-23 14:27:20 CoarseGrainedSchedulerBackend$DriverEndpoint: INFO: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.0.3.1:59550) with ID 0 2021-09-23 14:27:20 SharedState: INFO: loading hive config file: file:/cluster/spark/conf/hive-site.xml 2021-09-23 14:27:20 SharedState: INFO: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/cluster/spark/spark-warehouse'). 2021-09-23 14:27:20 SharedState: INFO: Warehouse path is '/cluster/spark/spark-warehouse'. 2021-09-23 14:27:20 BlockManagerMasterEndpoint: INFO: Registering block manager ip-10-60-146-200.eu-west-2.compute.internal:44000 with 11.5 GB RAM, BlockManagerId(0, ip-10-60-146-200.eu-west-2.compute.internal, 44000, None) 2021-09-23 14:27:20 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@cc976f5{/SQL,null,AVAILABLE,@Spark} 2021-09-23 14:27:20 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@1877475d{/SQL/json,null,AVAILABLE,@Spark} 2021-09-23 14:27:20 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@32853993{/SQL/execution,null,AVAILABLE,@Spark} 2021-09-23 14:27:20 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@7e532a3{/SQL/execution/json,null,AVAILABLE,@Spark} 2021-09-23 14:27:20 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@4ca4b356{/static/sql,null,AVAILABLE,@Spark} 2021-09-23 14:27:20 StateStoreCoordinatorRef: INFO: Registered StateStoreCoordinator endpoint 2021-09-23 14:27:20 SparkSession$Builder: WARN: Using an existing SparkSession; some configuration may not take effect. 2021-09-23 14:27:21 root: INFO: RegionPool: initialized for thread 24: Thread-4 2021-09-23 14:27:21 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 1699.1 KB, free 11.5 GB) 2021-09-23 14:27:21 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 123.7 KB, free 11.5 GB) 2021-09-23 14:27:21 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 123.7 KB, free: 11.5 GB) 2021-09-23 14:27:21 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:250 2021-09-23 14:27:21 root: INFO: timing SparkBackend.parse_value_ir total 606.688ms self 606.688ms children 0.000ms %children 0.00% 2021-09-23 14:27:21 root: INFO: starting execution of query hail_query_1 of initial size 18 2021-09-23 14:27:21 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_3 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_3)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_3)))) (Apply includesStart () Boolean (Ref __uid_3)) (Apply includesEnd () Boolean (Ref __uid_3))))) 2021-09-23 14:27:21 root: INFO: instruction count: 3: __C1Compiled. 2021-09-23 14:27:21 root: INFO: instruction count: 319: __C1Compiled.apply 2021-09-23 14:27:21 root: INFO: instruction count: 10: __C1Compiled.__m15addIRIntermediate 2021-09-23 14:27:21 root: INFO: instruction count: 9: __C1Compiled.setPartitionIndex 2021-09-23 14:27:21 root: INFO: instruction count: 4: __C1Compiled.addPartitionRegion 2021-09-23 14:27:21 root: INFO: instruction count: 3: __C4RGContainer_GRCh38. 2021-09-23 14:27:21 root: INFO: instruction count: 15: __C4RGContainer_GRCh38. 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_1 stored as values in memory (estimated size 1699.1 KB, free 11.5 GB) 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_1_piece0 stored as bytes in memory (estimated size 123.7 KB, free 11.5 GB) 2021-09-23 14:27:22 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 123.7 KB, free: 11.5 GB) 2021-09-23 14:27:22 SparkContext: INFO: Created broadcast 1 from broadcast at SparkBackend.scala:250 2021-09-23 14:27:22 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_2 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_2)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_2)))) (Apply includesStart () Boolean (Ref __iruid_2)) (Apply includesEnd () Boolean (Ref __iruid_2))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_2 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_2)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_2)))) (Apply includesStart () Boolean (Ref __iruid_2)) (Apply includesEnd () Boolean (Ref __iruid_2))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_5 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_5)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_5)))) (Apply includesStart () Boolean (Ref __iruid_5)) (Apply includesEnd () Boolean (Ref __iruid_5))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_5 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_5)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_5)))) (Apply includesStart () Boolean (Ref __iruid_5)) (Apply includesEnd () Boolean (Ref __iruid_5))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_8 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_8)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_8)))) (Apply includesStart () Boolean (Ref __iruid_8)) (Apply includesEnd () Boolean (Ref __iruid_8))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, initial IR: before: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_8 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_8)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_8)))) (Apply includesStart () Boolean (Ref __iruid_8)) (Apply includesEnd () Boolean (Ref __iruid_8)))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, initial IR: after: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_11 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_11)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_11)))) (Apply includesStart () Boolean (Ref __iruid_11)) (Apply includesEnd () Boolean (Ref __iruid_11)))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: before: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_11 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_11)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_11)))) (Apply includesStart () Boolean (Ref __iruid_11)) (Apply includesEnd () Boolean (Ref __iruid_11)))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, after InlineApplyIR: after: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_14 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_14)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_14)))) (Apply includesStart () Boolean (Ref __iruid_14)) (Apply includesEnd () Boolean (Ref __iruid_14)))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: before: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_14 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_14)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_14)))) (Apply includesStart () Boolean (Ref __iruid_14)) (Apply includesEnd () Boolean (Ref __iruid_14)))))) 2021-09-23 14:27:22 root: INFO: optimize optimize: compileLowerer, after LowerArrayAggsToRunAggs: after: IR size 16: (MakeTuple (0) (ToArray (StreamMap __iruid_17 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_17)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_17)))) (Apply includesStart () Boolean (Ref __iruid_17)) (Apply includesEnd () Boolean (Ref __iruid_17)))))) 2021-09-23 14:27:22 root: INFO: encoder cache miss (0 hits, 1 misses, 0.000) 2021-09-23 14:27:22 root: INFO: instruction count: 3: __C57etypeEncode. 2021-09-23 14:27:22 root: INFO: instruction count: 5: __C57etypeEncode.apply 2021-09-23 14:27:22 root: INFO: instruction count: 13: __C57etypeEncode.__m58ENCODE_r_tuple_of_r_array_of_r_interval_of_r_locusEND_TO_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolENDEND 2021-09-23 14:27:22 root: INFO: instruction count: 49: __C57etypeEncode.__m59ENCODE_r_array_of_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDANDr_tuple_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND 2021-09-23 14:27:22 root: INFO: instruction count: 35: __C57etypeEncode.__m60ENCODE_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDANDr_tuple_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND 2021-09-23 14:27:22 root: INFO: instruction count: 23: __C57etypeEncode.__m61ENCODE_r_tuple_of_r_binaryANDr_int32END_TO_r_struct_of_r_binaryANDr_int32END 2021-09-23 14:27:22 root: INFO: instruction count: 16: __C57etypeEncode.__m62ENCODE_r_binary_TO_r_binary 2021-09-23 14:27:22 root: INFO: instruction count: 4: __C57etypeEncode.__m63ENCODE_r_int32_TO_r_int32 2021-09-23 14:27:22 root: INFO: instruction count: 4: __C57etypeEncode.__m64ENCODE_r_bool_TO_r_bool 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_2 stored as values in memory (estimated size 104.0 B, free 11.5 GB) 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_2_piece0 stored as bytes in memory (estimated size 87.0 B, free 11.5 GB) 2021-09-23 14:27:22 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 87.0 B, free: 11.5 GB) 2021-09-23 14:27:22 SparkContext: INFO: Created broadcast 2 from broadcast at SparkBackend.scala:250 2021-09-23 14:27:22 root: INFO: instruction count: 3: __C18Compiled. 2021-09-23 14:27:22 root: INFO: instruction count: 328: __C18Compiled.apply 2021-09-23 14:27:22 root: INFO: instruction count: 10: __C18Compiled.__m38addIRIntermediate 2021-09-23 14:27:22 root: INFO: instruction count: 8: __C18Compiled.__m41includesStart 2021-09-23 14:27:22 root: INFO: instruction count: 8: __C18Compiled.__m42includesEnd 2021-09-23 14:27:22 root: INFO: instruction count: 10: __C18Compiled.__m45addIRIntermediate 2021-09-23 14:27:22 root: INFO: instruction count: 10: __C18Compiled.__m46addIRIntermediate 2021-09-23 14:27:22 root: INFO: instruction count: 5: __C18Compiled.__m47addIRIntermediate 2021-09-23 14:27:22 root: INFO: instruction count: 9: __C18Compiled.setPartitionIndex 2021-09-23 14:27:22 root: INFO: instruction count: 4: __C18Compiled.addPartitionRegion 2021-09-23 14:27:22 root: INFO: instruction count: 23: __C18Compiled.__m50DECODE_r_struct_of_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolENDEND_TO_r_tuple_of_r_array_of_r_interval_of_r_locusEND 2021-09-23 14:27:22 root: INFO: instruction count: 52: __C18Compiled.__m51INPLACE_DECODE_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_r_array_of_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDANDr_tuple_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND 2021-09-23 14:27:22 root: INFO: instruction count: 36: __C18Compiled.__m52INPLACE_DECODE_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_struct_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND_TO_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDANDr_tuple_of_r_binaryANDr_int32ENDANDr_boolANDr_boolEND 2021-09-23 14:27:22 root: INFO: instruction count: 22: __C18Compiled.__m53INPLACE_DECODE_r_struct_of_r_binaryANDr_int32END_TO_r_tuple_of_r_binaryANDr_int32END 2021-09-23 14:27:22 root: INFO: instruction count: 31: __C18Compiled.__m54INPLACE_DECODE_r_binary_TO_r_binary 2021-09-23 14:27:22 root: INFO: instruction count: 8: __C18Compiled.__m55INPLACE_DECODE_r_int32_TO_r_int32 2021-09-23 14:27:22 root: INFO: instruction count: 8: __C18Compiled.__m56INPLACE_DECODE_r_bool_TO_r_bool 2021-09-23 14:27:22 root: INFO: instruction count: 50: __C18Compiled.addLiterals 2021-09-23 14:27:22 root: INFO: finished execution of query hail_query_1 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON total 633.307ms self 33.450ms children 599.857ms %children 94.72% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 396.653ms self 0.271ms children 396.382ms %children 99.93% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.294ms self 0.294ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 396.063ms self 41.370ms children 354.693ms %children 89.55% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 354.693ms self 1.076ms children 353.617ms %children 99.70% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 320.568ms self 305.880ms children 14.689ms %children 4.58% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/InlineApplyIR total 0.637ms self 0.174ms children 0.463ms %children 72.76% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/InlineApplyIR/Verify total 0.204ms self 0.204ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/InlineApplyIR/LoweringTransformation total 0.231ms self 0.231ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/InlineApplyIR/Verify total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/LowerArrayAggsToRunAggs total 14.052ms self 0.013ms children 14.039ms %children 99.91% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/LowerArrayAggsToRunAggs/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/LowerArrayAggsToRunAggs/LoweringTransformation total 13.814ms self 13.814ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants/LowerArrayAggsToRunAggs/Verify total 0.208ms self 0.208ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 2.069ms self 2.069ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 5.537ms self 5.537ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 4.398ms self 4.398ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 1.263ms self 1.263ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 13.655ms self 13.655ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.639ms self 0.639ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.132ms self 0.132ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.038ms self 1.038ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.871ms self 0.871ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.621ms self 0.621ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.147ms self 0.147ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.230ms self 1.230ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.092ms self 0.092ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 1.160ms self 1.160ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 5.936ms self 0.017ms children 5.919ms %children 99.72% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 5.876ms self 5.876ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.025ms self 0.025ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 12.070ms self 0.013ms children 12.057ms %children 99.89% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.014ms self 0.014ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 12.027ms self 3.041ms children 8.986ms %children 74.72% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 8.986ms self 0.091ms children 8.895ms %children 98.99% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.137ms self 0.137ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.103ms self 0.103ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.155ms self 0.155ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.955ms self 0.955ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.119ms self 0.119ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.697ms self 0.697ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.594ms self 0.594ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.122ms self 0.122ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.226ms self 0.226ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 1.416ms self 1.416ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.144ms self 0.144ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 1.600ms self 1.600ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.712ms self 0.712ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.147ms self 0.147ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.872ms self 0.872ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.086ms self 0.086ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.750ms self 0.750ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 1.202ms self 0.016ms children 1.186ms %children 98.65% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.023ms self 0.023ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 1.131ms self 1.131ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 12.657ms self 0.023ms children 12.633ms %children 99.82% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 12.603ms self 2.315ms children 10.288ms %children 81.63% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 10.288ms self 0.087ms children 10.201ms %children 99.15% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.169ms self 0.169ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.130ms self 0.130ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 4.130ms self 4.130ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.933ms self 0.933ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.670ms self 0.670ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.843ms self 0.843ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.680ms self 0.680ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.386ms self 0.386ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.210ms self 0.210ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.871ms self 0.871ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.639ms self 0.639ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.017ms self 0.017ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile total 162.498ms self 113.574ms children 48.924ms %children 30.11% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR total 19.743ms self 0.013ms children 19.730ms %children 99.94% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation total 19.680ms self 9.278ms children 10.402ms %children 52.85% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize total 10.402ms self 0.120ms children 10.281ms %children 98.84% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.166ms self 0.166ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.157ms self 0.157ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.446ms self 1.446ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.109ms self 0.109ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 3.699ms self 3.699ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.539ms self 0.539ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.155ms self 0.155ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.816ms self 0.816ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.082ms self 0.082ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.627ms self 0.627ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.719ms self 0.719ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.122ms self 0.122ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.802ms self 0.802ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.581ms self 0.581ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, initial IR/Verify total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR total 0.184ms self 0.008ms children 0.176ms %children 95.54% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/Verify total 0.030ms self 0.030ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/LoweringTransformation total 0.107ms self 0.107ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/InlineApplyIR/Verify total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR total 11.026ms self 0.011ms children 11.014ms %children 99.90% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 2.240ms self 2.240ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation total 8.732ms self 2.444ms children 6.288ms %children 72.01% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize total 6.288ms self 0.123ms children 6.165ms %children 98.04% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.113ms self 0.113ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.210ms self 0.210ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 1.030ms self 1.030ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.639ms self 0.639ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.420ms self 0.420ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.097ms self 0.097ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.719ms self 0.719ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.561ms self 0.561ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/FoldConstants total 0.523ms self 0.523ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/Simplify total 0.099ms self 0.099ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardLets total 0.632ms self 0.632ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/LoweringTransformation/Optimize/PruneDeadFields total 0.688ms self 0.688ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after InlineApplyIR/Verify total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs total 9.312ms self 0.011ms children 9.301ms %children 99.88% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/Verify total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/LoweringTransformation total 9.215ms self 9.215ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/LowerArrayAggsToRunAggs/Verify total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs total 8.659ms self 0.010ms children 8.649ms %children 99.88% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation total 8.621ms self 2.295ms children 6.326ms %children 73.38% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize total 6.326ms self 0.076ms children 6.251ms %children 98.80% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.102ms self 0.102ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.081ms self 0.081ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.113ms self 0.113ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.921ms self 0.921ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.074ms self 0.074ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.901ms self 0.901ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.320ms self 0.320ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.843ms self 0.843ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.656ms self 0.656ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/FoldConstants total 0.308ms self 0.308ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/Simplify total 0.118ms self 0.118ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardLets total 0.773ms self 0.773ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/ForwardRelationalLets total 0.123ms self 0.123ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/LoweringTransformation/Optimize/PruneDeadFields total 0.616ms self 0.616ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/Compile/optimize: compileLowerer, after LowerArrayAggsToRunAggs/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 7.710ms self 7.710ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 1.091ms self 1.091ms children 0.000ms %children 0.00% 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_3 stored as values in memory (estimated size 253.7 KB, free 11.5 GB) 2021-09-23 14:27:22 MemoryStore: INFO: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.2 KB, free 11.5 GB) 2021-09-23 14:27:22 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 24.2 KB, free: 11.5 GB) 2021-09-23 14:27:22 SparkContext: INFO: Created broadcast 3 from broadcast at SparkBackend.scala:250 2021-09-23 14:27:22 DNAxApiSvc: INFO: initializing DNAxApiSvc 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] Initializing file system (URI = dnax:///mnt/project/Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz): com.dnanexus.hadoop.fs.DNAxFileSystem@28d874e3 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] inputStreamBufferSize: 33554432 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] inputStreamMaxAttempts: 6 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] outputStreamBufferSize: 33554432 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] inputStreamRetryableCodes: [500, 503] 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] fileStatusCacheEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] fileStatusCacheSiblingsEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] statusListUrlCacheEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] readFileUrlCacheEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] listStatusSignedUrlEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] fileStatusSignedUrlEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] createDirectorySemaphoreEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] fileStatusSemaphoreEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] listStatusSemaphoreEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] readFileSemaphoreEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] deleteObjectsInBatchesEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] parallelPartUploadEnabled: true 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] uploadChunkSize: 16777216 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] multiPartThreshold: 1073741824 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] semaphorePermitWaitTime: 10 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] semaphoreCreateDirectoryPermitCount: 1 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] semaphoreFileStatusPermitCount: 1 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] semaphoreListStatusPermitCount: 1 2021-09-23 14:27:22 DNAxFileSystem: INFO: [initialize] semaphoreReadFilePermitCount: 1 2021-09-23 14:27:22 root: INFO: The exact same exception object, java.lang.RuntimeException: Unexpected null value for databaseId, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:27:22 root: ERROR: RuntimeException: Unexpected null value for databaseId From java.lang.RuntimeException: Unexpected null value for databaseId at com.dnanexus.hadoop.fs.DNAxFileSystem.checkDatabaseId(DNAxFileSystem.java:3061) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:423) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:413) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:405) at com.dnanexus.hadoop.fs.DNAxFileSystem.getFileStatus(DNAxFileSystem.java:2292) at com.dnanexus.hadoop.fs.DNAxFileSystem.open(DNAxFileSystem.java:536) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:83) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:27:30 root: INFO: timing SparkBackend.parse_value_ir total 6.713ms self 6.713ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: starting execution of query hail_query_2 of initial size 18 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_4 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_4)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_4)))) (Apply includesStart () Boolean (Ref __uid_4)) (Apply includesEnd () Boolean (Ref __uid_4))))) 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_20 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_20)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_20)))) (Apply includesStart () Boolean (Ref __iruid_20)) (Apply includesEnd () Boolean (Ref __iruid_20))))) 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_20 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_20)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_20)))) (Apply includesStart () Boolean (Ref __iruid_20)) (Apply includesEnd () Boolean (Ref __iruid_20))))) 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_23 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_23)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_23)))) (Apply includesStart () Boolean (Ref __iruid_23)) (Apply includesEnd () Boolean (Ref __iruid_23))))) 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_23 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_23)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_23)))) (Apply includesStart () Boolean (Ref __iruid_23)) (Apply includesEnd () Boolean (Ref __iruid_23))))) 2021-09-23 14:27:30 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_26 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_26)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_26)))) (Apply includesStart () Boolean (Ref __iruid_26)) (Apply includesEnd () Boolean (Ref __iruid_26))))) 2021-09-23 14:27:30 root: INFO: finished execution of query hail_query_2 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON total 39.878ms self 1.777ms children 38.101ms %children 95.54% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 25.054ms self 0.012ms children 25.041ms %children 99.95% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 25.018ms self 1.995ms children 23.023ms %children 92.03% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 23.023ms self 0.070ms children 22.954ms %children 99.70% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 17.921ms self 17.921ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.104ms self 0.104ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.844ms self 0.844ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.796ms self 0.796ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.307ms self 0.307ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.619ms self 0.619ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.493ms self 0.493ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.331ms self 0.331ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.086ms self 0.086ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.526ms self 0.526ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.451ms self 0.451ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.081ms self 0.010ms children 0.071ms %children 87.96% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 5.874ms self 0.009ms children 5.865ms %children 99.84% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 5.846ms self 1.585ms children 4.261ms %children 72.89% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 4.261ms self 0.078ms children 4.183ms %children 98.17% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.123ms self 0.123ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.059ms self 0.059ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.522ms self 0.522ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.466ms self 0.466ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.321ms self 0.321ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.082ms self 0.082ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.515ms self 0.515ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.413ms self 0.413ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.276ms self 0.276ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.554ms self 0.554ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.414ms self 0.414ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.087ms self 0.007ms children 0.080ms %children 91.97% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.022ms self 0.022ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 6.440ms self 0.008ms children 6.432ms %children 99.87% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 6.422ms self 1.756ms children 4.665ms %children 72.65% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 4.665ms self 0.065ms children 4.600ms %children 98.61% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.139ms self 0.139ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.079ms self 0.079ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.281ms self 0.281ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.580ms self 0.580ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.521ms self 0.521ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.274ms self 0.274ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.048ms self 0.048ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.527ms self 0.527ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.052ms self 0.052ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.470ms self 0.470ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.282ms self 0.282ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.161ms self 0.161ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.506ms self 0.506ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.403ms self 0.403ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/Compile total 0.373ms self 0.373ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.096ms self 0.096ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2021-09-23 14:27:30 root: INFO: The exact same exception object, java.lang.RuntimeException: Unexpected null value for databaseId, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:27:30 root: ERROR: RuntimeException: Unexpected null value for databaseId From java.lang.RuntimeException: Unexpected null value for databaseId at com.dnanexus.hadoop.fs.DNAxFileSystem.checkDatabaseId(DNAxFileSystem.java:3061) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:423) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:413) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:405) at com.dnanexus.hadoop.fs.DNAxFileSystem.getFileStatus(DNAxFileSystem.java:2292) at com.dnanexus.hadoop.fs.DNAxFileSystem.open(DNAxFileSystem.java:536) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:83) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:27:35 root: INFO: timing SparkBackend.parse_value_ir total 3.702ms self 3.702ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: starting execution of query hail_query_3 of initial size 18 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_5 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_5)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_5)))) (Apply includesStart () Boolean (Ref __uid_5)) (Apply includesEnd () Boolean (Ref __uid_5))))) 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_29 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_29)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_29)))) (Apply includesStart () Boolean (Ref __iruid_29)) (Apply includesEnd () Boolean (Ref __iruid_29))))) 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_29 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_29)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_29)))) (Apply includesStart () Boolean (Ref __iruid_29)) (Apply includesEnd () Boolean (Ref __iruid_29))))) 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_32 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_32)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_32)))) (Apply includesStart () Boolean (Ref __iruid_32)) (Apply includesEnd () Boolean (Ref __iruid_32))))) 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_32 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_32)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_32)))) (Apply includesStart () Boolean (Ref __iruid_32)) (Apply includesEnd () Boolean (Ref __iruid_32))))) 2021-09-23 14:27:35 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_35 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_35)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_35)))) (Apply includesStart () Boolean (Ref __iruid_35)) (Apply includesEnd () Boolean (Ref __iruid_35))))) 2021-09-23 14:27:35 root: INFO: finished execution of query hail_query_3 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON total 32.324ms self 1.647ms children 30.676ms %children 94.90% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 17.344ms self 0.010ms children 17.334ms %children 99.94% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 17.306ms self 1.443ms children 15.863ms %children 91.66% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 15.863ms self 0.123ms children 15.740ms %children 99.22% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 11.215ms self 11.215ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.101ms self 0.101ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.115ms self 0.115ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.611ms self 0.611ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.515ms self 0.515ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.293ms self 0.293ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.074ms self 0.074ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.593ms self 0.593ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.445ms self 0.445ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.454ms self 0.454ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.552ms self 0.552ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.429ms self 0.429ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.066ms self 0.006ms children 0.060ms %children 91.59% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 5.462ms self 0.006ms children 5.457ms %children 99.90% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 5.445ms self 1.090ms children 4.355ms %children 79.98% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 4.355ms self 0.044ms children 4.311ms %children 98.98% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.079ms self 0.079ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.604ms self 0.604ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.087ms self 0.087ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.429ms self 0.429ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.275ms self 0.275ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.077ms self 0.077ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.526ms self 0.526ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.628ms self 0.628ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.259ms self 0.259ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.073ms self 0.073ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.496ms self 0.496ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.444ms self 0.444ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.069ms self 0.003ms children 0.066ms %children 95.23% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.021ms self 0.021ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 7.228ms self 0.006ms children 7.222ms %children 99.92% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 7.210ms self 1.023ms children 6.187ms %children 85.81% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 6.187ms self 0.044ms children 6.142ms %children 99.28% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.092ms self 0.092ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.495ms self 0.495ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 2.264ms self 2.264ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.306ms self 0.306ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.464ms self 0.464ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.512ms self 0.512ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.434ms self 0.434ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.125ms self 0.125ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.488ms self 0.488ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.552ms self 0.552ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/Compile total 0.288ms self 0.288ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.072ms self 0.072ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.088ms self 0.088ms children 0.000ms %children 0.00% 2021-09-23 14:27:35 root: INFO: The exact same exception object, java.lang.RuntimeException: Unexpected null value for databaseId, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:27:35 root: ERROR: RuntimeException: Unexpected null value for databaseId From java.lang.RuntimeException: Unexpected null value for databaseId at com.dnanexus.hadoop.fs.DNAxFileSystem.checkDatabaseId(DNAxFileSystem.java:3061) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:423) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:413) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:405) at com.dnanexus.hadoop.fs.DNAxFileSystem.getFileStatus(DNAxFileSystem.java:2292) at com.dnanexus.hadoop.fs.DNAxFileSystem.open(DNAxFileSystem.java:536) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:83) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:27:59 root: INFO: timing SparkBackend.parse_value_ir total 3.778ms self 3.778ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: starting execution of query hail_query_4 of initial size 18 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_6 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_6)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_6)))) (Apply includesStart () Boolean (Ref __uid_6)) (Apply includesEnd () Boolean (Ref __uid_6))))) 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_38 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_38)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_38)))) (Apply includesStart () Boolean (Ref __iruid_38)) (Apply includesEnd () Boolean (Ref __iruid_38))))) 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_38 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_38)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_38)))) (Apply includesStart () Boolean (Ref __iruid_38)) (Apply includesEnd () Boolean (Ref __iruid_38))))) 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_41 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_41)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_41)))) (Apply includesStart () Boolean (Ref __iruid_41)) (Apply includesEnd () Boolean (Ref __iruid_41))))) 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_41 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_41)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_41)))) (Apply includesStart () Boolean (Ref __iruid_41)) (Apply includesEnd () Boolean (Ref __iruid_41))))) 2021-09-23 14:27:59 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_44 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_44)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_44)))) (Apply includesStart () Boolean (Ref __iruid_44)) (Apply includesEnd () Boolean (Ref __iruid_44))))) 2021-09-23 14:27:59 root: INFO: finished execution of query hail_query_4 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON total 29.676ms self 1.482ms children 28.194ms %children 95.01% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 17.212ms self 0.007ms children 17.205ms %children 99.96% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 17.187ms self 1.516ms children 15.671ms %children 91.18% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 15.671ms self 0.073ms children 15.598ms %children 99.54% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 11.654ms self 11.654ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.091ms self 0.091ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.484ms self 0.484ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.450ms self 0.450ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.366ms self 0.366ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.085ms self 0.085ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.441ms self 0.441ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.465ms self 0.465ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.239ms self 0.239ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.462ms self 0.462ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.511ms self 0.511ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.061ms self 0.003ms children 0.058ms %children 95.24% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 5.068ms self 0.005ms children 5.063ms %children 99.91% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 5.050ms self 1.099ms children 3.950ms %children 78.23% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 3.950ms self 0.054ms children 3.896ms %children 98.63% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.573ms self 0.573ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.467ms self 0.467ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.253ms self 0.253ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.409ms self 0.409ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.497ms self 0.497ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.261ms self 0.261ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.447ms self 0.447ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.423ms self 0.423ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.059ms self 0.002ms children 0.057ms %children 96.42% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.013ms self 0.013ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 5.331ms self 0.004ms children 5.326ms %children 99.92% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 5.308ms self 1.042ms children 4.266ms %children 80.37% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 4.266ms self 0.066ms children 4.200ms %children 98.45% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.095ms self 0.095ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.518ms self 0.518ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.053ms self 0.053ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.463ms self 0.463ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.400ms self 0.400ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.510ms self 0.510ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.488ms self 0.488ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.352ms self 0.352ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.483ms self 0.483ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.390ms self 0.390ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/Compile total 0.290ms self 0.290ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.086ms self 0.086ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:27:59 root: INFO: The exact same exception object, java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: DNAnexus:Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:27:59 root: ERROR: URISyntaxException: Relative path in absolute URI: DNAnexus:Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz From java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: DNAnexus:Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz at org.apache.hadoop.fs.Path.initialize(Path.java:205) at org.apache.hadoop.fs.Path.(Path.java:171) at is.hail.io.fs.HadoopFS.getFileSystem(HadoopFS.scala:98) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:80) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) java.net.URISyntaxException: Relative path in absolute URI: DNAnexus:Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz at java.net.URI.checkPath(URI.java:1823) at java.net.URI.(URI.java:745) at org.apache.hadoop.fs.Path.initialize(Path.java:202) at org.apache.hadoop.fs.Path.(Path.java:171) at is.hail.io.fs.HadoopFS.getFileSystem(HadoopFS.scala:98) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:80) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:28:08 root: INFO: timing SparkBackend.parse_value_ir total 2.751ms self 2.751ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: starting execution of query hail_query_5 of initial size 18 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_7 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_7)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_7)))) (Apply includesStart () Boolean (Ref __uid_7)) (Apply includesEnd () Boolean (Ref __uid_7))))) 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_47 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_47)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_47)))) (Apply includesStart () Boolean (Ref __iruid_47)) (Apply includesEnd () Boolean (Ref __iruid_47))))) 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_47 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_47)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_47)))) (Apply includesStart () Boolean (Ref __iruid_47)) (Apply includesEnd () Boolean (Ref __iruid_47))))) 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_50 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_50)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_50)))) (Apply includesStart () Boolean (Ref __iruid_50)) (Apply includesEnd () Boolean (Ref __iruid_50))))) 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_50 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_50)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_50)))) (Apply includesStart () Boolean (Ref __iruid_50)) (Apply includesEnd () Boolean (Ref __iruid_50))))) 2021-09-23 14:28:08 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_53 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_53)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_53)))) (Apply includesStart () Boolean (Ref __iruid_53)) (Apply includesEnd () Boolean (Ref __iruid_53))))) 2021-09-23 14:28:08 root: INFO: finished execution of query hail_query_5 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON total 26.660ms self 1.326ms children 25.334ms %children 95.03% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 16.203ms self 0.007ms children 16.196ms %children 99.95% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 16.180ms self 1.275ms children 14.905ms %children 92.12% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 14.905ms self 0.037ms children 14.867ms %children 99.75% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 11.290ms self 11.290ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.084ms self 0.084ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.467ms self 0.467ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.457ms self 0.457ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.253ms self 0.253ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.074ms self 0.074ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.452ms self 0.452ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.412ms self 0.412ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.212ms self 0.212ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.454ms self 0.454ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.378ms self 0.378ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.080ms self 0.004ms children 0.076ms %children 94.76% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 4.432ms self 0.005ms children 4.427ms %children 99.90% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 4.418ms self 0.880ms children 3.537ms %children 80.07% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 3.537ms self 0.032ms children 3.506ms %children 99.11% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.061ms self 0.061ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.075ms self 0.075ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.432ms self 0.432ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.054ms self 0.054ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.398ms self 0.398ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.212ms self 0.212ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.449ms self 0.449ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.379ms self 0.379ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.267ms self 0.267ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.451ms self 0.451ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.361ms self 0.361ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.062ms self 0.002ms children 0.060ms %children 96.55% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.015ms self 0.015ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 4.140ms self 0.004ms children 4.136ms %children 99.91% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 4.127ms self 0.840ms children 3.287ms %children 79.64% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 3.287ms self 0.029ms children 3.258ms %children 99.13% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.096ms self 0.096ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.419ms self 0.419ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.399ms self 0.399ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.228ms self 0.228ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.416ms self 0.416ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.334ms self 0.334ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.203ms self 0.203ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.387ms self 0.387ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.325ms self 0.325ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/Compile total 0.264ms self 0.264ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.067ms self 0.067ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:28:08 root: INFO: The exact same exception object, java.io.IOException: No FileSystem for scheme: DNAnexus, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:28:08 root: ERROR: IOException: No FileSystem for scheme: DNAnexus From java.io.IOException: No FileSystem for scheme: DNAnexus at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at is.hail.io.fs.HadoopFS.getFileSystem(HadoopFS.scala:98) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:80) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:28:48 root: INFO: timing SparkBackend.parse_value_ir total 2.622ms self 2.622ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: starting execution of query hail_query_6 of initial size 18 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_8 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_8)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_8)))) (Apply includesStart () Boolean (Ref __uid_8)) (Apply includesEnd () Boolean (Ref __uid_8))))) 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_56 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_56)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_56)))) (Apply includesStart () Boolean (Ref __iruid_56)) (Apply includesEnd () Boolean (Ref __iruid_56))))) 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_56 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_56)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_56)))) (Apply includesStart () Boolean (Ref __iruid_56)) (Apply includesEnd () Boolean (Ref __iruid_56))))) 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_59 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_59)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_59)))) (Apply includesStart () Boolean (Ref __iruid_59)) (Apply includesEnd () Boolean (Ref __iruid_59))))) 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_59 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_59)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_59)))) (Apply includesStart () Boolean (Ref __iruid_59)) (Apply includesEnd () Boolean (Ref __iruid_59))))) 2021-09-23 14:28:48 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_62 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_62)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_62)))) (Apply includesStart () Boolean (Ref __iruid_62)) (Apply includesEnd () Boolean (Ref __iruid_62))))) 2021-09-23 14:28:48 root: INFO: finished execution of query hail_query_6 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON total 25.608ms self 1.179ms children 24.429ms %children 95.39% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 16.237ms self 0.006ms children 16.231ms %children 99.96% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 16.212ms self 0.919ms children 15.292ms %children 94.33% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 15.292ms self 0.036ms children 15.257ms %children 99.77% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 11.309ms self 11.309ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.051ms self 0.051ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.442ms self 0.442ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.425ms self 0.425ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.190ms self 0.190ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 1.232ms self 1.232ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.318ms self 0.318ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.203ms self 0.203ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.369ms self 0.369ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.303ms self 0.303ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.060ms self 0.003ms children 0.057ms %children 95.22% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 3.733ms self 0.004ms children 3.729ms %children 99.90% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 3.721ms self 0.823ms children 2.897ms %children 77.87% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 2.897ms self 0.027ms children 2.870ms %children 99.06% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.346ms self 0.346ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.044ms self 0.044ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.306ms self 0.306ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.195ms self 0.195ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.337ms self 0.337ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.327ms self 0.327ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.379ms self 0.379ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.296ms self 0.296ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.057ms self 0.002ms children 0.055ms %children 96.55% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.036ms self 0.036ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 3.971ms self 0.004ms children 3.968ms %children 99.91% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 3.957ms self 0.847ms children 3.110ms %children 78.59% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 3.110ms self 0.032ms children 3.078ms %children 98.97% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.070ms self 0.070ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.395ms self 0.395ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.308ms self 0.308ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.202ms self 0.202ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.042ms self 0.042ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.421ms self 0.421ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.315ms self 0.315ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.219ms self 0.219ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.369ms self 0.369ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.329ms self 0.329ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/Compile total 0.218ms self 0.218ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2021-09-23 14:28:48 root: INFO: The exact same exception object, java.lang.RuntimeException: Unexpected null value for databaseId, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:28:48 root: ERROR: RuntimeException: Unexpected null value for databaseId From java.lang.RuntimeException: Unexpected null value for databaseId at com.dnanexus.hadoop.fs.DNAxFileSystem.checkDatabaseId(DNAxFileSystem.java:3061) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:423) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:413) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:405) at com.dnanexus.hadoop.fs.DNAxFileSystem.getFileStatus(DNAxFileSystem.java:2292) at com.dnanexus.hadoop.fs.DNAxFileSystem.open(DNAxFileSystem.java:536) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:83) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:28:54 root: INFO: timing SparkBackend.parse_value_ir total 2.428ms self 2.428ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: starting execution of query hail_query_7 of initial size 18 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_9 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_9)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_9)))) (Apply includesStart () Boolean (Ref __uid_9)) (Apply includesEnd () Boolean (Ref __uid_9))))) 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_65 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_65)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_65)))) (Apply includesStart () Boolean (Ref __iruid_65)) (Apply includesEnd () Boolean (Ref __iruid_65))))) 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_65 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_65)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_65)))) (Apply includesStart () Boolean (Ref __iruid_65)) (Apply includesEnd () Boolean (Ref __iruid_65))))) 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_68 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_68)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_68)))) (Apply includesStart () Boolean (Ref __iruid_68)) (Apply includesEnd () Boolean (Ref __iruid_68))))) 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_68 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_68)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_68)))) (Apply includesStart () Boolean (Ref __iruid_68)) (Apply includesEnd () Boolean (Ref __iruid_68))))) 2021-09-23 14:28:54 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_71 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_71)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_71)))) (Apply includesStart () Boolean (Ref __iruid_71)) (Apply includesEnd () Boolean (Ref __iruid_71))))) 2021-09-23 14:28:54 root: INFO: finished execution of query hail_query_7 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON total 24.353ms self 1.183ms children 23.169ms %children 95.14% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 15.324ms self 0.006ms children 15.318ms %children 99.96% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 15.303ms self 0.939ms children 14.364ms %children 93.87% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 14.364ms self 0.040ms children 14.324ms %children 99.72% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 10.963ms self 10.963ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.047ms self 0.047ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.414ms self 0.414ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.058ms self 0.058ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.495ms self 0.495ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.193ms self 0.193ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.389ms self 0.389ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.404ms self 0.404ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.200ms self 0.200ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.064ms self 0.064ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.401ms self 0.401ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.056ms self 0.056ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.368ms self 0.368ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.059ms self 0.003ms children 0.056ms %children 95.33% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.043ms self 0.043ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 3.702ms self 0.004ms children 3.699ms %children 99.90% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 3.689ms self 0.741ms children 2.948ms %children 79.90% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 2.948ms self 0.026ms children 2.922ms %children 99.11% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.057ms self 0.057ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.357ms self 0.357ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.300ms self 0.300ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.414ms self 0.414ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.313ms self 0.313ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.179ms self 0.179ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.373ms self 0.373ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.335ms self 0.335ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.058ms self 0.002ms children 0.056ms %children 95.77% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.012ms self 0.012ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 3.579ms self 0.003ms children 3.576ms %children 99.91% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 3.567ms self 0.683ms children 2.883ms %children 80.84% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 2.883ms self 0.026ms children 2.857ms %children 99.10% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.404ms self 0.404ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.321ms self 0.321ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.178ms self 0.178ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.367ms self 0.367ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.307ms self 0.307ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.175ms self 0.175ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.327ms self 0.327ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.290ms self 0.290ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/Compile total 0.287ms self 0.287ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.011ms self 0.011ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.081ms self 0.081ms children 0.000ms %children 0.00% 2021-09-23 14:28:54 DNAxApiSvc: INFO: initializing DNAxApiSvc 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] Initializing file system (URI = dnax://Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz): com.dnanexus.hadoop.fs.DNAxFileSystem@8eea6d6 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] inputStreamBufferSize: 33554432 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] inputStreamMaxAttempts: 6 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] outputStreamBufferSize: 33554432 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] inputStreamRetryableCodes: [500, 503] 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] fileStatusCacheEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] fileStatusCacheSiblingsEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] statusListUrlCacheEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] readFileUrlCacheEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] listStatusSignedUrlEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] fileStatusSignedUrlEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] createDirectorySemaphoreEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] fileStatusSemaphoreEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] listStatusSemaphoreEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] readFileSemaphoreEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] deleteObjectsInBatchesEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] parallelPartUploadEnabled: true 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] uploadChunkSize: 16777216 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] multiPartThreshold: 1073741824 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] semaphorePermitWaitTime: 10 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] semaphoreCreateDirectoryPermitCount: 1 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] semaphoreFileStatusPermitCount: 1 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] semaphoreListStatusPermitCount: 1 2021-09-23 14:28:54 DNAxFileSystem: INFO: [initialize] semaphoreReadFilePermitCount: 1 2021-09-23 14:28:54 root: INFO: The exact same exception object, java.lang.RuntimeException: Unexpected null value for databaseId, was thrown by both the consumer and the close method. I will throw the original. 2021-09-23 14:28:54 root: ERROR: RuntimeException: Unexpected null value for databaseId From java.lang.RuntimeException: Unexpected null value for databaseId at com.dnanexus.hadoop.fs.DNAxFileSystem.checkDatabaseId(DNAxFileSystem.java:3061) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:423) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:413) at com.dnanexus.hadoop.fs.DNAxFileSystem.preparePathForApiRoute(DNAxFileSystem.java:405) at com.dnanexus.hadoop.fs.DNAxFileSystem.getFileStatus(DNAxFileSystem.java:2292) at com.dnanexus.hadoop.fs.DNAxFileSystem.open(DNAxFileSystem.java:536) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:83) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.readLines(FS.scala:208) at is.hail.io.fs.HadoopFS.readLines(HadoopFS.scala:70) at is.hail.io.vcf.LoadVCF$.getHeaderLines(LoadVCF.scala:1278) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1829) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:502) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:501) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:501) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:500) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:500) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2021-09-23 14:29:38 root: INFO: timing SparkBackend.parse_value_ir total 2.330ms self 2.330ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: starting execution of query hail_query_8 of initial size 18 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 18: (ToArray (StreamMap __uid_10 (ToStream False (MakeArray Array[Interval[Locus(GRCh38)]] (ApplySpecial LocusInterval () Interval[Locus(GRCh38)] (Str "[chr19:456...") (False)))) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __uid_10)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __uid_10)))) (Apply includesStart () Boolean (Ref __uid_10)) (Apply includesEnd () Boolean (Ref __uid_10))))) 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 15: (ToArray (StreamMap __iruid_74 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_74)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_74)))) (Apply includesStart () Boolean (Ref __iruid_74)) (Apply includesEnd () Boolean (Ref __iruid_74))))) 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 15: (ToArray (StreamMap __iruid_74 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_74)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_74)))) (Apply includesStart () Boolean (Ref __iruid_74)) (Apply includesEnd () Boolean (Ref __iruid_74))))) 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 15: (ToArray (StreamMap __iruid_77 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_77)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_77)))) (Apply includesStart () Boolean (Ref __iruid_77)) (Apply includesEnd () Boolean (Ref __iruid_77))))) 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 15: (ToArray (StreamMap __iruid_77 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_77)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_77)))) (Apply includesStart () Boolean (Ref __iruid_77)) (Apply includesEnd () Boolean (Ref __iruid_77))))) 2021-09-23 14:29:38 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 15: (ToArray (StreamMap __iruid_80 (ToStream False (Literal Array[Interval[Locus(GRCh38)]] )) (ApplySpecial Interval () Interval[Struct{locus:Locus(GRCh38)}] (MakeStruct (locus (ApplySpecial start () Locus(GRCh38) (Ref __iruid_80)))) (MakeStruct (locus (ApplySpecial end () Locus(GRCh38) (Ref __iruid_80)))) (Apply includesStart () Boolean (Ref __iruid_80)) (Apply includesEnd () Boolean (Ref __iruid_80))))) 2021-09-23 14:29:38 root: INFO: finished execution of query hail_query_8 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON total 23.561ms self 1.223ms children 22.338ms %children 94.81% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 14.756ms self 0.007ms children 14.748ms %children 99.95% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.007ms self 0.007ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 14.732ms self 0.922ms children 13.811ms %children 93.74% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 13.811ms self 0.035ms children 13.776ms %children 99.75% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 11.017ms self 11.017ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.050ms self 0.050ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.083ms self 0.083ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.380ms self 0.380ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.032ms self 0.032ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.354ms self 0.354ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.181ms self 0.181ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.039ms self 0.039ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.069ms self 0.069ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.320ms self 0.320ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.031ms self 0.031ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.303ms self 0.303ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.195ms self 0.195ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.040ms self 0.040ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 0.063ms self 0.063ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.304ms self 0.304ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.289ms self 0.289ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.009ms self 0.009ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 0.062ms self 0.006ms children 0.056ms %children 90.60% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 3.524ms self 0.004ms children 3.521ms %children 99.90% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 3.512ms self 0.819ms children 2.693ms %children 76.68% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 2.693ms self 0.034ms children 2.658ms %children 98.72% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.071ms self 0.071ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.041ms self 0.041ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.331ms self 0.331ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.291ms self 0.291ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.175ms self 0.175ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.301ms self 0.301ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.024ms self 0.024ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.317ms self 0.317ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.167ms self 0.167ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.037ms self 0.037ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.343ms self 0.343ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.278ms self 0.278ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 0.091ms self 0.002ms children 0.089ms %children 97.31% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.008ms self 0.008ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.046ms self 0.046ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 3.512ms self 0.003ms children 3.508ms %children 99.91% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 3.498ms self 0.657ms children 2.841ms %children 81.22% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 2.841ms self 0.026ms children 2.815ms %children 99.09% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.060ms self 0.060ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.038ms self 0.038ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.068ms self 0.068ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.464ms self 0.464ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.028ms self 0.028ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.298ms self 0.298ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.180ms self 0.180ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.330ms self 0.330ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.296ms self 0.296ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.177ms self 0.177ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.035ms self 0.035ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.062ms self 0.062ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.314ms self 0.314ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.312ms self 0.312ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/Compile total 0.237ms self 0.237ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/InitializeCompiledFunction total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/RunCompiledFunction total 0.010ms self 0.010ms children 0.000ms %children 0.00% 2021-09-23 14:29:38 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.080ms self 0.080ms children 0.000ms %children 0.00% 2021-09-23 14:29:51 MemoryStore: INFO: Block broadcast_4 stored as values in memory (estimated size 463.0 KB, free 11.5 GB) 2021-09-23 14:29:51 MemoryStore: INFO: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1286.7 KB, free 11.5 GB) 2021-09-23 14:29:51 BlockManagerInfo: INFO: Added broadcast_4_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 1286.7 KB, free: 11.5 GB) 2021-09-23 14:29:51 SparkContext: INFO: Created broadcast 4 from broadcast at SparkBackend.scala:250 2021-09-23 14:29:51 SparkContext: INFO: Starting job: collect at LoadVCF.scala:1909 2021-09-23 14:29:51 DAGScheduler: INFO: [DNAX] Setting local property com.dnanexus.auth 2021-09-23 14:29:51 DAGScheduler: INFO: Unable to extract and set 'user' or 'containsPHI' in SparkSession 2021-09-23 14:29:51 DAGScheduler: INFO: Got job 0 (collect at LoadVCF.scala:1909) with 1 output partitions 2021-09-23 14:29:51 DAGScheduler: INFO: Final stage: ResultStage 0 (collect at LoadVCF.scala:1909) 2021-09-23 14:29:51 DAGScheduler: INFO: Parents of final stage: List() 2021-09-23 14:29:51 DAGScheduler: INFO: Missing parents: List() 2021-09-23 14:29:51 DAGScheduler: INFO: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at LoadVCF.scala:1888), which has no missing parents 2021-09-23 14:29:52 MemoryStore: INFO: Block broadcast_5 stored as values in memory (estimated size 11.9 KB, free 11.5 GB) 2021-09-23 14:29:52 MemoryStore: INFO: Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.7 KB, free 11.5 GB) 2021-09-23 14:29:52 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 5.7 KB, free: 11.5 GB) 2021-09-23 14:29:52 SparkContext: INFO: Created broadcast 5 from broadcast at DAGScheduler.scala:1268 2021-09-23 14:29:52 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at LoadVCF.scala:1888) (first 15 tasks are for partitions Vector(0)) 2021-09-23 14:29:52 TaskSchedulerImpl: INFO: Adding task set 0.0 with 1 tasks 2021-09-23 14:29:52 TaskSetManager: INFO: Starting task 0.0 in stage 0.0 (TID 0, ip-10-60-146-200.eu-west-2.compute.internal, executor 0, partition 0, PROCESS_LOCAL, 8043 bytes) 2021-09-23 14:29:52 BlockManagerInfo: INFO: Added broadcast_5_piece0 in memory on ip-10-60-146-200.eu-west-2.compute.internal:44000 (size: 5.7 KB, free: 11.5 GB) 2021-09-23 14:29:52 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on ip-10-60-146-200.eu-west-2.compute.internal:44000 (size: 24.2 KB, free: 11.5 GB) 2021-09-23 14:30:05 BlockManagerInfo: INFO: Added taskresult_0 in memory on ip-10-60-146-200.eu-west-2.compute.internal:44000 (size: 1575.9 KB, free: 11.5 GB) 2021-09-23 14:30:05 TransportClientFactory: INFO: Successfully created connection to ip-10-60-146-200.eu-west-2.compute.internal/10.60.146.200:44000 after 2 ms (0 ms spent in bootstraps) 2021-09-23 14:30:05 TaskSetManager: INFO: Finished task 0.0 in stage 0.0 (TID 0) in 13940 ms on ip-10-60-146-200.eu-west-2.compute.internal (executor 0) (1/1) 2021-09-23 14:30:05 TaskSchedulerImpl: INFO: Removed TaskSet 0.0, whose tasks have all completed, from pool 2021-09-23 14:30:05 DAGScheduler: INFO: ResultStage 0 (collect at LoadVCF.scala:1909) finished in 13.976 s 2021-09-23 14:30:05 BlockManagerInfo: INFO: Removed taskresult_0 on ip-10-60-146-200.eu-west-2.compute.internal:44000 in memory (size: 1575.9 KB, free: 11.5 GB) 2021-09-23 14:30:05 DAGScheduler: INFO: Job 0 finished: collect at LoadVCF.scala:1909, took 14.014202 s 2021-09-23 14:30:06 root: INFO: RegionPool: REPORT_THRESHOLD: 1.7M allocated (192.0K blocks / 1.5M chunks), regions.size = 3, 0 current java objects, 0 max java objects, thread 24: Thread-4 2021-09-23 14:30:06 root: INFO: RegionPool: REPORT_THRESHOLD: 1.8M allocated (256.0K blocks / 1.5M chunks), regions.size = 3, 0 current java objects, 0 max java objects, thread 24: Thread-4 2021-09-23 14:30:06 root: INFO: RegionPool: REPORT_THRESHOLD: 1.8M allocated (320.0K blocks / 1.5M chunks), regions.size = 3, 0 current java objects, 0 max java objects, thread 24: Thread-4 2021-09-23 14:30:06 root: INFO: RegionPool: REPORT_THRESHOLD: 2.0M allocated (512.0K blocks / 1.5M chunks), regions.size = 3, 0 current java objects, 0 max java objects, thread 24: Thread-4 2021-09-23 14:30:06 root: INFO: encoder cache miss (0 hits, 2 misses, 0.000) 2021-09-23 14:30:06 root: INFO: instruction count: 3: __C65etypeEncode. 2021-09-23 14:30:06 root: INFO: instruction count: 5: __C65etypeEncode.apply 2021-09-23 14:30:06 root: INFO: instruction count: 13: __C65etypeEncode.__m66ENCODE_r_tuple_of_r_array_of_r_tuple_of_r_stringENDEND_TO_r_struct_of_r_array_of_r_struct_of_r_binaryENDEND 2021-09-23 14:30:06 root: INFO: instruction count: 49: __C65etypeEncode.__m67ENCODE_r_array_of_r_tuple_of_r_binaryEND_TO_r_array_of_r_struct_of_r_binaryEND 2021-09-23 14:30:06 root: INFO: instruction count: 13: __C65etypeEncode.__m68ENCODE_r_tuple_of_r_binaryEND_TO_r_struct_of_r_binaryEND 2021-09-23 14:30:06 root: INFO: instruction count: 16: __C65etypeEncode.__m69ENCODE_r_binary_TO_r_binary 2021-09-23 14:30:06 root: INFO: timing SparkBackend.pyImportVCFs total 27.801s self 27.801s children 0.000ms %children 0.00% 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 6 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 7 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 13 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 18 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 16 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 11 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 9 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 2 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 5 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 15 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 3 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 19 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 21 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 23 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 26 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 4 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 12 2021-09-23 14:30:22 BlockManagerInfo: INFO: Removed broadcast_4_piece0 on ip-10-60-95-0.eu-west-2.compute.internal:43001 in memory (size: 1286.7 KB, free: 11.5 GB) 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 8 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 17 2021-09-23 14:30:22 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on ip-10-60-95-0.eu-west-2.compute.internal:43001 in memory (size: 5.7 KB, free: 11.5 GB) 2021-09-23 14:30:22 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on ip-10-60-146-200.eu-west-2.compute.internal:44000 in memory (size: 5.7 KB, free: 11.5 GB) 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 24 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 20 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 25 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 1 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 10 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 14 2021-09-23 14:30:22 ContextCleaner: INFO: Cleaned accumulator 22 2021-09-23 14:30:23 HiveUtils: INFO: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 2021-09-23 14:30:23 Hive: INFO: [dnax] Created MetaStoreClient cache with max size 100, expire time 30 seconds. 2021-09-23 14:30:23 metastore: INFO: Trying to connect to metastore with URI http://d1-MetastoreLB-652488164.us-east-1.elb.amazonaws.com:9083 2021-09-23 14:30:23 ShellBasedUnixGroupsMapping: WARN: got exception trying to get groups for user 7Jx7G7jVp7Fpfj6Z5VjQkkzBFj4VGKy5FvkYB0YQ__project-G493Kx8J860q68yGFpxkz18y: id: ‘7Jx7G7jVp7Fpfj6Z5VjQkkzBFj4VGKy5FvkYB0YQ__project-G493Kx8J860q68yGFpxkz18y’: no such user id: ‘7Jx7G7jVp7Fpfj6Z5VjQkkzBFj4VGKy5FvkYB0YQ__project-G493Kx8J860q68yGFpxkz18y’: no such user 2021-09-23 14:30:24 metastore: INFO: Connected to metastore. 2021-09-23 14:30:24 SessionState: INFO: Created HDFS directory: /tmp/hive/root 2021-09-23 14:30:24 SessionState: INFO: Created local directory: /tmp/root 2021-09-23 14:30:24 SessionState: INFO: Created local directory: /tmp/c4758b15-d224-4efe-a1e0-dcec9ff89241_resources 2021-09-23 14:30:24 SessionState: INFO: Created HDFS directory: /tmp/hive/root/c4758b15-d224-4efe-a1e0-dcec9ff89241 2021-09-23 14:30:24 SessionState: INFO: Created local directory: /tmp/root/c4758b15-d224-4efe-a1e0-dcec9ff89241 2021-09-23 14:30:24 SessionState: INFO: Created HDFS directory: /tmp/hive/root/c4758b15-d224-4efe-a1e0-dcec9ff89241/_tmp_space.db 2021-09-23 14:30:24 HiveClientImpl: INFO: Warehouse location for Hive client (version 1.2.2) is /cluster/spark/spark-warehouse 2021-09-23 14:30:24 Hive: INFO: **** getDatabase for dbName: default 2021-09-23 14:30:24 Hive: INFO: **** getDatabase for dbName: global_temp 2021-09-23 14:30:40 root: INFO: timing SparkBackend.parse_value_ir total 9.720ms self 9.720ms children 0.000ms %children 0.00% 2021-09-23 14:30:40 root: INFO: starting execution of query hail_query_9 of initial size 2 2021-09-23 14:30:40 root: INFO: optimize optimize: relationalLowerer, initial IR: before: IR size 2: (MatrixWrite "{\"name\":\"MatrixNativeWriter\",\"path\":\"dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt\",\"overwrite\":false,\"stageLocally\":false,\"codecSpecJSONStr\":null,\"partitions\":null,\"partitionsTypeStr\":null}" (MatrixLiteral)) 2021-09-23 14:30:40 root: INFO: optimize optimize: relationalLowerer, initial IR: after: IR size 2: (MatrixWrite "{\"name\":\"MatrixNativeWriter\",\"path\":\"dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt\",\"overwrite\":false,\"stageLocally\":false,\"codecSpecJSONStr\":null,\"partitions\":null,\"partitionsTypeStr\":null}" (MatrixLiteral)) 2021-09-23 14:30:40 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: before: IR size 2: (TableWrite "{\"writer\":{\"path\":\"dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt\",\"overwrite\":false,\"stageLocally\":false,\"codecSpecJSONStr\":null,\"partitions\":null,\"partitionsTypeStr\":null},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}" (TableLiteral)) 2021-09-23 14:30:40 root: INFO: optimize optimize: relationalLowerer, after LowerMatrixToTable: after: IR size 2: (TableWrite "{\"writer\":{\"path\":\"dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt\",\"overwrite\":false,\"stageLocally\":false,\"codecSpecJSONStr\":null,\"partitions\":null,\"partitionsTypeStr\":null},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}" (TableLiteral)) 2021-09-23 14:30:40 root: INFO: interpreting non compilable node: TableWrite 2021-09-23 14:30:40 root: INFO: decoder cache miss (0 hits, 1 misses, 0.000 2021-09-23 14:30:40 root: INFO: instruction count: 3: __C70etypeDecode. 2021-09-23 14:30:40 root: INFO: instruction count: 5: __C70etypeDecode.apply 2021-09-23 14:30:40 root: INFO: instruction count: 23: __C70etypeDecode.__m71DECODE_r_struct_of_r_array_of_r_struct_of_r_binaryENDEND_TO_r_tuple_of_r_array_of_r_tuple_of_r_stringENDEND 2021-09-23 14:30:40 root: INFO: instruction count: 52: __C70etypeDecode.__m72INPLACE_DECODE_r_array_of_r_struct_of_r_binaryEND_TO_r_array_of_r_tuple_of_r_binaryEND 2021-09-23 14:30:40 root: INFO: instruction count: 15: __C70etypeDecode.__m73INPLACE_DECODE_r_struct_of_r_binaryEND_TO_r_tuple_of_r_binaryEND 2021-09-23 14:30:40 root: INFO: instruction count: 31: __C70etypeDecode.__m74INPLACE_DECODE_r_binary_TO_r_binary 2021-09-23 14:30:40 DNAxApiSvc: INFO: initializing DNAxApiSvc 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] Initializing file system (URI = dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt): com.dnanexus.hadoop.fs.DNAxFileSystem@228d6c41 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] inputStreamBufferSize: 33554432 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] inputStreamMaxAttempts: 6 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] outputStreamBufferSize: 33554432 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] inputStreamRetryableCodes: [500, 503] 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] fileStatusCacheEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] fileStatusCacheSiblingsEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] statusListUrlCacheEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] readFileUrlCacheEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] listStatusSignedUrlEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] fileStatusSignedUrlEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] createDirectorySemaphoreEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] fileStatusSemaphoreEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] listStatusSemaphoreEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] readFileSemaphoreEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] deleteObjectsInBatchesEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] parallelPartUploadEnabled: true 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] uploadChunkSize: 16777216 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] multiPartThreshold: 1073741824 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] semaphorePermitWaitTime: 10 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] semaphoreCreateDirectoryPermitCount: 1 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] semaphoreFileStatusPermitCount: 1 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] semaphoreListStatusPermitCount: 1 2021-09-23 14:30:40 DNAxFileSystem: INFO: [initialize] semaphoreReadFilePermitCount: 1 2021-09-23 14:30:40 routes: INFO: [getFileStatusUrl] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/getFileStatusUrl: input = {"filename":"matrix.mt","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:40 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:40 routes: INFO: [getStatusListUrl] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/getStatusListUrl: input = {"filename":"matrix.mt","marker":"","recurse":false,"maxKeys":1,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:40 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:41 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:41 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:43 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/rows/rows/parts","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:43 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:45 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/entries/rows/parts","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:45 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:47 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/index","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:30:47 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:30:49 root: INFO: encoder cache miss (0 hits, 3 misses, 0.000) 2021-09-23 14:30:49 root: INFO: instruction count: 3: __C75etypeEncode. 2021-09-23 14:30:49 root: INFO: instruction count: 5: __C75etypeEncode.apply 2021-09-23 14:30:49 root: INFO: instruction count: 113: __C75etypeEncode.__m76ENCODE_r_tuple_of_r_locusANDr_array_of_r_stringANDo_stringANDo_float64ANDo_set_of_r_stringANDr_tuple_of_o_array_of_o_float64ANDo_array_of_o_int32ANDo_array_of_o_int32ANDo_int32ENDANDr_array_of_r_tuple_of_o_callANDo_array_of_o_stringANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32ENDEND_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryANDo_binaryANDo_float64ANDo_array_of_r_binaryANDr_struct_of_o_array_of_o_float64ANDo_array_of_o_int32ANDo_array_of_o_int32ANDo_int32ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 23: __C75etypeEncode.__m77ENCODE_r_tuple_of_r_binaryANDr_int32END_TO_r_struct_of_r_binaryANDr_int32END 2021-09-23 14:30:49 root: INFO: instruction count: 16: __C75etypeEncode.__m78ENCODE_r_binary_TO_r_binary 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C75etypeEncode.__m79ENCODE_r_int32_TO_r_int32 2021-09-23 14:30:49 root: INFO: instruction count: 53: __C75etypeEncode.__m80ENCODE_r_array_of_r_binary_TO_r_array_of_r_binary 2021-09-23 14:30:49 root: INFO: instruction count: 16: __C75etypeEncode.__m81ENCODE_o_binary_TO_o_binary 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C75etypeEncode.__m82ENCODE_o_float64_TO_o_float64 2021-09-23 14:30:49 root: INFO: instruction count: 53: __C75etypeEncode.__m83ENCODE_o_array_of_r_binary_TO_o_array_of_r_binary 2021-09-23 14:30:49 root: INFO: instruction count: 91: __C75etypeEncode.__m84ENCODE_r_tuple_of_o_array_of_o_float64ANDo_array_of_o_int32ANDo_array_of_o_int32ANDo_int32END_TO_r_struct_of_o_array_of_o_float64ANDo_array_of_o_int32ANDo_array_of_o_int32ANDo_int32END 2021-09-23 14:30:49 root: INFO: instruction count: 129: __C75etypeEncode.__m85ENCODE_o_array_of_o_float64_TO_o_array_of_o_float64 2021-09-23 14:30:49 root: INFO: instruction count: 129: __C75etypeEncode.__m86ENCODE_o_array_of_o_int32_TO_o_array_of_o_int32 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C75etypeEncode.__m87ENCODE_o_int32_TO_o_int32 2021-09-23 14:30:49 root: INFO: encoder cache miss (0 hits, 4 misses, 0.000) 2021-09-23 14:30:49 root: INFO: instruction count: 3: __C88etypeEncode. 2021-09-23 14:30:49 root: INFO: instruction count: 5: __C88etypeEncode.apply 2021-09-23 14:30:49 root: INFO: instruction count: 13: __C88etypeEncode.__m89ENCODE_r_tuple_of_r_locusANDr_array_of_r_stringANDo_stringANDo_float64ANDo_set_of_r_stringANDr_tuple_of_o_array_of_o_float64ANDo_array_of_o_int32ANDo_array_of_o_int32ANDo_int32ENDANDr_array_of_r_tuple_of_o_callANDo_array_of_o_stringANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32ENDEND_TO_r_struct_of_r_array_of_r_struct_of_o_int32ANDo_array_of_o_binaryANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 49: __C88etypeEncode.__m90ENCODE_r_array_of_r_tuple_of_o_int32ANDo_array_of_o_binaryANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32END_TO_r_array_of_r_struct_of_o_int32ANDo_array_of_o_binaryANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32END 2021-09-23 14:30:49 root: INFO: instruction count: 129: __C88etypeEncode.__m91ENCODE_r_tuple_of_o_int32ANDo_array_of_o_binaryANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32END_TO_r_struct_of_o_int32ANDo_array_of_o_binaryANDo_int32ANDo_array_of_o_int32ANDo_int32ANDo_array_of_o_int32END 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C88etypeEncode.__m92ENCODE_o_int32_TO_o_int32 2021-09-23 14:30:49 root: INFO: instruction count: 129: __C88etypeEncode.__m93ENCODE_o_array_of_o_binary_TO_o_array_of_o_binary 2021-09-23 14:30:49 root: INFO: instruction count: 16: __C88etypeEncode.__m94ENCODE_o_binary_TO_o_binary 2021-09-23 14:30:49 root: INFO: instruction count: 129: __C88etypeEncode.__m95ENCODE_o_array_of_o_int32_TO_o_array_of_o_int32 2021-09-23 14:30:49 root: INFO: instruction count: 3: __C96indexwriter. 2021-09-23 14:30:49 root: INFO: instruction count: 119: __C96indexwriter.apply 2021-09-23 14:30:49 root: INFO: instruction count: 332: __C96indexwriter.__m105writeInternalNode 2021-09-23 14:30:49 root: INFO: instruction count: 13: __C96indexwriter.__m106ENCODE_o_tuple_of_r_array_of_r_tuple_of_r_int64ANDr_int64ANDr_tuple_of_r_locusENDANDr_int64ANDr_tuple_of_o_int64ENDENDEND_TO_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 49: __C96indexwriter.__m107ENCODE_r_array_of_r_tuple_of_r_int64ANDr_int64ANDr_tuple_of_r_tuple_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_tuple_of_o_int64ENDEND_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDr_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 45: __C96indexwriter.__m108ENCODE_r_tuple_of_r_int64ANDr_int64ANDr_tuple_of_r_tuple_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_tuple_of_o_int64ENDEND_TO_r_struct_of_r_int64ANDr_int64ANDr_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C96indexwriter.__m109ENCODE_r_int64_TO_r_int64 2021-09-23 14:30:49 root: INFO: instruction count: 9: __C96indexwriter.__m110ENCODE_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDEND_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 23: __C96indexwriter.__m111ENCODE_r_tuple_of_r_binaryANDr_int32END_TO_r_struct_of_r_binaryANDr_int32END 2021-09-23 14:30:49 root: INFO: instruction count: 16: __C96indexwriter.__m112ENCODE_r_binary_TO_r_binary 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C96indexwriter.__m113ENCODE_r_int32_TO_r_int32 2021-09-23 14:30:49 root: INFO: instruction count: 34: __C96indexwriter.__m114ENCODE_r_tuple_of_o_int64END_TO_r_struct_of_o_int64END 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C96indexwriter.__m115ENCODE_o_int64_TO_o_int64 2021-09-23 14:30:49 root: INFO: instruction count: 266: __C96indexwriter.__m116writeLeafNode 2021-09-23 14:30:49 root: INFO: instruction count: 23: __C96indexwriter.__m117ENCODE_o_tuple_of_r_int64ANDr_array_of_r_tuple_of_r_tuple_of_r_locusENDANDr_int64ANDr_tuple_of_o_int64ENDENDEND_TO_o_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 49: __C96indexwriter.__m118ENCODE_r_array_of_r_tuple_of_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_tuple_of_o_int64ENDEND_TO_r_array_of_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 25: __C96indexwriter.__m119ENCODE_r_tuple_of_r_tuple_of_r_tuple_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_tuple_of_o_int64ENDEND_TO_r_struct_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDENDANDr_int64ANDr_struct_of_o_int64ENDEND 2021-09-23 14:30:49 root: INFO: instruction count: 52: __C96indexwriter.__m120flush 2021-09-23 14:30:49 root: INFO: instruction count: 150: __C96indexwriter.init 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C96indexwriter.setObjects 2021-09-23 14:30:49 root: INFO: instruction count: 28: __C96indexwriter.close 2021-09-23 14:30:49 root: INFO: instruction count: 7: __C96indexwriter.trackedOS 2021-09-23 14:30:49 root: INFO: instruction count: 9: __C96indexwriter.setPartitionIndex 2021-09-23 14:30:49 root: INFO: instruction count: 4: __C96indexwriter.addPartitionRegion 2021-09-23 14:30:49 root: INFO: instruction count: 3: __C122FSContainer. 2021-09-23 14:30:49 root: INFO: instruction count: 28: __C122FSContainer.init_filesystem 2021-09-23 14:30:49 root: INFO: instruction count: 3: __C122FSContainer. 2021-09-23 14:30:49 SparkContext: INFO: Starting job: collect at ContextRDD.scala:166 2021-09-23 14:30:49 DAGScheduler: INFO: [DNAX] Setting local property com.dnanexus.auth 2021-09-23 14:30:49 DAGScheduler: INFO: Unable to extract and set 'user' or 'containsPHI' in SparkSession 2021-09-23 14:30:49 DAGScheduler: INFO: Got job 1 (collect at ContextRDD.scala:166) with 1 output partitions 2021-09-23 14:30:49 DAGScheduler: INFO: Final stage: ResultStage 1 (collect at ContextRDD.scala:166) 2021-09-23 14:30:49 DAGScheduler: INFO: Parents of final stage: List() 2021-09-23 14:30:49 DAGScheduler: INFO: Missing parents: List() 2021-09-23 14:30:49 DAGScheduler: INFO: Submitting ResultStage 1 (MapPartitionsRDD[8] at mapPartitions at ContextRDD.scala:160), which has no missing parents 2021-09-23 14:30:49 MemoryStore: INFO: Block broadcast_6 stored as values in memory (estimated size 2.2 MB, free 11.5 GB) 2021-09-23 14:30:49 MemoryStore: INFO: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1479.4 KB, free 11.5 GB) 2021-09-23 14:30:49 BlockManagerInfo: INFO: Added broadcast_6_piece0 in memory on ip-10-60-95-0.eu-west-2.compute.internal:43001 (size: 1479.4 KB, free: 11.5 GB) 2021-09-23 14:30:49 SparkContext: INFO: Created broadcast 6 from broadcast at DAGScheduler.scala:1268 2021-09-23 14:30:49 DAGScheduler: INFO: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at mapPartitions at ContextRDD.scala:160) (first 15 tasks are for partitions Vector(0)) 2021-09-23 14:30:49 TaskSchedulerImpl: INFO: Adding task set 1.0 with 1 tasks 2021-09-23 14:30:49 TaskSetManager: INFO: Starting task 0.0 in stage 1.0 (TID 1, ip-10-60-146-200.eu-west-2.compute.internal, executor 0, partition 0, PROCESS_LOCAL, 7896 bytes) 2021-09-23 14:30:49 BlockManagerInfo: INFO: Added broadcast_6_piece0 in memory on ip-10-60-146-200.eu-west-2.compute.internal:44000 (size: 1479.4 KB, free: 11.5 GB) 2021-09-23 14:30:53 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on ip-10-60-146-200.eu-west-2.compute.internal:44000 (size: 123.7 KB, free: 11.5 GB) 2021-09-23 14:35:54 TaskSetManager: INFO: Finished task 0.0 in stage 1.0 (TID 1) in 305051 ms on ip-10-60-146-200.eu-west-2.compute.internal (executor 0) (1/1) 2021-09-23 14:35:54 TaskSchedulerImpl: INFO: Removed TaskSet 1.0, whose tasks have all completed, from pool 2021-09-23 14:35:54 DAGScheduler: INFO: ResultStage 1 (collect at ContextRDD.scala:166) finished in 305.103 s 2021-09-23 14:35:54 DAGScheduler: INFO: Job 1 finished: collect at ContextRDD.scala:166, took 305.106476 s 2021-09-23 14:35:54 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/rows/rows/metadata.json.gz","isMulti":false,"index":1,"md5":"97710cb7c6b9eae0278c3ecf977371e2","size":649,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:35:54 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:35:55 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/rows/rows/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:35:55 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:35:55 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/entries/rows/metadata.json.gz","isMulti":false,"index":1,"md5":"5072f3e847439bb45666a06a3d2cf3ef","size":630,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:35:55 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:35:56 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/entries/rows/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:35:56 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:35:56 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/globals","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:35:56 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:35:58 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/globals/rows/parts","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:35:58 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 36 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 41 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 49 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 27 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 31 2021-09-23 14:36:00 BlockManagerInfo: INFO: Removed broadcast_6_piece0 on ip-10-60-95-0.eu-west-2.compute.internal:43001 in memory (size: 1479.4 KB, free: 11.5 GB) 2021-09-23 14:36:00 BlockManagerInfo: INFO: Removed broadcast_6_piece0 on ip-10-60-146-200.eu-west-2.compute.internal:44000 in memory (size: 1479.4 KB, free: 11.5 GB) 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 51 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 38 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 42 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 44 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 46 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 50 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 40 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 34 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 43 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 47 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 45 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 52 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 37 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 30 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 28 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 33 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 32 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 35 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 29 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 39 2021-09-23 14:36:00 ContextCleaner: INFO: Cleaned accumulator 48 2021-09-23 14:36:00 root: INFO: encoder cache miss (0 hits, 5 misses, 0.000) 2021-09-23 14:36:00 root: INFO: instruction count: 3: __C125etypeEncode. 2021-09-23 14:36:00 root: INFO: instruction count: 5: __C125etypeEncode.apply 2021-09-23 14:36:00 root: INFO: instruction count: 3: __C125etypeEncode.__m126ENCODE_r_tuple_of_END_TO_r_struct_of_END 2021-09-23 14:36:00 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/rows/parts/part-0","isMulti":false,"index":1,"md5":"087d71efbce2d8c8d96488a947a1c53d","size":11,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:00 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:01 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/globals/rows/parts/part-0","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:01 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:02 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/rows/metadata.json.gz","isMulti":false,"index":1,"md5":"cd82fbbfa4e395cefdc5824e3c2a154f","size":240,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:02 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:02 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/globals/rows/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:02 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:03 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/globals/globals/parts","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:03 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:05 root: INFO: encoder cache hit 2021-09-23 14:36:05 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/globals/parts/part-0","isMulti":false,"index":1,"md5":"087d71efbce2d8c8d96488a947a1c53d","size":11,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:05 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:05 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/globals/globals/parts/part-0","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:05 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:06 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/globals/metadata.json.gz","isMulti":false,"index":1,"md5":"cd82fbbfa4e395cefdc5824e3c2a154f","size":240,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:06 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:06 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/globals/globals/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:06 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:07 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/metadata.json.gz","isMulti":false,"index":1,"md5":"f4aaa5c1aa3c2157a52e8c0ada858f28","size":220,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:07 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:07 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/globals/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:07 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:08 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/globals/_SUCCESS","isMulti":false,"index":1,"md5":"d41d8cd98f00b204e9800998ecf8427e","size":0,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:08 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:08 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/rows/metadata.json.gz","isMulti":false,"index":1,"md5":"d5be6b641e7af3e479f8740cdf634956","size":314,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:08 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:09 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/rows/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:09 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:10 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/rows/_SUCCESS","isMulti":false,"index":1,"md5":"d41d8cd98f00b204e9800998ecf8427e","size":0,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:10 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:10 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/entries/metadata.json.gz","isMulti":false,"index":1,"md5":"6f2fe6573ab322f3b108a9afb0ad8f29","size":313,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:10 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:10 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/entries/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:10 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:11 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/entries/_SUCCESS","isMulti":false,"index":1,"md5":"d41d8cd98f00b204e9800998ecf8427e","size":0,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:11 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:11 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/cols","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:11 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:13 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/cols/rows/parts","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:13 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:15 root: INFO: encoder cache miss (1 hits, 6 misses, 0.143) 2021-09-23 14:36:15 root: INFO: instruction count: 3: __C127etypeEncode. 2021-09-23 14:36:15 root: INFO: instruction count: 5: __C127etypeEncode.apply 2021-09-23 14:36:15 root: INFO: instruction count: 13: __C127etypeEncode.__m128ENCODE_r_tuple_of_r_stringEND_TO_r_struct_of_r_binaryEND 2021-09-23 14:36:15 root: INFO: instruction count: 16: __C127etypeEncode.__m129ENCODE_r_binary_TO_r_binary 2021-09-23 14:36:16 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/cols/rows/parts/part-0","isMulti":false,"index":1,"md5":"494723d65b2269772ec3307f13f81967","size":1185150,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:16 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:16 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/cols/rows/parts/part-0","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:16 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:17 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/cols/rows/metadata.json.gz","isMulti":false,"index":1,"md5":"d3e1fe9ddf9b843cc9c056c6d4c0bfb4","size":253,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:17 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:17 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/cols/rows/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:17 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:18 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/cols/metadata.json.gz","isMulti":false,"index":1,"md5":"1b29856b604e3a61e59b21eb00bde849","size":234,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:18 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:18 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/cols/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:18 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:19 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/cols/_SUCCESS","isMulti":false,"index":1,"md5":"d41d8cd98f00b204e9800998ecf8427e","size":0,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:19 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:19 routes: INFO: [createDirectory] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/createDirectory: input = {"filename":"matrix.mt/references","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:19 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:21 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/metadata.json.gz","isMulti":false,"index":1,"md5":"1944c19660671c946975ae28b6212532","size":383,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:21 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:22 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/metadata.json.gz","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:22 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:22 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/README.txt","isMulti":false,"index":1,"md5":"18c821613cbf0c92188497c8cc10aa03","size":147,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:22 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:23 routes: INFO: [joinFileParts] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/joinFileParts: input = {"filename":"matrix.mt/README.txt","numParts":1,"isMulti":false,"jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"},"extras":{}} 2021-09-23 14:36:23 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:24 routes: INFO: [writeFile] Calling api server route /database-G568z00J860f64kZ4vXkjzJ6/writeFile: input = {"filename":"matrix.mt/_SUCCESS","isMulti":false,"index":1,"md5":"d41d8cd98f00b204e9800998ecf8427e","size":0,"projectContext":"project-G493Kx8J860q68yGFpxkz18y","jobInfo":{"dxJobId":"job-G568p1jJ860xpfZ94Yf78PYb"}} 2021-09-23 14:36:24 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2021-09-23 14:36:24 Hail: INFO: wrote matrix table with 828 rows and 200643 columns in 1 partition to dnax://database-G568z00J860f64kZ4vXkjzJ6/matrix.mt Total size: 452.53 MiB * Rows/entries: 451.40 MiB * Columns: 1.13 MiB * Globals: 11.00 B * Smallest partition: 828 rows (451.40 MiB) * Largest partition: 828 rows (451.40 MiB) 2021-09-23 14:36:24 root: INFO: took 5m44.3s 2021-09-23 14:36:24 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: before: IR size 1: (Begin) 2021-09-23 14:36:24 root: INFO: optimize optimize: relationalLowerer, after InterpretNonCompilable: after: IR size 1: (Begin) 2021-09-23 14:36:24 root: INFO: finished execution of query hail_query_9 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON total 5m44.3s self 1.071ms children 5m44.3s %children 100.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR total 4.732ms self 0.016ms children 4.716ms %children 99.66% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.016ms self 0.016ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation total 4.694ms self 1.303ms children 3.390ms %children 72.23% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize total 3.390ms self 0.039ms children 3.351ms %children 98.85% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/FoldConstants total 0.079ms self 0.079ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.065ms self 0.065ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/Simplify total 2.148ms self 2.148ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardLets total 0.620ms self 0.620ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/ForwardRelationalLets total 0.066ms self 0.066ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/LoweringTransformation/Optimize/PruneDeadFields total 0.374ms self 0.374ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, initial IR/Verify total 0.006ms self 0.006ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable total 1.698ms self 0.006ms children 1.693ms %children 99.67% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.001ms self 0.001ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/LoweringTransformation total 1.657ms self 1.657ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/LowerMatrixToTable/Verify total 0.034ms self 0.034ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable total 9.011ms self 0.005ms children 9.006ms %children 99.95% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.002ms self 0.002ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation total 8.999ms self 3.876ms children 5.124ms %children 56.93% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize total 5.124ms self 0.058ms children 5.066ms %children 98.87% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/FoldConstants total 0.055ms self 0.055ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.076ms self 0.076ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/Simplify total 4.355ms self 4.355ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardLets total 0.215ms self 0.215ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.020ms self 0.020ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/LoweringTransformation/Optimize/PruneDeadFields total 0.345ms self 0.345ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after LowerMatrixToTable/Verify total 0.005ms self 0.005ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable total 5m44.3s self 0.006ms children 5m44.3s %children 100.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/LoweringTransformation total 5m44.3s self 5m44.3s children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/InterpretNonCompilable/Verify total 0.027ms self 0.027ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable total 0.741ms self 0.004ms children 0.736ms %children 99.41% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation total 0.730ms self 0.223ms children 0.507ms %children 69.47% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize total 0.507ms self 0.060ms children 0.448ms %children 88.26% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/FoldConstants total 0.045ms self 0.045ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ExtractIntervalFilters total 0.189ms self 0.189ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/Simplify total 0.033ms self 0.033ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardLets total 0.114ms self 0.114ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/ForwardRelationalLets total 0.018ms self 0.018ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/LoweringTransformation/Optimize/PruneDeadFields total 0.049ms self 0.049ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/optimize: relationalLowerer, after InterpretNonCompilable/Verify total 0.003ms self 0.003ms children 0.000ms %children 0.00% 2021-09-23 14:36:24 root: INFO: timing SparkBackend.executeJSON/convertRegionValueToAnnotation total 0.004ms self 0.004ms children 0.000ms %children 0.00% 2021-09-23 14:37:03 DNAxApiSvc: WARN: Shutting down Runtime service for Connection Pools 2021-09-23 14:37:03 DNAxApiSvc: INFO: shutting down httpClientConnManager 2021-09-23 14:37:03 DNAxApiSvc: INFO: shutting down httpsClientConnManager 2021-09-23 14:37:03 SparkContext: INFO: Invoking stop() from shutdown hook 2021-09-23 14:37:03 AbstractConnector: INFO: Stopped Spark@1104078f{HTTP/1.1,[http/1.1]}{0.0.0.0:8081} 2021-09-23 14:37:03 SparkUI: INFO: Stopped Spark web UI at http://ip-10-60-95-0.eu-west-2.compute.internal:8081 2021-09-23 14:37:03 StandaloneSchedulerBackend: INFO: Shutting down all executors 2021-09-23 14:37:03 CoarseGrainedSchedulerBackend$DriverEndpoint: INFO: Asking each executor to shut down 2021-09-23 14:37:03 MapOutputTrackerMasterEndpoint: INFO: MapOutputTrackerMasterEndpoint stopped! 2021-09-23 14:37:03 MemoryStore: INFO: MemoryStore cleared 2021-09-23 14:37:03 BlockManager: INFO: BlockManager stopped 2021-09-23 14:37:03 BlockManagerMaster: INFO: BlockManagerMaster stopped 2021-09-23 14:37:03 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: INFO: OutputCommitCoordinator stopped! 2021-09-23 14:37:03 SparkContext: INFO: Successfully stopped SparkContext 2021-09-23 14:37:03 ShutdownHookManager: INFO: Shutdown hook called 2021-09-23 14:37:03 ShutdownHookManager: INFO: Deleting directory /tmp/spark-aa491c6b-c548-415d-ac58-51d6700b6887 2021-09-23 14:37:03 ShutdownHookManager: INFO: Deleting directory /tmp/spark-aa491c6b-c548-415d-ac58-51d6700b6887/pyspark-7d70f0e6-8085-44bd-accf-35d546dc4631 2021-09-23 14:37:03 ShutdownHookManager: INFO: Deleting directory /tmp/spark-41a9aee3-519a-462d-b0c8-ff9c8ec7e17b 2021-09-23 14:37:03 DNAxFileSystem: INFO: [close] Closing file system (uri = dnax://Bulk) com.dnanexus.hadoop.fs.DNAxFileSystem@8eea6d6 2021-09-23 14:37:03 DNAxFileSystem: INFO: [close] Closing file system (uri = dnax://database-G568z00J860f64kZ4vXkjzJ6) com.dnanexus.hadoop.fs.DNAxFileSystem@228d6c41 2021-09-23 14:37:03 DNAxFileSystem: INFO: [close] Closing file system (uri = dnax:///mnt/project/Bulk/Exome%20sequences/Population%20level%20exome%20OQFE%20variants,%20pVCF%20format/ukb23156_c19_b46_v1.vcf.gz) com.dnanexus.hadoop.fs.DNAxFileSystem@28d874e3