Index_bgen() on UKBB imputed data expected time

Hello Hail team,

I’m trying to index UKBBs BGEN imputed genotyping data with index_bgen(), but it’s taking quite some time. For example, the indexing for the chr22 file (36.6 GB) processed for >6h and didn’t finished.

I also made a test with a small file (1.000 variants) and it was processed in a reasonable time.

Is there some expected time to complete the index_bgen() in these larger files? Or maybe a way to speed up this process.

Thank you for the support!
Rodrigo

do you have a hail log file for the slow pipeline?

I’m afraid not. The slow pipeline broke just after the 6h for what appears to be a unrelated issue.

We are running it on a Amazon EMR Jupyter Notebook with PySpark, so I guess this specific log file was lost when the cluster was terminated.

I can try to generate it again if necessary though.

Hello again,

I’ve managed to obtain a hail log from the same pipeline execution. I hope this helps.

Thank you all,

2022-01-12 14:49:45 Hail: INFO: Running Hail version 0.2.78-b17627756568
2022-01-12 14:49:45 SharedState: INFO: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('hdfs:///user/spark/warehouse').
2022-01-12 14:49:45 SharedState: INFO: Warehouse path is 'hdfs:///user/spark/warehouse'.
2022-01-12 14:49:45 ServerInfo: INFO: Adding filter to /SQL: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2022-01-12 14:49:45 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@8b80fca{/SQL,null,AVAILABLE,@Spark}
2022-01-12 14:49:45 ServerInfo: INFO: Adding filter to /SQL/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2022-01-12 14:49:45 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@756976cc{/SQL/json,null,AVAILABLE,@Spark}
2022-01-12 14:49:45 ServerInfo: INFO: Adding filter to /SQL/execution: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2022-01-12 14:49:45 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@4ab04c04{/SQL/execution,null,AVAILABLE,@Spark}
2022-01-12 14:49:45 ServerInfo: INFO: Adding filter to /SQL/execution/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2022-01-12 14:49:45 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@562fdca4{/SQL/execution/json,null,AVAILABLE,@Spark}
2022-01-12 14:49:45 ServerInfo: INFO: Adding filter to /static/sql: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2022-01-12 14:49:45 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@109f0478{/static/sql,null,AVAILABLE,@Spark}
2022-01-12 14:49:46 SparkSession$Builder: WARN: Using an existing SparkSession; the static sql configurations will not take effect.
2022-01-12 14:49:46 SparkSession$Builder: WARN: Using an existing SparkSession; some spark core configurations may not take effect.
2022-01-12 14:49:47 root: INFO: RegionPool: initialized for thread 112: Thread-60
2022-01-12 14:49:48 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 336.9 KiB, free 912.0 MiB)
2022-01-12 14:49:48 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.4 KiB, free 911.9 MiB)
2022-01-12 14:49:48 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on ip-xx-xx-xx-xxx.ec2.internal:39933 (size: 32.4 KiB, free: 912.3 MiB)
2022-01-12 14:49:48 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:278
2022-01-12 14:49:49 ClientConfigurationFactory: INFO: Set initial getObject socket timeout to 2000 ms.
2022-01-12 14:49:50 root: INFO: globbing path s3://bucket_name/folder_for_microarray_imputed/raw/ukb22828_chr22_v3.bgen returned 1 files: ukb22828_chr22_v3.bgen
2022-01-12 14:49:50 S3NativeFileSystem: INFO: Opening 's3://bucket_name/folder_for_microarray_imputed/raw/ukb22828_chr22_v3.bgen' for reading
2022-01-12 14:49:50 MemoryStore: INFO: Block broadcast_1 stored as values in memory (estimated size 34.7 KiB, free 911.9 MiB)
2022-01-12 14:49:50 MemoryStore: INFO: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 911.9 MiB)
2022-01-12 14:49:50 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on ip-xx-xx-xx-xxx.ec2.internal:39933 (size: 3.2 KiB, free: 912.3 MiB)
2022-01-12 14:49:50 SparkContext: INFO: Created broadcast 1 from broadcast at SparkBackend.scala:278
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C1FSContainer.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C1FSContainer.<clinit>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C2bgen_rdd_decoder.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 437: __C2bgen_rdd_decoder.apply
2022-01-12 14:49:51 root: INFO: instruction count: 16: __C2bgen_rdd_decoder.apply
2022-01-12 14:49:51 root: INFO: instruction count: 9: __C2bgen_rdd_decoder.setPartitionIndex
2022-01-12 14:49:51 root: INFO: instruction count: 4: __C2bgen_rdd_decoder.addPartitionRegion
2022-01-12 14:49:51 root: INFO: instruction count: 4: __C2bgen_rdd_decoder.setPool
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C2bgen_rdd_decoder.addFS
2022-01-12 14:49:51 root: INFO: decoder cache miss (0 hits, 1 misses, 0.000
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C13FSContainer.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C13FSContainer.<clinit>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C14etypeDecode.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 9: __C14etypeDecode.apply
2022-01-12 14:49:51 root: INFO: instruction count: 34: __C14etypeDecode.__m15DECODE_o_struct_of_r_int64ANDr_array_of_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND_TO_SBaseStructPointer
2022-01-12 14:49:51 root: INFO: instruction count: 10: __C14etypeDecode.__m16INPLACE_DECODE_r_int64_TO_r_int64
2022-01-12 14:49:51 root: INFO: instruction count: 64: __C14etypeDecode.__m17INPLACE_DECODE_r_array_of_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_array_of_r_struct_of_o_struct_of_o_locusANDo_array_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:51 root: INFO: instruction count: 60: __C14etypeDecode.__m18INPLACE_DECODE_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_struct_of_o_struct_of_o_locusANDo_array_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:51 root: INFO: instruction count: 71: __C14etypeDecode.__m19INPLACE_DECODE_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryEND_TO_o_struct_of_o_locusANDo_array_of_o_stringEND
2022-01-12 14:49:51 root: INFO: instruction count: 22: __C14etypeDecode.__m20INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus
2022-01-12 14:49:51 root: INFO: instruction count: 33: __C14etypeDecode.__m21INPLACE_DECODE_r_binary_TO_r_string
2022-01-12 14:49:51 root: INFO: instruction count: 10: __C14etypeDecode.__m22INPLACE_DECODE_r_int32_TO_r_int32
2022-01-12 14:49:51 root: INFO: instruction count: 120: __C14etypeDecode.__m23INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string
2022-01-12 14:49:51 root: INFO: instruction count: 33: __C14etypeDecode.__m24INPLACE_DECODE_o_binary_TO_o_string
2022-01-12 14:49:51 root: INFO: instruction count: 8: __C14etypeDecode.__m25INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END
2022-01-12 14:49:51 root: INFO: decoder cache miss (0 hits, 2 misses, 0.000
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C26FSContainer.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C26FSContainer.<clinit>
2022-01-12 14:49:51 root: INFO: instruction count: 3: __C27etypeDecode.<init>
2022-01-12 14:49:51 root: INFO: instruction count: 9: __C27etypeDecode.apply
2022-01-12 14:49:51 root: INFO: instruction count: 27: __C27etypeDecode.__m28DECODE_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND_TO_SBaseStructPointer
2022-01-12 14:49:51 root: INFO: instruction count: 64: __C27etypeDecode.__m29INPLACE_DECODE_r_array_of_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_locusANDo_array_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:51 root: INFO: instruction count: 74: __C27etypeDecode.__m30INPLACE_DECODE_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND_TO_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_locusANDo_array_of_o_stringENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:51 root: INFO: instruction count: 10: __C27etypeDecode.__m31INPLACE_DECODE_r_int64_TO_r_int64
2022-01-12 14:49:51 root: INFO: instruction count: 71: __C27etypeDecode.__m32INPLACE_DECODE_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryEND_TO_o_struct_of_o_locusANDo_array_of_o_stringEND
2022-01-12 14:49:51 root: INFO: instruction count: 22: __C27etypeDecode.__m33INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus
2022-01-12 14:49:51 root: INFO: instruction count: 33: __C27etypeDecode.__m34INPLACE_DECODE_r_binary_TO_r_string
2022-01-12 14:49:51 root: INFO: instruction count: 10: __C27etypeDecode.__m35INPLACE_DECODE_r_int32_TO_r_int32
2022-01-12 14:49:51 root: INFO: instruction count: 120: __C27etypeDecode.__m36INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string
2022-01-12 14:49:51 root: INFO: instruction count: 33: __C27etypeDecode.__m37INPLACE_DECODE_o_binary_TO_o_string
2022-01-12 14:49:51 root: INFO: instruction count: 8: __C27etypeDecode.__m38INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C39FSContainer.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C39FSContainer.<clinit>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C40indexwriter.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 372: __C40indexwriter.apply
2022-01-12 14:49:52 root: INFO: instruction count: 674: __C40indexwriter.__m49writeInternalNode
2022-01-12 14:49:52 root: INFO: instruction count: 37: __C40indexwriter.__m50ENCODE_SBaseStructPointer_TO_o_struct_of_r_array_of_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 37: __C40indexwriter.__m51ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 86: __C40indexwriter.__m52ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64ANDr_int64ANDo_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C40indexwriter.__m53ENCODE_SInt64$_TO_r_int64
2022-01-12 14:49:52 root: INFO: instruction count: 137: __C40indexwriter.__m54ENCODE_SBaseStructPointer_TO_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryEND
2022-01-12 14:49:52 root: INFO: instruction count: 29: __C40indexwriter.__m55ENCODE_SCanonicalLocusPointer_TO_o_struct_of_r_binaryANDr_int32END
2022-01-12 14:49:52 root: INFO: instruction count: 16: __C40indexwriter.__m56ENCODE_SStringPointer_TO_r_binary
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C40indexwriter.__m57ENCODE_SInt32$_TO_r_int32
2022-01-12 14:49:52 root: INFO: instruction count: 117: __C40indexwriter.__m58ENCODE_SIndexablePointer_TO_o_array_of_o_binary
2022-01-12 14:49:52 root: INFO: instruction count: 16: __C40indexwriter.__m59ENCODE_SStringPointer_TO_o_binary
2022-01-12 14:49:52 root: INFO: instruction count: 1: __C40indexwriter.__m60ENCODE_SBaseStructPointer_TO_r_struct_of_END
2022-01-12 14:49:52 root: INFO: instruction count: 574: __C40indexwriter.__m62writeLeafNode
2022-01-12 14:49:52 root: INFO: instruction count: 51: __C40indexwriter.__m63ENCODE_SBaseStructPointer_TO_o_struct_of_r_int64ANDr_array_of_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 37: __C40indexwriter.__m64ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 58: __C40indexwriter.__m65ENCODE_SBaseStructPointer_TO_r_struct_of_o_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryENDANDr_int64ANDr_struct_of_ENDEND
2022-01-12 14:49:52 root: INFO: instruction count: 52: __C40indexwriter.__m66flush
2022-01-12 14:49:52 root: INFO: instruction count: 182: __C40indexwriter.init
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C40indexwriter.setObjects
2022-01-12 14:49:52 root: INFO: instruction count: 28: __C40indexwriter.close
2022-01-12 14:49:52 root: INFO: instruction count: 7: __C40indexwriter.trackedOS
2022-01-12 14:49:52 root: INFO: instruction count: 9: __C40indexwriter.setPartitionIndex
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C40indexwriter.addPartitionRegion
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C40indexwriter.setPool
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C40indexwriter.addFS
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C70__m49writeInternalNodeSpills.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C73__m62writeLeafNodeSpills.<init>
2022-01-12 14:49:52 MemoryStore: INFO: Block broadcast_2 stored as values in memory (estimated size 60.6 KiB, free 911.8 MiB)
2022-01-12 14:49:52 MemoryStore: INFO: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1495.0 B, free 911.8 MiB)
2022-01-12 14:49:52 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on ip-xx-xx-xx-xxx.ec2.internal:39933 (size: 1495.0 B, free: 912.3 MiB)
2022-01-12 14:49:52 SparkContext: INFO: Created broadcast 2 from broadcast at RVDPartitioner.scala:93
2022-01-12 14:49:52 root: INFO: encoder cache miss (0 hits, 1 misses, 0.000)
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C74FSContainer.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C74FSContainer.<clinit>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C75etypeEncode.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 9: __C75etypeEncode.apply
2022-01-12 14:49:52 root: INFO: instruction count: 187: __C75etypeEncode.__m76ENCODE_SBaseStructPointer_TO_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryANDo_int64ANDo_int32END
2022-01-12 14:49:52 root: INFO: instruction count: 29: __C75etypeEncode.__m77ENCODE_SCanonicalLocusPointer_TO_o_struct_of_r_binaryANDr_int32END
2022-01-12 14:49:52 root: INFO: instruction count: 16: __C75etypeEncode.__m78ENCODE_SStringPointer_TO_r_binary
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C75etypeEncode.__m79ENCODE_SInt32$_TO_r_int32
2022-01-12 14:49:52 root: INFO: instruction count: 117: __C75etypeEncode.__m80ENCODE_SIndexablePointer_TO_o_array_of_o_binary
2022-01-12 14:49:52 root: INFO: instruction count: 16: __C75etypeEncode.__m81ENCODE_SStringPointer_TO_o_binary
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C75etypeEncode.__m82ENCODE_SInt64$_TO_o_int64
2022-01-12 14:49:52 root: INFO: instruction count: 4: __C75etypeEncode.__m83ENCODE_SInt32$_TO_o_int32
2022-01-12 14:49:52 root: INFO: decoder cache miss (0 hits, 3 misses, 0.000
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C84FSContainer.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C84FSContainer.<clinit>
2022-01-12 14:49:52 root: INFO: instruction count: 3: __C85etypeDecode.<init>
2022-01-12 14:49:52 root: INFO: instruction count: 9: __C85etypeDecode.apply
2022-01-12 14:49:52 root: INFO: instruction count: 133: __C85etypeDecode.__m86DECODE_r_struct_of_o_struct_of_r_binaryANDr_int32ENDANDo_array_of_o_binaryANDo_int64ANDo_int32END_TO_SBaseStructPointer
2022-01-12 14:49:52 root: INFO: instruction count: 22: __C85etypeDecode.__m87INPLACE_DECODE_o_struct_of_r_binaryANDr_int32END_TO_o_locus
2022-01-12 14:49:52 root: INFO: instruction count: 33: __C85etypeDecode.__m88INPLACE_DECODE_r_binary_TO_r_string
2022-01-12 14:49:52 root: INFO: instruction count: 10: __C85etypeDecode.__m89INPLACE_DECODE_r_int32_TO_r_int32
2022-01-12 14:49:52 root: INFO: instruction count: 120: __C85etypeDecode.__m90INPLACE_DECODE_o_array_of_o_binary_TO_o_array_of_o_string
2022-01-12 14:49:52 root: INFO: instruction count: 33: __C85etypeDecode.__m91INPLACE_DECODE_o_binary_TO_o_string
2022-01-12 14:49:52 root: INFO: instruction count: 10: __C85etypeDecode.__m92INPLACE_DECODE_o_int64_TO_o_int64
2022-01-12 14:49:52 root: INFO: instruction count: 10: __C85etypeDecode.__m93INPLACE_DECODE_o_int32_TO_o_int32
2022-01-12 14:49:52 SparkContext: INFO: Starting job: foreachPartition at IndexBgen.scala:116
2022-01-12 14:49:52 DAGScheduler: INFO: Registering RDD 5 (mapPartitions at ContextRDD.scala:168) as input to shuffle 0
2022-01-12 14:49:52 DAGScheduler: INFO: Got job 0 (foreachPartition at IndexBgen.scala:116) with 1 output partitions
2022-01-12 14:49:52 DAGScheduler: INFO: Final stage: ResultStage 1 (foreachPartition at IndexBgen.scala:116)
2022-01-12 14:49:52 DAGScheduler: INFO: Parents of final stage: List(ShuffleMapStage 0)
2022-01-12 14:49:52 DAGScheduler: INFO: Missing parents: List(ShuffleMapStage 0)
2022-01-12 14:49:52 DAGScheduler: INFO: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at mapPartitions at ContextRDD.scala:168), which has no missing parents
2022-01-12 14:49:52 MemoryStore: INFO: Block broadcast_3 stored as values in memory (estimated size 50.0 KiB, free 911.8 MiB)
2022-01-12 14:49:52 MemoryStore: INFO: Block broadcast_3_piece0 stored as bytes in memory (estimated size 25.0 KiB, free 911.8 MiB)
2022-01-12 14:49:52 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on ip-xx-xx-xx-xxx.ec2.internal:39933 (size: 25.0 KiB, free: 912.2 MiB)
2022-01-12 14:49:52 SparkContext: INFO: Created broadcast 3 from broadcast at DAGScheduler.scala:1479
2022-01-12 14:49:52 DAGScheduler: INFO: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at mapPartitions at ContextRDD.scala:168) (first 15 tasks are for partitions Vector(0))
2022-01-12 14:49:52 YarnScheduler: INFO: Adding task set 0.0 with 1 tasks resource profile 0
2022-01-12 14:49:52 TaskSetManager: INFO: Starting task 0.0 in stage 0.0 (TID 0) (ip-xx-xx-xx-xx.ec2.internal, executor 1, partition 0, PROCESS_LOCAL, 4748 bytes) taskResourceAssignments Map()
2022-01-12 14:49:52 BlockManagerInfo: INFO: Added broadcast_3_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:42841 (size: 25.0 KiB, free: 25.8 GiB)
2022-01-12 14:49:54 BlockManagerInfo: INFO: Added broadcast_2_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:42841 (size: 1495.0 B, free: 25.8 GiB)
2022-01-12 14:49:54 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:42841 (size: 32.4 KiB, free: 25.8 GiB)
2022-01-12 14:49:55 BlockManagerInfo: INFO: Added broadcast_1_piece0 in memory on ip-xx-xx-xx-xx.ec2.internal:42841 (size: 3.2 KiB, free: 25.8 GiB)
2022-01-12 14:51:28 BlockManagerInfo: INFO: Removed broadcast_3_piece0 on ip-xx-xx-xx-xxx.ec2.internal:44653 in memory (size: 49.0 B, free: 911.9 MiB)
2022-01-12 14:51:28 BlockManagerInfo: INFO: Removed broadcast_5_piece0 on ip-xx-xx-xx-xxx.ec2.internal:44653 in memory (size: 201.2 KiB, free: 912.1 MiB)
2022-01-12 15:39:43 HealthTracker: INFO: Removing node (ip-xx-xx-xx-xx.ec2.internal,org.apache.spark.scheduler.ExcludeOnFailureTimedOut$@266fabbc,Some(1642001983741)) from ExclusionList because Exclude on failure timeout has reached.
2022-01-12 16:18:23 Hail: INFO: copying log to 's3://bucket_name/folder_for_sandbox/hail_log.log'...