2024-02-18 20:27:53.289 Hail: WARN: This Hail JAR was compiled for Spark 3.3.2, running with Spark 3.3.4. Compatibility is not guaranteed. 2024-02-18 20:27:53.321 SparkContext: INFO: Running Spark version 3.3.4 2024-02-18 20:27:53.426 ResourceUtils: INFO: ============================================================== 2024-02-18 20:27:53.426 ResourceUtils: INFO: No custom resources configured for spark.driver. 2024-02-18 20:27:53.427 ResourceUtils: INFO: ============================================================== 2024-02-18 20:27:53.427 SparkContext: INFO: Submitted application: Hail 2024-02-18 20:27:53.441 SparkContext: INFO: Spark configuration: spark.app.name=Hail spark.app.startTime=1708288073321 spark.app.submitTime=1708288071274 spark.driver.extraClassPath=/home/unix/tzayats/.local/lib/python3.9/site-packages/hail/backend/hail-all-spark.jar spark.driver.extraJavaOptions=-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED spark.driver.maxResultSize=0 spark.executor.extraClassPath=./hail-all-spark.jar spark.executor.extraJavaOptions=-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED spark.hadoop.io.compression.codecs=org.apache.hadoop.io.compress.DefaultCodec,is.hail.io.compress.BGzipCodec,is.hail.io.compress.BGzipCodecTbi,org.apache.hadoop.io.compress.GzipCodec spark.hadoop.mapreduce.input.fileinputformat.split.minsize=0 spark.jars=file:///home/unix/tzayats/.local/lib/python3.9/site-packages/hail/backend/hail-all-spark.jar spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator spark.kryoserializer.buffer.max=1g spark.logConf=true spark.master=local[*] spark.repl.local.jars=file:///home/unix/tzayats/.local/lib/python3.9/site-packages/hail/backend/hail-all-spark.jar spark.serializer=org.apache.spark.serializer.KryoSerializer spark.submit.deployMode=client spark.submit.pyFiles= spark.ui.showConsoleProgress=false 2024-02-18 20:27:53.521 ResourceProfile: INFO: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 2024-02-18 20:27:53.539 ResourceProfile: INFO: Limiting resource is cpu 2024-02-18 20:27:53.539 ResourceProfileManager: INFO: Added ResourceProfile id: 0 2024-02-18 20:27:53.722 SecurityManager: INFO: Changing view acls to: tzayats 2024-02-18 20:27:53.723 SecurityManager: INFO: Changing modify acls to: tzayats 2024-02-18 20:27:53.723 SecurityManager: INFO: Changing view acls groups to: 2024-02-18 20:27:53.724 SecurityManager: INFO: Changing modify acls groups to: 2024-02-18 20:27:53.724 SecurityManager: INFO: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tzayats); groups with view permissions: Set(); users with modify permissions: Set(tzayats); groups with modify permissions: Set() 2024-02-18 20:27:54.570 Utils: INFO: Successfully started service 'sparkDriver' on port 38940. 2024-02-18 20:27:54.819 SparkEnv: INFO: Registering MapOutputTracker 2024-02-18 20:27:54.879 SparkEnv: INFO: Registering BlockManagerMaster 2024-02-18 20:27:54.901 BlockManagerMasterEndpoint: INFO: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 2024-02-18 20:27:54.901 BlockManagerMasterEndpoint: INFO: BlockManagerMasterEndpoint up 2024-02-18 20:27:54.910 SparkEnv: INFO: Registering BlockManagerMasterHeartbeat 2024-02-18 20:27:54.970 DiskBlockManager: INFO: Created local directory at /tmp/blockmgr-ff993422-1355-47a4-9a6d-f18fbf3b5b14 2024-02-18 20:27:54.995 MemoryStore: INFO: MemoryStore started with capacity 413.9 MiB 2024-02-18 20:27:55.020 SparkEnv: INFO: Registering OutputCommitCoordinator 2024-02-18 20:27:55.091 log: INFO: Logging initialized @19311ms to org.sparkproject.jetty.util.log.Slf4jLog 2024-02-18 20:27:55.290 Server: INFO: jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.16.1+1 2024-02-18 20:27:55.315 Server: INFO: Started @19536ms 2024-02-18 20:27:55.378 AbstractConnector: INFO: Started ServerConnector@1c540a68{HTTP/1.1, (http/1.1)}{0.0.0.0:4040} 2024-02-18 20:27:55.379 Utils: INFO: Successfully started service 'SparkUI' on port 4040. 2024-02-18 20:27:55.414 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@b15a312{/,null,AVAILABLE,@Spark} 2024-02-18 20:27:55.450 SparkContext: INFO: Added JAR file:///home/unix/tzayats/.local/lib/python3.9/site-packages/hail/backend/hail-all-spark.jar at spark://uger-r7-c001.broadinstitute.org:38940/jars/hail-all-spark.jar with timestamp 1708288073321 2024-02-18 20:27:55.673 Executor: INFO: Starting executor ID driver on host uger-r7-c001.broadinstitute.org 2024-02-18 20:27:55.689 Executor: INFO: Starting executor with user classpath (userClassPathFirst = false): 'file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/./hail-all-spark.jar,file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/hail-all-spark.jar' 2024-02-18 20:27:55.785 Utils: INFO: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43954. 2024-02-18 20:27:55.785 NettyBlockTransferService: INFO: Server created on uger-r7-c001.broadinstitute.org:43954 2024-02-18 20:27:55.806 BlockManager: INFO: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 2024-02-18 20:27:55.813 BlockManagerMaster: INFO: Registering BlockManager BlockManagerId(driver, uger-r7-c001.broadinstitute.org, 43954, None) 2024-02-18 20:27:55.819 BlockManagerMasterEndpoint: INFO: Registering block manager uger-r7-c001.broadinstitute.org:43954 with 413.9 MiB RAM, BlockManagerId(driver, uger-r7-c001.broadinstitute.org, 43954, None) 2024-02-18 20:27:55.824 BlockManagerMaster: INFO: Registered BlockManager BlockManagerId(driver, uger-r7-c001.broadinstitute.org, 43954, None) 2024-02-18 20:27:55.844 BlockManager: INFO: Initialized BlockManager: BlockManagerId(driver, uger-r7-c001.broadinstitute.org, 43954, None) 2024-02-18 20:27:56.253 ContextHandler: INFO: Stopped o.s.j.s.ServletContextHandler@b15a312{/,null,STOPPED,@Spark} 2024-02-18 20:27:56.257 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@4a3f98c9{/jobs,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.258 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@1ee90eda{/jobs/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.260 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@ffe90b6{/jobs/job,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.261 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@4027e3fe{/jobs/job/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.262 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@5ca33d66{/stages,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.263 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@70120b51{/stages/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.264 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@6084dba0{/stages/stage,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.265 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@46fbb82f{/stages/stage/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.266 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@1605ba6b{/stages/pool,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.267 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@43495f4{/stages/pool/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.325 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@3ed29165{/storage,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.326 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@2aaf7ff1{/storage/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.327 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@7831e816{/storage/rdd,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.328 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@50b8d25{/storage/rdd/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.329 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@30196829{/environment,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.330 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@6ca8100a{/environment/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.331 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@2a65c5c3{/executors,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.332 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@2d5033c6{/executors/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.334 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@4d02779e{/executors/threadDump,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.335 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@424a4c73{/executors/threadDump/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.360 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@3a84d9a6{/static,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.364 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@6baf5599{/,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.367 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@22e72fb6{/api,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.367 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@38ef32de{/jobs/job/kill,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.368 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@550e2532{/stages/stage/kill,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.381 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@69798609{/metrics/json,null,AVAILABLE,@Spark} 2024-02-18 20:27:56.398 Hail: INFO: SparkUI: http://uger-r7-c001.broadinstitute.org:4040 2024-02-18 20:27:59.846 Hail: INFO: Running Hail version 0.2.127-bb535cd096c5 2024-02-18 20:28:00.273 SparkContext: WARN: Using an existing SparkContext; some configuration may not take effect. 2024-02-18 20:28:04.843 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:28:06.084 : INFO: globbing path file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf returned 1 files: LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf 2024-02-18 20:28:06.950 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:28:06.951 : INFO: TaskReport: stage=0, partition=0, attempt=0, peakBytes=0, peakBytesReadable=0.00 B, chunks requested=0, cache hits=0 2024-02-18 20:28:06.952 : INFO: RegionPool: FREE: 0 allocated (0 blocks / 0 chunks), regions.size = 0, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:28:06.953 : INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:28:06.953 : INFO: timing matrixTableType total 2.115s self 2.115s children 0.000ms %children 0.00% 2024-02-18 20:29:00.539 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:29:00.565 : INFO: globbing path file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf returned 1 files: LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf 2024-02-18 20:29:00.674 : INFO: starting execution of query hail_query_1 of initial size 5 2024-02-18 20:29:00.806 : INFO: IR Semantic Hash: Some(640856651) 2024-02-18 20:29:00.947 : INFO: initial IR: IR size 5: %1 = RNGStateLiteral !2 = MatrixRead [Matrix{global:Struct{},col_key:[s],col:Struct{s:String},row_key:[[locus,alleles]],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64}},entry:Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}}, False, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] !ht = MatrixColsTable(!2) TableCount(!ht) 2024-02-18 20:29:01.115 : INFO: after optimize: relationalLowerer, initial IR: IR size 1: I64 [1278] 2024-02-18 20:29:01.117 : INFO: after LowerMatrixToTable: IR size 1: I64 [1278] 2024-02-18 20:29:01.119 : INFO: after optimize: relationalLowerer, after LowerMatrixToTable: IR size 1: I64 [1278] 2024-02-18 20:29:01.121 : INFO: after LiftRelationalValuesToRelationalLets: IR size 1: I64 [1278] 2024-02-18 20:29:01.122 : INFO: after EvalRelationalLets: IR size 1: I64 [1278] 2024-02-18 20:29:01.124 : INFO: after LowerAndExecuteShuffles: IR size 1: I64 [1278] 2024-02-18 20:29:01.125 : INFO: after optimize: relationalLowerer, after LowerAndExecuteShuffles: IR size 1: I64 [1278] 2024-02-18 20:29:01.127 : INFO: after LowerOrInterpretNonCompilable: IR size 1: I64 [1278] 2024-02-18 20:29:01.130 : INFO: after optimize: relationalLowerer, after LowerOrInterpretNonCompilable: IR size 1: I64 [1278] 2024-02-18 20:29:01.315 : INFO: initial IR: IR size 2: !1 = I64 [1278] MakeTuple(!1) [(0)] 2024-02-18 20:29:01.358 : INFO: after optimize: compileLowerer, initial IR: IR size 1: Literal [Tuple[Int64], ] 2024-02-18 20:29:01.359 : INFO: after InlineApplyIR: IR size 1: Literal [Tuple[Int64], ] 2024-02-18 20:29:01.360 : INFO: after optimize: compileLowerer, after InlineApplyIR: IR size 1: Literal [Tuple[Int64], ] 2024-02-18 20:29:01.381 : INFO: after LowerArrayAggsToRunAggs: IR size 1: Literal [Tuple[Int64], ] 2024-02-18 20:29:01.383 : INFO: after optimize: compileLowerer, after LowerArrayAggsToRunAggs: IR size 1: Literal [Tuple[Int64], ] 2024-02-18 20:29:01.724 : INFO: encoder cache miss (0 hits, 1 misses, 0.000) 2024-02-18 20:29:01.880 : INFO: instruction count: 3: __C14HailClassLoaderContainer. 2024-02-18 20:29:01.881 : INFO: instruction count: 3: __C14HailClassLoaderContainer. 2024-02-18 20:29:01.884 : INFO: instruction count: 3: __C16FSContainer. 2024-02-18 20:29:01.884 : INFO: instruction count: 3: __C16FSContainer. 2024-02-18 20:29:01.919 : INFO: instruction count: 3: __C18etypeEncode. 2024-02-18 20:29:01.919 : INFO: instruction count: 7: __C18etypeEncode.apply 2024-02-18 20:29:01.920 : INFO: instruction count: 9: __C18etypeEncode.__m20ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_int64ENDEND 2024-02-18 20:29:01.920 : INFO: instruction count: 13: __C18etypeEncode.__m21ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64END 2024-02-18 20:29:01.920 : INFO: instruction count: 4: __C18etypeEncode.__m22ENCODE_SInt64$_TO_r_int64 2024-02-18 20:29:02.013 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:29:02.013 : INFO: TaskReport: stage=0, partition=0, attempt=0, peakBytes=0, peakBytesReadable=0.00 B, chunks requested=0, cache hits=0 2024-02-18 20:29:02.013 : INFO: RegionPool: FREE: 0 allocated (0 blocks / 0 chunks), regions.size = 0, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:29:02.013 : INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:31:17.626 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:31:17.679 : INFO: globbing path file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf returned 1 files: LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf 2024-02-18 20:31:17.735 : INFO: starting execution of query hail_query_2 of initial size 4 2024-02-18 20:31:17.751 : INFO: IR Semantic Hash: Some(1057170404) 2024-02-18 20:31:17.757 : INFO: initial IR: IR size 4: %1 = RNGStateLiteral !2 = MatrixRead [Matrix{global:Struct{},col_key:[s],col:Struct{s:String},row_key:[[locus,alleles]],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64}},entry:Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}}, False, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] MatrixWrite(!2) ["{\"name\":\"MatrixVCFWriter\",\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false}"] 2024-02-18 20:31:17.783 : INFO: after optimize: relationalLowerer, initial IR: IR size 2: !1 = MatrixRead [Matrix{global:Struct{},col_key:[s],col:Struct{s:String},row_key:[[locus,alleles]],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64}},entry:Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}}, False, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] MatrixWrite(!1) ["{\"name\":\"MatrixVCFWriter\",\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false}"] 2024-02-18 20:31:17.799 : INFO: after LowerMatrixToTable: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.822 : INFO: after optimize: relationalLowerer, after LowerMatrixToTable: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.825 : INFO: after LiftRelationalValuesToRelationalLets: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.827 : INFO: after EvalRelationalLets: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.829 : INFO: after LowerAndExecuteShuffles: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.848 : INFO: after optimize: relationalLowerer, after LowerAndExecuteShuffles: IR size 2: !ht = TableRead [Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],ALLELE_A:String,ALLELE_B:String,AN:Int32,BEADSET_ID:Int32,GC_SCORE:Float64,ILLUMINA_BUILD:String,ILLUMINA_CHR:String,ILLUMINA_POS:Int32,ILLUMINA_STRAND:String,N_AA:Int32,N_AB:Int32,N_BB:Int32,PROBE_A:String,PROBE_B:String,SOURCE:String,devR_AA:Float64,devR_AB:Float64,devR_BB:Float64,devTHETA_AA:Float64,devTHETA_AB:Float64,devTHETA_BB:Float64,devX_AA:Float64,devX_AB:Float64,devX_BB:Float64,devY_AA:Float64,devY_AB:Float64,devY_BB:Float64,meanR_AA:Float64,meanR_AB:Float64,meanR_BB:Float64,meanTHETA_AA:Float64,meanTHETA_AB:Float64,meanTHETA_BB:Float64,meanX_AA:Float64,meanX_AB:Float64,meanX_BB:Float64,meanY_AA:Float64,meanY_AB:Float64,meanY_BB:Float64,refSNP:String,zthresh_X:Float64,zthresh_Y:Float64},`the entries! [877f12a8827e18f61222c6c8c5fb04a8]`:Array[Struct{BAF:Float64,GT:Call,GTA:Call,GTZ:String,IGC:Float64,LRR:Float64,NORMX:Float64,NORMY:Float64,R:Float64,THETA:Float64,X:Int32,Y:Int32}]}}, False, {\"files\":[\"file:/stanley/genetics/analysis/adhd/LOGIC/geno/Feb2023/LOGIC_Doyle_Batch1_GSA-MDv3_Feb2023-tz.vcf\"],\"callFields\":[\"GTA\"],\"entryFloatTypeName\":\"Float64\",\"rg\":\"GRCh37\",\"contigRecoding\":{},\"arrayElementsRequired\":true,\"skipInvalidLoci\":false,\"gzAsBGZ\":false,\"forceGZ\":false,\"filterAndReplace\":{},\"name\":\"MatrixVCFReader\"}] TableWrite(!ht) ["{\"writer\":{\"path\":\"test.vcf\",\"exportType\":\"concatenated\",\"tabix\":false},\"colsFieldName\":\"__cols\",\"entriesFieldName\":\"the entries! [877f12a8827e18f61222c6c8c5fb04a8]\",\"colKey\":[\"s\"]}"] 2024-02-18 20:31:17.851 : INFO: LowerOrInterpretNonCompilable: whole stage code generation is a go! 2024-02-18 20:31:17.851 : INFO: lowering result: TableWrite 2024-02-18 20:31:18.072 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 34.4 KiB, free 413.9 MiB) 2024-02-18 20:31:19.886 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 413.9 MiB) 2024-02-18 20:31:19.889 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on uger-r7-c001.broadinstitute.org:43954 (size: 3.2 KiB, free: 413.9 MiB) 2024-02-18 20:31:19.898 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:411 2024-02-18 20:31:19.942 Hail: INFO: scanning VCF for sortedness... 2024-02-18 20:31:20.603 : INFO: initial IR: IR size 169: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = MakeStruct() !3 = NA [String] %38 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !s3 = StreamAggScan(!s2) { (%elt) => !4 = F64 [0.0] !5 = F64 [1.0] !6 = RNGStateLiteral !7 = ApplySeeded(!4, !5, !6) [rand_unif, 1, Float64] !8 = ApplyScanOp(%elt) [PrevNonnull] MakeStruct(key: %elt, token: !7, prevkey: !8) } StreamAgg(!s3) { (%elt2) => %9 = ApplyAggOp [Count] !10 = GetField(%elt2) [key] AggLet(!10) [False] { (%11) => !c1 = I32 [1] !12 = ApplyAggOp(!c1, %11, %11) [TakeBy] !c1_2 = I32 [1] !13 = ApplyAggOp(!c1_2, %11, %11) [TakeBy] !14 = GetField(%elt2) [prevkey] !15 = IsNA(!14) !16 = GetField(%elt2) [prevkey] !17 = GetField(%elt2) [key] !18 = ApplyComparisonOp(!16, !17) [LTEQ] !19 = ApplySpecial(!15, !18) [-1, lor, (), Boolean] !20 = Apply(!19) [-1, toInt64, (), Int64] !21 = ApplyAggOp(!20) [Sum] !22 = ApplyComparisonOp(!21, %9) [EQ] !23 = GetField(%elt2) [prevkey] !24 = SelectFields(!23) [(locus)] !25 = IsNA(!24) !26 = GetField(%elt2) [prevkey] !27 = SelectFields(!26) [(locus)] !28 = GetField(%elt2) [key] !29 = SelectFields(!28) [(locus)] !30 = ApplyComparisonOp(!27, !29) [LTEQ] !31 = ApplySpecial(!25, !30) [-1, lor, (), Boolean] !32 = Apply(!31) [-1, toInt64, (), Int64] !33 = ApplyAggOp(!32) [Sum] !34 = ApplyComparisonOp(!33, %9) [EQ] !c20 = I32 [20] !35 = GetField(%elt2) [key] !36 = GetField(%elt2) [token] !37 = ApplyAggOp(!c20, !35, !36) [TakeBy] MakeStruct(n: %9, minkey: !12, maxkey: !13, ksorted: !22, pksorted: !34, sample: !37) } } } !c0 = I32 [0] !39 = ArrayLen(%38) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !39, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt3) => !40 = ArrayRef(%38, %elt3) [-1] InsertFields !40 (i: %elt3) } !s6 = StreamFilter(!s5) { (%elt4) => !41 = GetField(%elt4) [minkey] !42 = ArrayLen(!41) !c0_2 = I32 [0] ApplyComparisonOp(!42, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt5) => !43 = GetField(%elt5) [minkey] !c0_3 = I32 [0] !44 = ArrayRef(!43, !c0_3) [-1] !45 = GetField(%elt5) [maxkey] !c0_4 = I32 [0] !46 = ArrayRef(!45, !c0_4) [-1] InsertFields %elt5 (minkey: !44, maxkey: !46) } %49 = ArraySort(!s7) { (%l, %r) => !47 = SelectFields(%l) [(minkey maxkey)] !48 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!47, !48) [LT] } !s8 = ToStream(%49) [False] !true = True !51 = StreamFold(!s8, !true) { (%accum, %elt6) => !50 = GetField(%elt6) [ksorted] ApplySpecial(%accum, !50) [-1, land, (), Boolean] } !c0_5 = I32 [0] !52 = ArrayLen(%49) !c1_4 = I32 [1] !53 = ApplyBinaryPrimOp(!52, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !53, !c1_5) [-1, False] !true2 = True !60 = StreamFold(!s9, !true2) { (%accum2, %elt7) => !54 = ArrayRef(%49, %elt7) [-1] !55 = GetField(!54) [maxkey] !c1_6 = I32 [1] !56 = ApplyBinaryPrimOp(%elt7, !c1_6) [Add] !57 = ArrayRef(%49, !56) [-1] !58 = GetField(!57) [minkey] !59 = ApplyComparisonOp(!55, !58) [LTEQ] ApplySpecial(%accum2, !59) [-1, land, (), Boolean] } !61 = ApplySpecial(!51, !60) [-1, land, (), Boolean] !s10 = ToStream(%49) [False] !true3 = True !63 = StreamFold(!s10, !true3) { (%accum3, %elt8) => !62 = GetField(%elt8) [pksorted] ApplySpecial(%accum3, !62) [-1, land, (), Boolean] } !c0_6 = I32 [0] !64 = ArrayLen(%49) !c1_7 = I32 [1] !65 = ApplyBinaryPrimOp(!64, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !65, !c1_8) [-1, False] !true4 = True !74 = StreamFold(!s11, !true4) { (%accum4, %elt9) => !66 = ArrayRef(%49, %elt9) [-1] !67 = GetField(!66) [maxkey] !68 = SelectFields(!67) [(locus)] !c1_9 = I32 [1] !69 = ApplyBinaryPrimOp(%elt9, !c1_9) [Add] !70 = ArrayRef(%49, !69) [-1] !71 = GetField(!70) [minkey] !72 = SelectFields(!71) [(locus)] !73 = ApplyComparisonOp(!68, !72) [LTEQ] ApplySpecial(%accum4, !73) [-1, land, (), Boolean] } !75 = ApplySpecial(!63, !74) [-1, land, (), Boolean] MakeStruct(ksorted: !61, pksorted: !75, sortedPartData: %49) 2024-02-18 20:31:21.113 : INFO: after optimize: compileLowerer, initial IR: IR size 169: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = Literal [Struct{}, ] !3 = NA [String] %38 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !s3 = StreamAggScan(!s2) { (%elt) => !4 = F64 [0.0] !5 = F64 [1.0] !6 = RNGStateLiteral !7 = ApplySeeded(!4, !5, !6) [rand_unif, 1, Float64] !8 = ApplyScanOp(%elt) [PrevNonnull] MakeStruct(key: %elt, token: !7, prevkey: !8) } StreamAgg(!s3) { (%elt2) => %9 = ApplyAggOp [Count] !10 = GetField(%elt2) [key] AggLet(!10) [False] { (%11) => !c1 = I32 [1] !12 = ApplyAggOp(!c1, %11, %11) [TakeBy] !c1_2 = I32 [1] !13 = ApplyAggOp(!c1_2, %11, %11) [TakeBy] !14 = GetField(%elt2) [prevkey] !15 = IsNA(!14) !16 = GetField(%elt2) [prevkey] !17 = GetField(%elt2) [key] !18 = ApplyComparisonOp(!16, !17) [LTEQ] !19 = ApplySpecial(!15, !18) [-1, lor, (), Boolean] !20 = Apply(!19) [-1, toInt64, (), Int64] !21 = ApplyAggOp(!20) [Sum] !22 = ApplyComparisonOp(!21, %9) [EQ] !23 = GetField(%elt2) [prevkey] !24 = SelectFields(!23) [()] !25 = IsNA(!24) !26 = GetField(%elt2) [prevkey] !27 = SelectFields(!26) [(locus)] !28 = GetField(%elt2) [key] !29 = SelectFields(!28) [(locus)] !30 = ApplyComparisonOp(!27, !29) [LTEQ] !31 = ApplySpecial(!25, !30) [-1, lor, (), Boolean] !32 = Apply(!31) [-1, toInt64, (), Int64] !33 = ApplyAggOp(!32) [Sum] !34 = ApplyComparisonOp(!33, %9) [EQ] !c20 = I32 [20] !35 = GetField(%elt2) [key] !36 = GetField(%elt2) [token] !37 = ApplyAggOp(!c20, !35, !36) [TakeBy] MakeStruct(n: %9, minkey: !12, maxkey: !13, ksorted: !22, pksorted: !34, sample: !37) } } } !c0 = I32 [0] !39 = ArrayLen(%38) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !39, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt3) => !40 = ArrayRef(%38, %elt3) [-1] InsertFields !40 (i: %elt3) } !s6 = StreamFilter(!s5) { (%elt4) => !41 = GetField(%elt4) [minkey] !42 = ArrayLen(!41) !c0_2 = I32 [0] ApplyComparisonOp(!42, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt5) => !43 = GetField(%elt5) [minkey] !c0_3 = I32 [0] !44 = ArrayRef(!43, !c0_3) [-1] !45 = GetField(%elt5) [maxkey] !c0_4 = I32 [0] !46 = ArrayRef(!45, !c0_4) [-1] InsertFields %elt5 (minkey: !44, maxkey: !46) } %49 = ArraySort(!s7) { (%l, %r) => !47 = SelectFields(%l) [(minkey maxkey)] !48 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!47, !48) [LT] } !s8 = ToStream(%49) [False] !true = True !51 = StreamFold(!s8, !true) { (%accum, %elt6) => !50 = GetField(%elt6) [ksorted] ApplySpecial(%accum, !50) [-1, land, (), Boolean] } !c0_5 = I32 [0] !52 = ArrayLen(%49) !c1_4 = I32 [1] !53 = ApplyBinaryPrimOp(!52, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !53, !c1_5) [-1, False] !true2 = True !60 = StreamFold(!s9, !true2) { (%accum2, %elt7) => !54 = ArrayRef(%49, %elt7) [-1] !55 = GetField(!54) [maxkey] !c1_6 = I32 [1] !56 = ApplyBinaryPrimOp(%elt7, !c1_6) [Add] !57 = ArrayRef(%49, !56) [-1] !58 = GetField(!57) [minkey] !59 = ApplyComparisonOp(!55, !58) [LTEQ] ApplySpecial(%accum2, !59) [-1, land, (), Boolean] } !61 = ApplySpecial(!51, !60) [-1, land, (), Boolean] !s10 = ToStream(%49) [False] !true3 = True !63 = StreamFold(!s10, !true3) { (%accum3, %elt8) => !62 = GetField(%elt8) [pksorted] ApplySpecial(%accum3, !62) [-1, land, (), Boolean] } !c0_6 = I32 [0] !64 = ArrayLen(%49) !c1_7 = I32 [1] !65 = ApplyBinaryPrimOp(!64, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !65, !c1_8) [-1, False] !true4 = True !74 = StreamFold(!s11, !true4) { (%accum4, %elt9) => !66 = ArrayRef(%49, %elt9) [-1] !67 = GetField(!66) [maxkey] !68 = SelectFields(!67) [(locus)] !c1_9 = I32 [1] !69 = ApplyBinaryPrimOp(%elt9, !c1_9) [Add] !70 = ArrayRef(%49, !69) [-1] !71 = GetField(!70) [minkey] !72 = SelectFields(!71) [(locus)] !73 = ApplyComparisonOp(!68, !72) [LTEQ] ApplySpecial(%accum4, !73) [-1, land, (), Boolean] } !75 = ApplySpecial(!63, !74) [-1, land, (), Boolean] MakeStruct(ksorted: !61, pksorted: !75, sortedPartData: %49) 2024-02-18 20:31:21.155 : INFO: after InlineApplyIR: IR size 169: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = Literal [Struct{}, ] !3 = NA [String] %38 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !s3 = StreamAggScan(!s2) { (%elt) => !4 = F64 [0.0] !5 = F64 [1.0] !6 = RNGStateLiteral !7 = ApplySeeded(!4, !5, !6) [rand_unif, 1, Float64] !8 = ApplyScanOp(%elt) [PrevNonnull] MakeStruct(key: %elt, token: !7, prevkey: !8) } StreamAgg(!s3) { (%elt2) => %9 = ApplyAggOp [Count] !10 = GetField(%elt2) [key] AggLet(!10) [False] { (%11) => !c1 = I32 [1] !12 = ApplyAggOp(!c1, %11, %11) [TakeBy] !c1_2 = I32 [1] !13 = ApplyAggOp(!c1_2, %11, %11) [TakeBy] !14 = GetField(%elt2) [prevkey] !15 = IsNA(!14) !16 = GetField(%elt2) [prevkey] !17 = GetField(%elt2) [key] !18 = ApplyComparisonOp(!16, !17) [LTEQ] !19 = ApplySpecial(!15, !18) [-1, lor, (), Boolean] !20 = Apply(!19) [-1, toInt64, (), Int64] !21 = ApplyAggOp(!20) [Sum] !22 = ApplyComparisonOp(!21, %9) [EQ] !23 = GetField(%elt2) [prevkey] !24 = SelectFields(!23) [()] !25 = IsNA(!24) !26 = GetField(%elt2) [prevkey] !27 = SelectFields(!26) [(locus)] !28 = GetField(%elt2) [key] !29 = SelectFields(!28) [(locus)] !30 = ApplyComparisonOp(!27, !29) [LTEQ] !31 = ApplySpecial(!25, !30) [-1, lor, (), Boolean] !32 = Apply(!31) [-1, toInt64, (), Int64] !33 = ApplyAggOp(!32) [Sum] !34 = ApplyComparisonOp(!33, %9) [EQ] !c20 = I32 [20] !35 = GetField(%elt2) [key] !36 = GetField(%elt2) [token] !37 = ApplyAggOp(!c20, !35, !36) [TakeBy] MakeStruct(n: %9, minkey: !12, maxkey: !13, ksorted: !22, pksorted: !34, sample: !37) } } } !c0 = I32 [0] !39 = ArrayLen(%38) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !39, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt3) => !40 = ArrayRef(%38, %elt3) [-1] InsertFields !40 (i: %elt3) } !s6 = StreamFilter(!s5) { (%elt4) => !41 = GetField(%elt4) [minkey] !42 = ArrayLen(!41) !c0_2 = I32 [0] ApplyComparisonOp(!42, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt5) => !43 = GetField(%elt5) [minkey] !c0_3 = I32 [0] !44 = ArrayRef(!43, !c0_3) [-1] !45 = GetField(%elt5) [maxkey] !c0_4 = I32 [0] !46 = ArrayRef(!45, !c0_4) [-1] InsertFields %elt5 (minkey: !44, maxkey: !46) } %49 = ArraySort(!s7) { (%l, %r) => !47 = SelectFields(%l) [(minkey maxkey)] !48 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!47, !48) [LT] } !s8 = ToStream(%49) [False] !true = True !51 = StreamFold(!s8, !true) { (%accum, %elt6) => !50 = GetField(%elt6) [ksorted] ApplySpecial(%accum, !50) [-1, land, (), Boolean] } !c0_5 = I32 [0] !52 = ArrayLen(%49) !c1_4 = I32 [1] !53 = ApplyBinaryPrimOp(!52, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !53, !c1_5) [-1, False] !true2 = True !60 = StreamFold(!s9, !true2) { (%accum2, %elt7) => !54 = ArrayRef(%49, %elt7) [-1] !55 = GetField(!54) [maxkey] !c1_6 = I32 [1] !56 = ApplyBinaryPrimOp(%elt7, !c1_6) [Add] !57 = ArrayRef(%49, !56) [-1] !58 = GetField(!57) [minkey] !59 = ApplyComparisonOp(!55, !58) [LTEQ] ApplySpecial(%accum2, !59) [-1, land, (), Boolean] } !61 = ApplySpecial(!51, !60) [-1, land, (), Boolean] !s10 = ToStream(%49) [False] !true3 = True !63 = StreamFold(!s10, !true3) { (%accum3, %elt8) => !62 = GetField(%elt8) [pksorted] ApplySpecial(%accum3, !62) [-1, land, (), Boolean] } !c0_6 = I32 [0] !64 = ArrayLen(%49) !c1_7 = I32 [1] !65 = ApplyBinaryPrimOp(!64, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !65, !c1_8) [-1, False] !true4 = True !74 = StreamFold(!s11, !true4) { (%accum4, %elt9) => !66 = ArrayRef(%49, %elt9) [-1] !67 = GetField(!66) [maxkey] !68 = SelectFields(!67) [(locus)] !c1_9 = I32 [1] !69 = ApplyBinaryPrimOp(%elt9, !c1_9) [Add] !70 = ArrayRef(%49, !69) [-1] !71 = GetField(!70) [minkey] !72 = SelectFields(!71) [(locus)] !73 = ApplyComparisonOp(!68, !72) [LTEQ] ApplySpecial(%accum4, !73) [-1, land, (), Boolean] } !75 = ApplySpecial(!63, !74) [-1, land, (), Boolean] MakeStruct(ksorted: !61, pksorted: !75, sortedPartData: %49) 2024-02-18 20:31:21.505 : INFO: after optimize: compileLowerer, after InlineApplyIR: IR size 169: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = Literal [Struct{}, ] !3 = NA [String] %38 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !s3 = StreamAggScan(!s2) { (%elt) => !4 = F64 [0.0] !5 = F64 [1.0] !6 = RNGStateLiteral !7 = ApplySeeded(!4, !5, !6) [rand_unif, 1, Float64] !8 = ApplyScanOp(%elt) [PrevNonnull] MakeStruct(key: %elt, token: !7, prevkey: !8) } StreamAgg(!s3) { (%elt2) => %9 = ApplyAggOp [Count] !10 = GetField(%elt2) [key] AggLet(!10) [False] { (%11) => !c1 = I32 [1] !12 = ApplyAggOp(!c1, %11, %11) [TakeBy] !c1_2 = I32 [1] !13 = ApplyAggOp(!c1_2, %11, %11) [TakeBy] !14 = GetField(%elt2) [prevkey] !15 = IsNA(!14) !16 = GetField(%elt2) [prevkey] !17 = GetField(%elt2) [key] !18 = ApplyComparisonOp(!16, !17) [LTEQ] !19 = ApplySpecial(!15, !18) [-1, lor, (), Boolean] !20 = Apply(!19) [-1, toInt64, (), Int64] !21 = ApplyAggOp(!20) [Sum] !22 = ApplyComparisonOp(!21, %9) [EQ] !23 = GetField(%elt2) [prevkey] !24 = SelectFields(!23) [()] !25 = IsNA(!24) !26 = GetField(%elt2) [prevkey] !27 = SelectFields(!26) [(locus)] !28 = GetField(%elt2) [key] !29 = SelectFields(!28) [(locus)] !30 = ApplyComparisonOp(!27, !29) [LTEQ] !31 = ApplySpecial(!25, !30) [-1, lor, (), Boolean] !32 = Apply(!31) [-1, toInt64, (), Int64] !33 = ApplyAggOp(!32) [Sum] !34 = ApplyComparisonOp(!33, %9) [EQ] !c20 = I32 [20] !35 = GetField(%elt2) [key] !36 = GetField(%elt2) [token] !37 = ApplyAggOp(!c20, !35, !36) [TakeBy] MakeStruct(n: %9, minkey: !12, maxkey: !13, ksorted: !22, pksorted: !34, sample: !37) } } } !c0 = I32 [0] !39 = ArrayLen(%38) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !39, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt3) => !40 = ArrayRef(%38, %elt3) [-1] InsertFields !40 (i: %elt3) } !s6 = StreamFilter(!s5) { (%elt4) => !41 = GetField(%elt4) [minkey] !42 = ArrayLen(!41) !c0_2 = I32 [0] ApplyComparisonOp(!42, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt5) => !43 = GetField(%elt5) [minkey] !c0_3 = I32 [0] !44 = ArrayRef(!43, !c0_3) [-1] !45 = GetField(%elt5) [maxkey] !c0_4 = I32 [0] !46 = ArrayRef(!45, !c0_4) [-1] InsertFields %elt5 (minkey: !44, maxkey: !46) } %49 = ArraySort(!s7) { (%l, %r) => !47 = SelectFields(%l) [(minkey maxkey)] !48 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!47, !48) [LT] } !s8 = ToStream(%49) [False] !true = True !51 = StreamFold(!s8, !true) { (%accum, %elt6) => !50 = GetField(%elt6) [ksorted] ApplySpecial(%accum, !50) [-1, land, (), Boolean] } !c0_5 = I32 [0] !52 = ArrayLen(%49) !c1_4 = I32 [1] !53 = ApplyBinaryPrimOp(!52, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !53, !c1_5) [-1, False] !true2 = True !60 = StreamFold(!s9, !true2) { (%accum2, %elt7) => !54 = ArrayRef(%49, %elt7) [-1] !55 = GetField(!54) [maxkey] !c1_6 = I32 [1] !56 = ApplyBinaryPrimOp(%elt7, !c1_6) [Add] !57 = ArrayRef(%49, !56) [-1] !58 = GetField(!57) [minkey] !59 = ApplyComparisonOp(!55, !58) [LTEQ] ApplySpecial(%accum2, !59) [-1, land, (), Boolean] } !61 = ApplySpecial(!51, !60) [-1, land, (), Boolean] !s10 = ToStream(%49) [False] !true3 = True !63 = StreamFold(!s10, !true3) { (%accum3, %elt8) => !62 = GetField(%elt8) [pksorted] ApplySpecial(%accum3, !62) [-1, land, (), Boolean] } !c0_6 = I32 [0] !64 = ArrayLen(%49) !c1_7 = I32 [1] !65 = ApplyBinaryPrimOp(!64, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !65, !c1_8) [-1, False] !true4 = True !74 = StreamFold(!s11, !true4) { (%accum4, %elt9) => !66 = ArrayRef(%49, %elt9) [-1] !67 = GetField(!66) [maxkey] !68 = SelectFields(!67) [(locus)] !c1_9 = I32 [1] !69 = ApplyBinaryPrimOp(%elt9, !c1_9) [Add] !70 = ArrayRef(%49, !69) [-1] !71 = GetField(!70) [minkey] !72 = SelectFields(!71) [(locus)] !73 = ApplyComparisonOp(!68, !72) [LTEQ] ApplySpecial(%accum4, !73) [-1, land, (), Boolean] } !75 = ApplySpecial(!63, !74) [-1, land, (), Boolean] MakeStruct(ksorted: !61, pksorted: !75, sortedPartData: %49) 2024-02-18 20:31:21.828 : INFO: after LowerArrayAggsToRunAggs: IR size 208: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = Literal [Struct{}, ] !3 = NA [String] %65 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !4 = InitOp [0, (Count (TypedStateSig +PInt64))] !c1 = I32 [1] !5 = InitOp(!c1) [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !c1_2 = I32 [1] !6 = InitOp(!c1_2) [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !7 = InitOp [3, (Sum (TypedStateSig +PInt64))] !8 = InitOp [4, (Sum (TypedStateSig +PInt64))] !c20 = I32 [20] !9 = InitOp(!c20) [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] !10 = Begin(!4, !5, !6, !7, !8, !9) !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !11 = InitOp [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !12 = Begin(!11) !s3 = RunAggScan(!s2, !12) [__iruid_246, ( (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] { (%elt) => !13 = SeqOp(%elt) [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] Begin(!13) } { (%elt2) => !14 = ResultOp [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] %15 = MakeTuple(!14) [(0)] !16 = F64 [0.0] !17 = F64 [1.0] !18 = RNGStateLiteral !19 = ApplySeeded(!16, !17, !18) [rand_unif, 1, Float64] !20 = GetTupleElement(%15) [0] MakeStruct(key: %elt2, token: !19, prevkey: !20) } !47 = StreamFor(!s3) { (%elt3) => %21 = GetField(%elt3) [key] !22 = SeqOp [0, (Count (TypedStateSig +PInt64))] !23 = SeqOp(%21, %21) [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !24 = SeqOp(%21, %21) [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !25 = GetField(%elt3) [prevkey] !26 = IsNA(!25) !27 = GetField(%elt3) [prevkey] !28 = GetField(%elt3) [key] !29 = ApplyComparisonOp(!27, !28) [LTEQ] !30 = ApplySpecial(!26, !29) [-1, lor, (), Boolean] !31 = Apply(!30) [-1, toInt64, (), Int64] !32 = SeqOp(!31) [3, (Sum (TypedStateSig +PInt64))] !33 = GetField(%elt3) [prevkey] !34 = SelectFields(!33) [()] !35 = IsNA(!34) !36 = GetField(%elt3) [prevkey] !37 = SelectFields(!36) [(locus)] !38 = GetField(%elt3) [key] !39 = SelectFields(!38) [(locus)] !40 = ApplyComparisonOp(!37, !39) [LTEQ] !41 = ApplySpecial(!35, !40) [-1, lor, (), Boolean] !42 = Apply(!41) [-1, toInt64, (), Int64] !43 = SeqOp(!42) [4, (Sum (TypedStateSig +PInt64))] !44 = GetField(%elt3) [key] !45 = GetField(%elt3) [token] !46 = SeqOp(!44, !45) [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] Begin(!22, !23, !24, !32, !43, !46) } !48 = Begin(!10, !47) !49 = ResultOp [0, (Count (TypedStateSig +PInt64))] !50 = ResultOp [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !51 = ResultOp [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !52 = ResultOp [3, (Sum (TypedStateSig +PInt64))] !53 = ResultOp [4, (Sum (TypedStateSig +PInt64))] !54 = ResultOp [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] !55 = MakeTuple(!49, !50, !51, !52, !53, !54) [(0 1 2 3 4 5)] %56 = RunAgg(!48, !55) [((TypedStateSig +PInt64) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}) (TypedStateSig +PInt64) (TypedStateSig +PInt64) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] %57 = GetTupleElement(%56) [0] !58 = GetTupleElement(%56) [1] !59 = GetTupleElement(%56) [2] !60 = GetTupleElement(%56) [3] !61 = ApplyComparisonOp(!60, %57) [EQ] !62 = GetTupleElement(%56) [4] !63 = ApplyComparisonOp(!62, %57) [EQ] !64 = GetTupleElement(%56) [5] MakeStruct(n: %57, minkey: !58, maxkey: !59, ksorted: !61, pksorted: !63, sample: !64) } !c0 = I32 [0] !66 = ArrayLen(%65) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !66, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt4) => !67 = ArrayRef(%65, %elt4) [-1] InsertFields !67 (i: %elt4) } !s6 = StreamFilter(!s5) { (%elt5) => !68 = GetField(%elt5) [minkey] !69 = ArrayLen(!68) !c0_2 = I32 [0] ApplyComparisonOp(!69, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt6) => !70 = GetField(%elt6) [minkey] !c0_3 = I32 [0] !71 = ArrayRef(!70, !c0_3) [-1] !72 = GetField(%elt6) [maxkey] !c0_4 = I32 [0] !73 = ArrayRef(!72, !c0_4) [-1] InsertFields %elt6 (minkey: !71, maxkey: !73) } %76 = ArraySort(!s7) { (%l, %r) => !74 = SelectFields(%l) [(minkey maxkey)] !75 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!74, !75) [LT] } !s8 = ToStream(%76) [False] !true = True !78 = StreamFold(!s8, !true) { (%accum, %elt7) => !77 = GetField(%elt7) [ksorted] ApplySpecial(%accum, !77) [-1, land, (), Boolean] } !c0_5 = I32 [0] !79 = ArrayLen(%76) !c1_4 = I32 [1] !80 = ApplyBinaryPrimOp(!79, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !80, !c1_5) [-1, False] !true2 = True !87 = StreamFold(!s9, !true2) { (%accum2, %elt8) => !81 = ArrayRef(%76, %elt8) [-1] !82 = GetField(!81) [maxkey] !c1_6 = I32 [1] !83 = ApplyBinaryPrimOp(%elt8, !c1_6) [Add] !84 = ArrayRef(%76, !83) [-1] !85 = GetField(!84) [minkey] !86 = ApplyComparisonOp(!82, !85) [LTEQ] ApplySpecial(%accum2, !86) [-1, land, (), Boolean] } !88 = ApplySpecial(!78, !87) [-1, land, (), Boolean] !s10 = ToStream(%76) [False] !true3 = True !90 = StreamFold(!s10, !true3) { (%accum3, %elt9) => !89 = GetField(%elt9) [pksorted] ApplySpecial(%accum3, !89) [-1, land, (), Boolean] } !c0_6 = I32 [0] !91 = ArrayLen(%76) !c1_7 = I32 [1] !92 = ApplyBinaryPrimOp(!91, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !92, !c1_8) [-1, False] !true4 = True !101 = StreamFold(!s11, !true4) { (%accum4, %elt10) => !93 = ArrayRef(%76, %elt10) [-1] !94 = GetField(!93) [maxkey] !95 = SelectFields(!94) [(locus)] !c1_9 = I32 [1] !96 = ApplyBinaryPrimOp(%elt10, !c1_9) [Add] !97 = ArrayRef(%76, !96) [-1] !98 = GetField(!97) [minkey] !99 = SelectFields(!98) [(locus)] !100 = ApplyComparisonOp(!95, !99) [LTEQ] ApplySpecial(%accum4, !100) [-1, land, (), Boolean] } !102 = ApplySpecial(!90, !101) [-1, land, (), Boolean] MakeStruct(ksorted: !88, pksorted: !102, sortedPartData: %76) 2024-02-18 20:31:22.203 : INFO: after optimize: compileLowerer, after LowerArrayAggsToRunAggs: IR size 206: !1 = Literal [Array[Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}], ] !s = ToStream(!1) [False] !2 = Literal [Struct{}, ] !3 = NA [String] %65 = CollectDistributedArray(!s, !2, !3) { (%ctx, %g) => !4 = InitOp [0, (Count (TypedStateSig +PInt64))] !c1 = I32 [1] !5 = InitOp(!c1) [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !c1_2 = I32 [1] !6 = InitOp(!c1_2) [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !7 = InitOp [3, (Sum (TypedStateSig +PInt64))] !8 = InitOp [4, (Sum (TypedStateSig +PInt64))] !c20 = I32 [20] !9 = InitOp(!c20) [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] !10 = Begin(!4, !5, !6, !7, !8, !9) !s2 = ReadPartition(%ctx) [Struct{locus:Locus(GRCh37),alleles:Array[String]}, "{\"category\":\"PartitionIteratorLongReader\",\"fullRowType\":\"Struct{locus:Locus(GRCh37),alleles:Array[String],__row_uid:Tuple[Int64,Int64]}\",\"uidFieldName\":\"__row_uid\",\"contextType\":\"Struct{partitionIndex:Int32,fileNum:Int32,file:String,start:Int64,end:Int64,split:Boolean}\"}"] !11 = InitOp [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !12 = Begin(!11) !s3 = RunAggScan(!s2, !12) [__iruid_384, ( (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] { (%elt) => !13 = SeqOp(%elt) [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] Begin(!13) } { (%elt2) => !14 = ResultOp [0, (PrevNonnull (TypedStateSig PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] %15 = MakeTuple(!14) [(0)] !16 = F64 [0.0] !17 = F64 [1.0] !18 = RNGStateLiteral !19 = ApplySeeded(!16, !17, !18) [rand_unif, 1, Float64] !20 = GetTupleElement(%15) [0] MakeStruct(key: %elt2, token: !19, prevkey: !20) } !47 = StreamFor(!s3) { (%elt3) => %21 = GetField(%elt3) [key] !22 = SeqOp [0, (Count (TypedStateSig +PInt64))] !23 = SeqOp(%21, %21) [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !24 = SeqOp(%21, %21) [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !25 = GetField(%elt3) [prevkey] !26 = IsNA(!25) !27 = GetField(%elt3) [prevkey] !28 = GetField(%elt3) [key] !29 = ApplyComparisonOp(!27, !28) [LTEQ] !30 = ApplySpecial(!26, !29) [-1, lor, (), Boolean] !31 = Apply(!30) [-1, toInt64, (), Int64] !32 = SeqOp(!31) [3, (Sum (TypedStateSig +PInt64))] !33 = GetField(%elt3) [prevkey] !34 = SelectFields(!33) [()] !35 = IsNA(!34) !36 = GetField(%elt3) [prevkey] !37 = SelectFields(!36) [(locus)] !38 = GetField(%elt3) [key] !39 = SelectFields(!38) [(locus)] !40 = ApplyComparisonOp(!37, !39) [LTEQ] !41 = ApplySpecial(!35, !40) [-1, lor, (), Boolean] !42 = Apply(!41) [-1, toInt64, (), Int64] !43 = SeqOp(!42) [4, (Sum (TypedStateSig +PInt64))] !44 = GetField(%elt3) [key] !45 = GetField(%elt3) [token] !46 = SeqOp(!44, !45) [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] Begin(!22, !23, !24, !32, !43, !46) } !48 = Begin(!10, !47) !49 = ResultOp [0, (Count (TypedStateSig +PInt64))] !50 = ResultOp [1, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !51 = ResultOp [2, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}))] !52 = ResultOp [3, (Sum (TypedStateSig +PInt64))] !53 = ResultOp [4, (Sum (TypedStateSig +PInt64))] !54 = ResultOp [5, (TakeBy (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] !55 = MakeTuple(!49, !50, !51, !52, !53, !54) [(0 1 2 3 4 5)] %56 = RunAgg(!48, !55) [((TypedStateSig +PInt64) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]}) (TypedStateSig +PInt64) (TypedStateSig +PInt64) (TakeByStateSig +PCStruct{locus:+PCLocus(GRCh37),alleles:+PCArray[+PCString]} +PFloat64))] %57 = GetTupleElement(%56) [0] !58 = GetTupleElement(%56) [1] !59 = GetTupleElement(%56) [2] !60 = GetTupleElement(%56) [3] !61 = ApplyComparisonOp(!60, %57) [EQ] !62 = GetTupleElement(%56) [4] !63 = ApplyComparisonOp(!62, %57) [EQ] !64 = GetTupleElement(%56) [5] MakeStruct(n: %57, minkey: !58, maxkey: !59, ksorted: !61, pksorted: !63, sample: !64) } !c0 = I32 [0] !66 = ArrayLen(%65) !c1_3 = I32 [1] !s4 = StreamRange(!c0, !66, !c1_3) [-1, False] !s5 = StreamMap(!s4) { (%elt4) => !67 = ArrayRef(%65, %elt4) [-1] InsertFields !67 (i: %elt4) } !s6 = StreamFilter(!s5) { (%elt5) => !68 = GetField(%elt5) [minkey] !69 = ArrayLen(!68) !c0_2 = I32 [0] ApplyComparisonOp(!69, !c0_2) [GT] } !s7 = StreamMap(!s6) { (%elt6) => !70 = GetField(%elt6) [minkey] !c0_3 = I32 [0] !71 = ArrayRef(!70, !c0_3) [-1] !72 = GetField(%elt6) [maxkey] !c0_4 = I32 [0] !73 = ArrayRef(!72, !c0_4) [-1] InsertFields %elt6 (minkey: !71, maxkey: !73) } %76 = ArraySort(!s7) { (%l, %r) => !74 = SelectFields(%l) [(minkey maxkey)] !75 = SelectFields(%r) [(minkey maxkey)] ApplyComparisonOp(!74, !75) [LT] } !s8 = ToStream(%76) [False] !true = True !78 = StreamFold(!s8, !true) { (%accum, %elt7) => !77 = GetField(%elt7) [ksorted] ApplySpecial(%accum, !77) [-1, land, (), Boolean] } !c0_5 = I32 [0] !79 = ArrayLen(%76) !c1_4 = I32 [1] !80 = ApplyBinaryPrimOp(!79, !c1_4) [Subtract] !c1_5 = I32 [1] !s9 = StreamRange(!c0_5, !80, !c1_5) [-1, False] !true2 = True !87 = StreamFold(!s9, !true2) { (%accum2, %elt8) => !81 = ArrayRef(%76, %elt8) [-1] !82 = GetField(!81) [maxkey] !c1_6 = I32 [1] !83 = ApplyBinaryPrimOp(%elt8, !c1_6) [Add] !84 = ArrayRef(%76, !83) [-1] !85 = GetField(!84) [minkey] !86 = ApplyComparisonOp(!82, !85) [LTEQ] ApplySpecial(%accum2, !86) [-1, land, (), Boolean] } !88 = ApplySpecial(!78, !87) [-1, land, (), Boolean] !s10 = ToStream(%76) [False] !true3 = True !90 = StreamFold(!s10, !true3) { (%accum3, %elt9) => !89 = GetField(%elt9) [pksorted] ApplySpecial(%accum3, !89) [-1, land, (), Boolean] } !c0_6 = I32 [0] !91 = ArrayLen(%76) !c1_7 = I32 [1] !92 = ApplyBinaryPrimOp(!91, !c1_7) [Subtract] !c1_8 = I32 [1] !s11 = StreamRange(!c0_6, !92, !c1_8) [-1, False] !true4 = True !101 = StreamFold(!s11, !true4) { (%accum4, %elt10) => !93 = ArrayRef(%76, %elt10) [-1] !94 = GetField(!93) [maxkey] !95 = SelectFields(!94) [(locus)] !c1_9 = I32 [1] !96 = ApplyBinaryPrimOp(%elt10, !c1_9) [Add] !97 = ArrayRef(%76, !96) [-1] !98 = GetField(!97) [minkey] !99 = SelectFields(!98) [(locus)] !100 = ApplyComparisonOp(!95, !99) [LTEQ] ApplySpecial(%accum4, !100) [-1, land, (), Boolean] } !102 = ApplySpecial(!90, !101) [-1, land, (), Boolean] MakeStruct(ksorted: !88, pksorted: !102, sortedPartData: %76) 2024-02-18 20:31:23.243 : INFO: instruction count: 3: __C34HailClassLoaderContainer. 2024-02-18 20:31:23.243 : INFO: instruction count: 3: __C34HailClassLoaderContainer. 2024-02-18 20:31:23.244 : INFO: instruction count: 3: __C36FSContainer. 2024-02-18 20:31:23.244 : INFO: instruction count: 3: __C36FSContainer. 2024-02-18 20:31:23.681 : INFO: instruction count: 3: __C38collect_distributed_array_table_coerce_sortedness. 2024-02-18 20:31:23.682 : INFO: instruction count: 150: __C38collect_distributed_array_table_coerce_sortedness.apply 2024-02-18 20:31:23.683 : INFO: instruction count: 17: __C38collect_distributed_array_table_coerce_sortedness.apply 2024-02-18 20:31:23.683 : INFO: instruction count: 27: __C38collect_distributed_array_table_coerce_sortedness.__m40DECODE_r_struct_of_r_struct_of_r_int32ANDr_int32ANDr_binaryANDr_int64ANDr_int64ANDr_boolENDEND_TO_SBaseStructPointer 2024-02-18 20:31:23.684 : INFO: instruction count: 62: __C38collect_distributed_array_table_coerce_sortedness.__m41INPLACE_DECODE_r_struct_of_r_int32ANDr_int32ANDr_binaryANDr_int64ANDr_int64ANDr_boolEND_TO_r_struct_of_r_int32ANDr_int32ANDr_stringANDr_int64ANDr_int64ANDr_boolEND 2024-02-18 20:31:23.684 : INFO: instruction count: 10: __C38collect_distributed_array_table_coerce_sortedness.__m42INPLACE_DECODE_r_int32_TO_r_int32 2024-02-18 20:31:23.684 : INFO: instruction count: 31: __C38collect_distributed_array_table_coerce_sortedness.__m43INPLACE_DECODE_r_binary_TO_r_string 2024-02-18 20:31:23.684 : INFO: instruction count: 10: __C38collect_distributed_array_table_coerce_sortedness.__m44INPLACE_DECODE_r_int64_TO_r_int64 2024-02-18 20:31:23.685 : INFO: instruction count: 10: __C38collect_distributed_array_table_coerce_sortedness.__m45INPLACE_DECODE_r_bool_TO_r_bool 2024-02-18 20:31:23.685 : INFO: instruction count: 27: __C38collect_distributed_array_table_coerce_sortedness.__m47DECODE_r_struct_of_r_struct_of_ENDEND_TO_SBaseStructPointer 2024-02-18 20:31:23.685 : INFO: instruction count: 8: __C38collect_distributed_array_table_coerce_sortedness.__m48INPLACE_DECODE_r_struct_of_END_TO_r_struct_of_END 2024-02-18 20:31:23.688 : INFO: instruction count: 319: __C38collect_distributed_array_table_coerce_sortedness.__m50split_Let 2024-02-18 20:31:23.688 : INFO: instruction count: 15: __C38collect_distributed_array_table_coerce_sortedness.__m67i_gt_j 2024-02-18 20:31:23.689 : INFO: instruction count: 67: __C38collect_distributed_array_table_coerce_sortedness.__m68ord_compareNonnull 2024-02-18 20:31:23.689 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m69ord_compare 2024-02-18 20:31:23.707 : INFO: instruction count: 159: __C38collect_distributed_array_table_coerce_sortedness.__m70ord_compareNonnull 2024-02-18 20:31:23.707 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m71ord_compare 2024-02-18 20:31:23.708 : INFO: instruction count: 72: __C38collect_distributed_array_table_coerce_sortedness.__m72ord_compareNonnull 2024-02-18 20:31:23.709 : INFO: instruction count: 89: __C38collect_distributed_array_table_coerce_sortedness.__m75ord_compareNonnull 2024-02-18 20:31:23.709 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m76ord_compare 2024-02-18 20:31:23.709 : INFO: instruction count: 80: __C38collect_distributed_array_table_coerce_sortedness.__m77ord_compareNonnull 2024-02-18 20:31:23.710 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m78ord_compare 2024-02-18 20:31:23.710 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m79ord_compare 2024-02-18 20:31:23.710 : INFO: instruction count: 8: __C38collect_distributed_array_table_coerce_sortedness.__m80ord_compareNonnull 2024-02-18 20:31:23.710 : INFO: instruction count: 18: __C38collect_distributed_array_table_coerce_sortedness.__m81swap 2024-02-18 20:31:23.711 : INFO: instruction count: 74: __C38collect_distributed_array_table_coerce_sortedness.__m82rebalance_up 2024-02-18 20:31:23.712 : INFO: instruction count: 150: __C38collect_distributed_array_table_coerce_sortedness.__m83rebalance_down 2024-02-18 20:31:23.712 : INFO: instruction count: 15: __C38collect_distributed_array_table_coerce_sortedness.__m95i_gt_j 2024-02-18 20:31:23.712 : INFO: instruction count: 18: __C38collect_distributed_array_table_coerce_sortedness.__m96swap 2024-02-18 20:31:23.712 : INFO: instruction count: 74: __C38collect_distributed_array_table_coerce_sortedness.__m97rebalance_up 2024-02-18 20:31:23.713 : INFO: instruction count: 150: __C38collect_distributed_array_table_coerce_sortedness.__m98rebalance_down 2024-02-18 20:31:23.713 : INFO: instruction count: 15: __C38collect_distributed_array_table_coerce_sortedness.__m112i_gt_j 2024-02-18 20:31:23.714 : INFO: instruction count: 75: __C38collect_distributed_array_table_coerce_sortedness.__m113ord_compareNonnull 2024-02-18 20:31:23.714 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m114ord_compare 2024-02-18 20:31:23.714 : INFO: instruction count: 8: __C38collect_distributed_array_table_coerce_sortedness.__m115ord_compareNonnull 2024-02-18 20:31:23.715 : INFO: instruction count: 18: __C38collect_distributed_array_table_coerce_sortedness.__m116swap 2024-02-18 20:31:23.715 : INFO: instruction count: 74: __C38collect_distributed_array_table_coerce_sortedness.__m117rebalance_up 2024-02-18 20:31:23.716 : INFO: instruction count: 150: __C38collect_distributed_array_table_coerce_sortedness.__m118rebalance_down 2024-02-18 20:31:23.716 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m119begin_group_0 2024-02-18 20:31:23.738 : INFO: instruction count: 403: __C38collect_distributed_array_table_coerce_sortedness.__m120begin_group_0 2024-02-18 20:31:23.740 : INFO: instruction count: 506: __C38collect_distributed_array_table_coerce_sortedness.__m121split_StreamFor 2024-02-18 20:31:23.741 : INFO: instruction count: 216: __C38collect_distributed_array_table_coerce_sortedness.__m140begin_group_0 2024-02-18 20:31:23.741 : INFO: instruction count: 18: __C38collect_distributed_array_table_coerce_sortedness.__m141begin_group_0 2024-02-18 20:31:23.742 : INFO: instruction count: 5: __C38collect_distributed_array_table_coerce_sortedness.__m149split_Begin 2024-02-18 20:31:23.743 : INFO: instruction count: 263: __C38collect_distributed_array_table_coerce_sortedness.__m151begin_group_0 2024-02-18 20:31:23.745 : INFO: instruction count: 365: __C38collect_distributed_array_table_coerce_sortedness.__m152take_by_seqop 2024-02-18 20:31:23.745 : INFO: instruction count: 156: __C38collect_distributed_array_table_coerce_sortedness.__m153take_by_garbage_collect 2024-02-18 20:31:23.765 : INFO: instruction count: 365: __C38collect_distributed_array_table_coerce_sortedness.__m154take_by_seqop 2024-02-18 20:31:23.765 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m155ord_compare 2024-02-18 20:31:23.765 : INFO: instruction count: 159: __C38collect_distributed_array_table_coerce_sortedness.__m156ord_compareNonnull 2024-02-18 20:31:23.766 : INFO: instruction count: 156: __C38collect_distributed_array_table_coerce_sortedness.__m157take_by_garbage_collect 2024-02-18 20:31:23.766 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m158ord_lteq 2024-02-18 20:31:23.767 : INFO: instruction count: 204: __C38collect_distributed_array_table_coerce_sortedness.__m159ord_lteqNonnull 2024-02-18 20:31:23.767 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m160ord_lteq 2024-02-18 20:31:23.767 : INFO: instruction count: 23: __C38collect_distributed_array_table_coerce_sortedness.__m161ord_lteqNonnull 2024-02-18 20:31:23.767 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m162ord_equiv 2024-02-18 20:31:23.768 : INFO: instruction count: 23: __C38collect_distributed_array_table_coerce_sortedness.__m163ord_equivNonnull 2024-02-18 20:31:23.768 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m164ord_lteq 2024-02-18 20:31:23.768 : INFO: instruction count: 95: __C38collect_distributed_array_table_coerce_sortedness.__m165ord_lteqNonnull 2024-02-18 20:31:23.768 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m166ord_lteq 2024-02-18 20:31:23.769 : INFO: instruction count: 19: __C38collect_distributed_array_table_coerce_sortedness.__m167ord_lteqNonnull 2024-02-18 20:31:23.769 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m168ord_equiv 2024-02-18 20:31:23.769 : INFO: instruction count: 19: __C38collect_distributed_array_table_coerce_sortedness.__m169ord_equivNonnull 2024-02-18 20:31:23.769 : INFO: instruction count: 13: __C38collect_distributed_array_table_coerce_sortedness.__m170ord_equiv 2024-02-18 20:31:23.770 : INFO: instruction count: 79: __C38collect_distributed_array_table_coerce_sortedness.__m171ord_equivNonnull 2024-02-18 20:31:23.770 : INFO: instruction count: 5: __C38collect_distributed_array_table_coerce_sortedness.__m172toInt64 2024-02-18 20:31:23.770 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m175ord_lteq 2024-02-18 20:31:23.770 : INFO: instruction count: 104: __C38collect_distributed_array_table_coerce_sortedness.__m176ord_lteqNonnull 2024-02-18 20:31:23.771 : INFO: instruction count: 363: __C38collect_distributed_array_table_coerce_sortedness.__m179take_by_seqop 2024-02-18 20:31:23.772 : INFO: instruction count: 390: __C38collect_distributed_array_table_coerce_sortedness.__m180take_by_garbage_collect 2024-02-18 20:31:23.794 : INFO: instruction count: 366: __C38collect_distributed_array_table_coerce_sortedness.__m181take_by_result 2024-02-18 20:31:23.795 : INFO: instruction count: 28: __C38collect_distributed_array_table_coerce_sortedness.__m182result_quicksort 2024-02-18 20:31:23.795 : INFO: instruction count: 19: __C38collect_distributed_array_table_coerce_sortedness.__m183quicksort_swap 2024-02-18 20:31:23.795 : INFO: instruction count: 185: __C38collect_distributed_array_table_coerce_sortedness.__m184quicksort_partition 2024-02-18 20:31:23.796 : INFO: instruction count: 366: __C38collect_distributed_array_table_coerce_sortedness.__m185take_by_result 2024-02-18 20:31:23.796 : INFO: instruction count: 28: __C38collect_distributed_array_table_coerce_sortedness.__m186result_quicksort 2024-02-18 20:31:23.797 : INFO: instruction count: 19: __C38collect_distributed_array_table_coerce_sortedness.__m187quicksort_swap 2024-02-18 20:31:23.797 : INFO: instruction count: 185: __C38collect_distributed_array_table_coerce_sortedness.__m188quicksort_partition 2024-02-18 20:31:23.798 : INFO: instruction count: 366: __C38collect_distributed_array_table_coerce_sortedness.__m189take_by_result 2024-02-18 20:31:23.798 : INFO: instruction count: 28: __C38collect_distributed_array_table_coerce_sortedness.__m190result_quicksort 2024-02-18 20:31:23.798 : INFO: instruction count: 19: __C38collect_distributed_array_table_coerce_sortedness.__m191quicksort_swap 2024-02-18 20:31:23.799 : INFO: instruction count: 185: __C38collect_distributed_array_table_coerce_sortedness.__m192quicksort_partition 2024-02-18 20:31:23.799 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m218ord_equiv 2024-02-18 20:31:23.799 : INFO: instruction count: 16: __C38collect_distributed_array_table_coerce_sortedness.__m219ord_equivNonnull 2024-02-18 20:31:23.799 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.__m232ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDANDr_boolANDr_boolANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDENDEND 2024-02-18 20:31:23.800 : INFO: instruction count: 109: __C38collect_distributed_array_table_coerce_sortedness.__m233ENCODE_SBaseStructPointer_TO_r_struct_of_r_int64ANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDANDr_boolANDr_boolANDr_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryENDEND 2024-02-18 20:31:23.800 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.__m234ENCODE_SInt64$_TO_r_int64 2024-02-18 20:31:23.800 : INFO: instruction count: 35: __C38collect_distributed_array_table_coerce_sortedness.__m235ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryEND 2024-02-18 20:31:23.800 : INFO: instruction count: 51: __C38collect_distributed_array_table_coerce_sortedness.__m236ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_r_binaryANDr_int32ENDANDr_array_of_r_binaryEND 2024-02-18 20:31:23.800 : INFO: instruction count: 25: __C38collect_distributed_array_table_coerce_sortedness.__m237ENCODE_SCanonicalLocusPointer_TO_r_struct_of_r_binaryANDr_int32END 2024-02-18 20:31:23.801 : INFO: instruction count: 16: __C38collect_distributed_array_table_coerce_sortedness.__m238ENCODE_SStringPointer_TO_r_binary 2024-02-18 20:31:23.801 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.__m239ENCODE_SInt32$_TO_r_int32 2024-02-18 20:31:23.801 : INFO: instruction count: 39: __C38collect_distributed_array_table_coerce_sortedness.__m240ENCODE_SIndexablePointer_TO_r_array_of_r_binary 2024-02-18 20:31:23.801 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.__m241ENCODE_SBoolean$_TO_r_bool 2024-02-18 20:31:23.801 : INFO: instruction count: 9: __C38collect_distributed_array_table_coerce_sortedness.setPartitionIndex 2024-02-18 20:31:23.801 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.addPartitionRegion 2024-02-18 20:31:23.802 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.setPool 2024-02-18 20:31:23.802 : INFO: instruction count: 3: __C38collect_distributed_array_table_coerce_sortedness.addHailClassLoader 2024-02-18 20:31:23.802 : INFO: instruction count: 3: __C38collect_distributed_array_table_coerce_sortedness.addFS 2024-02-18 20:31:23.802 : INFO: instruction count: 4: __C38collect_distributed_array_table_coerce_sortedness.addTaskContext 2024-02-18 20:31:23.802 : INFO: instruction count: 43: __C38collect_distributed_array_table_coerce_sortedness.addReferenceGenomes 2024-02-18 20:31:23.810 : INFO: instruction count: 3: __C38collect_distributed_array_table_coerce_sortedness.setObjects 2024-02-18 20:31:23.811 : INFO: instruction count: 367: __C38collect_distributed_array_table_coerce_sortedness.__m50split_Let_region12_17 2024-02-18 20:31:23.812 : INFO: instruction count: 751: __C38collect_distributed_array_table_coerce_sortedness.__m121split_StreamFor_region22_22 2024-02-18 20:31:23.812 : INFO: instruction count: 574: __C38collect_distributed_array_table_coerce_sortedness.__m121split_StreamFor_region19_24 2024-02-18 20:31:23.813 : INFO: instruction count: 281: __C38collect_distributed_array_table_coerce_sortedness.__m152take_by_seqop_region2_4 2024-02-18 20:31:23.813 : INFO: instruction count: 273: __C38collect_distributed_array_table_coerce_sortedness.__m152take_by_seqop_region15_17 2024-02-18 20:31:23.813 : INFO: instruction count: 227: __C38collect_distributed_array_table_coerce_sortedness.__m152take_by_seqop_region18_20 2024-02-18 20:31:23.814 : INFO: instruction count: 227: __C38collect_distributed_array_table_coerce_sortedness.__m152take_by_seqop_region5_7 2024-02-18 20:31:23.815 : INFO: instruction count: 486: __C38collect_distributed_array_table_coerce_sortedness.__m153take_by_garbage_collect_region4_12 2024-02-18 20:31:23.815 : INFO: instruction count: 281: __C38collect_distributed_array_table_coerce_sortedness.__m154take_by_seqop_region2_4 2024-02-18 20:31:23.816 : INFO: instruction count: 273: __C38collect_distributed_array_table_coerce_sortedness.__m154take_by_seqop_region15_17 2024-02-18 20:31:23.816 : INFO: instruction count: 227: __C38collect_distributed_array_table_coerce_sortedness.__m154take_by_seqop_region18_20 2024-02-18 20:31:23.816 : INFO: instruction count: 227: __C38collect_distributed_array_table_coerce_sortedness.__m154take_by_seqop_region5_7 2024-02-18 20:31:23.817 : INFO: instruction count: 486: __C38collect_distributed_array_table_coerce_sortedness.__m157take_by_garbage_collect_region4_12 2024-02-18 20:31:23.818 : INFO: instruction count: 261: __C38collect_distributed_array_table_coerce_sortedness.__m179take_by_seqop_region2_4 2024-02-18 20:31:23.818 : INFO: instruction count: 253: __C38collect_distributed_array_table_coerce_sortedness.__m179take_by_seqop_region12_14 2024-02-18 20:31:23.834 : INFO: instruction count: 3: __C245__m50split_LetSpills. 2024-02-18 20:31:23.834 : INFO: instruction count: 3: __C249__m121split_StreamForSpills. 2024-02-18 20:31:23.834 : INFO: instruction count: 3: __C263__m152take_by_seqopSpills. 2024-02-18 20:31:23.834 : INFO: instruction count: 3: __C288__m153take_by_garbage_collectSpills. 2024-02-18 20:31:23.835 : INFO: instruction count: 3: __C294__m154take_by_seqopSpills. 2024-02-18 20:31:23.835 : INFO: instruction count: 3: __C319__m157take_by_garbage_collectSpills. 2024-02-18 20:31:23.835 : INFO: instruction count: 3: __C325__m179take_by_seqopSpills. 2024-02-18 20:31:23.835 : INFO: instruction count: 3: __C73RGContainer_GRCh37. 2024-02-18 20:31:23.836 : INFO: instruction count: 3: __C73RGContainer_GRCh37. 2024-02-18 20:31:23.836 : INFO: instruction count: 3: __C142staticWrapperClass_1. 2024-02-18 20:31:24.128 : INFO: encoder cache miss (0 hits, 2 misses, 0.000) 2024-02-18 20:31:24.138 : INFO: instruction count: 3: __C471HailClassLoaderContainer. 2024-02-18 20:31:24.138 : INFO: instruction count: 3: __C471HailClassLoaderContainer. 2024-02-18 20:31:24.140 : INFO: instruction count: 3: __C473FSContainer. 2024-02-18 20:31:24.140 : INFO: instruction count: 3: __C473FSContainer. 2024-02-18 20:31:24.142 : INFO: instruction count: 3: __C475etypeEncode. 2024-02-18 20:31:24.143 : INFO: instruction count: 7: __C475etypeEncode.apply 2024-02-18 20:31:24.143 : INFO: instruction count: 33: __C475etypeEncode.__m477ENCODE_SBaseStructPointer_TO_r_struct_of_r_struct_of_ENDANDr_array_of_r_struct_of_r_int32ANDr_int32ANDr_binaryANDr_int64ANDr_int64ANDr_boolENDEND 2024-02-18 20:31:24.155 : INFO: instruction count: 1: __C475etypeEncode.__m478ENCODE_SBaseStructPointer_TO_r_struct_of_END 2024-02-18 20:31:24.156 : INFO: instruction count: 35: __C475etypeEncode.__m479ENCODE_SIndexablePointer_TO_r_array_of_r_struct_of_r_int32ANDr_int32ANDr_binaryANDr_int64ANDr_int64ANDr_boolEND 2024-02-18 20:31:24.156 : INFO: instruction count: 73: __C475etypeEncode.__m480ENCODE_SBaseStructPointer_TO_r_struct_of_r_int32ANDr_int32ANDr_binaryANDr_int64ANDr_int64ANDr_boolEND 2024-02-18 20:31:24.156 : INFO: instruction count: 4: __C475etypeEncode.__m481ENCODE_SInt32$_TO_r_int32 2024-02-18 20:31:24.156 : INFO: instruction count: 16: __C475etypeEncode.__m482ENCODE_SStringPointer_TO_r_binary 2024-02-18 20:31:24.156 : INFO: instruction count: 4: __C475etypeEncode.__m483ENCODE_SInt64$_TO_r_int64 2024-02-18 20:31:24.157 : INFO: instruction count: 4: __C475etypeEncode.__m484ENCODE_SBoolean$_TO_r_bool 2024-02-18 20:31:24.157 : ERROR: error while applying lowering 'LowerOrInterpretNonCompilable' 2024-02-18 20:31:24.157 : INFO: RegionPool: initialized for thread 58: HTTP-Dispatcher 2024-02-18 20:31:24.158 : INFO: TaskReport: stage=0, partition=0, attempt=0, peakBytes=0, peakBytesReadable=0.00 B, chunks requested=0, cache hits=0 2024-02-18 20:31:24.158 : INFO: RegionPool: FREE: 0 allocated (0 blocks / 0 chunks), regions.size = 0, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:31:24.158 : INFO: RegionPool: FREE: 128.0K allocated (128.0K blocks / 0 chunks), regions.size = 2, 0 current java objects, thread 58: HTTP-Dispatcher 2024-02-18 20:31:32.056 SparkContext: INFO: Invoking stop() from shutdown hook 2024-02-18 20:31:32.067 AbstractConnector: INFO: Stopped Spark@1c540a68{HTTP/1.1, (http/1.1)}{0.0.0.0:4040} 2024-02-18 20:31:32.074 SparkUI: INFO: Stopped Spark web UI at http://uger-r7-c001.broadinstitute.org:4040 2024-02-18 20:31:32.091 MapOutputTrackerMasterEndpoint: INFO: MapOutputTrackerMasterEndpoint stopped! 2024-02-18 20:31:32.114 MemoryStore: INFO: MemoryStore cleared 2024-02-18 20:31:32.114 BlockManager: INFO: BlockManager stopped 2024-02-18 20:31:32.122 BlockManagerMaster: INFO: BlockManagerMaster stopped 2024-02-18 20:31:32.126 OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: INFO: OutputCommitCoordinator stopped! 2024-02-18 20:31:32.135 SparkContext: INFO: Successfully stopped SparkContext