2022-05-05 09:48:49 Hail: INFO: Running Hail version 0.2.78-b17627756568 2022-05-05 09:48:49 SharedState: INFO: loading hive config file: file:/cluster/spark/conf/hive-site.xml 2022-05-05 09:48:49 SharedState: INFO: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/cluster/spark/spark-warehouse'). 2022-05-05 09:48:49 SharedState: INFO: Warehouse path is '/cluster/spark/spark-warehouse'. 2022-05-05 09:48:49 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@1581709e{/SQL,null,AVAILABLE,@Spark} 2022-05-05 09:48:49 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@482d3296{/SQL/json,null,AVAILABLE,@Spark} 2022-05-05 09:48:49 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@7ee5bf78{/SQL/execution,null,AVAILABLE,@Spark} 2022-05-05 09:48:49 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@5094544d{/SQL/execution/json,null,AVAILABLE,@Spark} 2022-05-05 09:48:49 ContextHandler: INFO: Started o.s.j.s.ServletContextHandler@231068db{/static/sql,null,AVAILABLE,@Spark} 2022-05-05 09:48:49 StateStoreCoordinatorRef: INFO: Registered StateStoreCoordinator endpoint 2022-05-05 09:48:49 SparkSession$Builder: WARN: Using an existing SparkSession; some configuration may not take effect. 2022-05-05 09:48:49 DNAxApiSvc: INFO: initializing DNAxApiSvc 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] Initializing file system (URI = dnax://database-G9bx6F8JbB0gkzyXJ8f584Y0/tmp): com.dnanexus.hadoop.fs.DNAxFileSystem@732000ba 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] inputStreamBufferSize: 33554432 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] inputStreamMaxAttempts: 6 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] readSocketExceptionMaxAttempts: 2 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] writeSocketExceptionMaxAttempts: 2 2022-05-05 09:48:49 DNAxFileSystem: INFO: [initialize] outputStreamBufferSize: 33554432 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] inputStreamRetryableCodes: [500, 503] 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] fileStatusCacheEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] fileStatusCacheSiblingsEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] statusListUrlCacheEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] readFileUrlCacheEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] listStatusSignedUrlEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] fileStatusSignedUrlEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] createDirectorySemaphoreEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] fileStatusSemaphoreEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] listStatusSemaphoreEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] readFileSemaphoreEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] deleteObjectsInBatchesEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] parallelPartUploadEnabled: true 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] uploadChunkSize: 16777216 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] multiPartThreshold: 1073741824 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] semaphorePermitWaitTime: 10 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] semaphoreCreateDirectoryPermitCount: 1 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] semaphoreFileStatusPermitCount: 1 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] semaphoreListStatusPermitCount: 1 2022-05-05 09:48:50 DNAxFileSystem: INFO: [initialize] semaphoreReadFilePermitCount: 1 2022-05-05 09:48:50 routes: INFO: [getFileStatusUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getFileStatusUrl: input = {"filename":"tmp","jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 09:48:50 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 09:48:50 routes: INFO: [getStatusListUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getStatusListUrl: input = {"filename":"tmp","marker":"","recurse":false,"maxKeys":1,"jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 09:48:50 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:01:18 root: INFO: RegionPool: initialized for thread 24: Thread-4 2022-05-05 10:01:18 MemoryStore: INFO: Block broadcast_0 stored as values in memory (estimated size 253.7 KB, free 11.5 GB) 2022-05-05 10:01:19 MemoryStore: INFO: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.3 KB, free 11.5 GB) 2022-05-05 10:01:19 BlockManagerInfo: INFO: Added broadcast_0_piece0 in memory on ip-10-60-98-113.eu-west-2.compute.internal:43002 (size: 24.3 KB, free: 11.5 GB) 2022-05-05 10:01:19 SparkContext: INFO: Created broadcast 0 from broadcast at SparkBackend.scala:278 2022-05-05 10:01:19 root: INFO: globbing path file:///mnt/project/Bulk/Exome sequences/Population level exome OQFE variants, BGEN format - interim 450k release/ukb23150_c21_b0_v1.bgen returned 1 files: ukb23150_c21_b0_v1.bgen 2022-05-05 10:01:19 routes: INFO: [getFileStatusUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getFileStatusUrl: input = {"filename":"exome/wes_450k/bgen/chr21.idx2","jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:01:19 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:01:19 routes: INFO: [getStatusListUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getStatusListUrl: input = {"filename":"exome/wes_450k/bgen/chr21.idx2","marker":"","recurse":false,"maxKeys":1,"jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:01:19 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:01:20 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 24: Thread-4 2022-05-05 10:01:20 root: ERROR: HailException: Hail only supports zlib compression. From is.hail.utils.HailException: Hail only supports zlib compression. at is.hail.utils.ErrorHandling$class.fatal(ErrorHandling.scala:11) at is.hail.utils.package$.fatal(package.scala:78) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:119) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:93) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:92) at is.hail.utils.package$.using(package.scala:638) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:92) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at is.hail.io.bgen.LoadBgen$.getFileHeaders(LoadBgen.scala:252) at is.hail.io.bgen.IndexBgen$.apply(IndexBgen.scala:63) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:460) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:459) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:46) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:459) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:458) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyIndexBgen(SparkBackend.scala:458) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2022-05-05 10:03:36 root: INFO: RegionPool: initialized for thread 24: Thread-4 2022-05-05 10:03:36 root: INFO: globbing path file:///mnt/project/Bulk/Exome sequences/Population level exome OQFE variants, BGEN format - interim 450k release/ukb23150_c10_b0_v1.bgen returned 1 files: ukb23150_c10_b0_v1.bgen 2022-05-05 10:03:36 routes: INFO: [getFileStatusUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getFileStatusUrl: input = {"filename":"exome/wes_450k/bgen/chr10.idx2","jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:03:36 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:03:36 routes: INFO: [getStatusListUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getStatusListUrl: input = {"filename":"exome/wes_450k/bgen/chr10.idx2","marker":"","recurse":false,"maxKeys":1,"jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:03:36 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:03:38 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 24: Thread-4 2022-05-05 10:03:38 root: ERROR: HailException: Hail only supports zlib compression. From is.hail.utils.HailException: Hail only supports zlib compression. at is.hail.utils.ErrorHandling$class.fatal(ErrorHandling.scala:11) at is.hail.utils.package$.fatal(package.scala:78) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:119) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:93) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:92) at is.hail.utils.package$.using(package.scala:638) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:92) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at is.hail.io.bgen.LoadBgen$.getFileHeaders(LoadBgen.scala:252) at is.hail.io.bgen.IndexBgen$.apply(IndexBgen.scala:63) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:460) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:459) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:46) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:459) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:458) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyIndexBgen(SparkBackend.scala:458) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) 2022-05-05 10:04:58 root: INFO: RegionPool: initialized for thread 24: Thread-4 2022-05-05 10:04:58 root: INFO: globbing path file:///mnt/project/Bulk/Exome sequences/Population level exome OQFE variants, BGEN format - interim 450k release/ukb23150_c10_b0_v1.bgen returned 1 files: ukb23150_c10_b0_v1.bgen 2022-05-05 10:04:58 routes: INFO: [getFileStatusUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getFileStatusUrl: input = {"filename":"exome/wes_450k/bgen/chr10.bgen.idx2","jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:04:58 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:04:58 routes: INFO: [getStatusListUrl] Calling api server route /database-G9bx6F8JbB0gkzyXJ8f584Y0/getStatusListUrl: input = {"filename":"exome/wes_450k/bgen/chr10.bgen.idx2","marker":"","recurse":false,"maxKeys":1,"jobInfo":{"dxJobId":"job-G9kkVf8JbB0QzJBJF0J5xyGF"},"extras":{}} 2022-05-05 10:04:58 DNAxApiSvc: INFO: httpsClientConnManager(public2): [leased: 0; pending: 0; available: 0; max: 10] 2022-05-05 10:04:59 root: INFO: RegionPool: FREE: 64.0K allocated (64.0K blocks / 0 chunks), regions.size = 1, 0 current java objects, thread 24: Thread-4 2022-05-05 10:04:59 root: ERROR: HailException: Hail only supports zlib compression. From is.hail.utils.HailException: Hail only supports zlib compression. at is.hail.utils.ErrorHandling$class.fatal(ErrorHandling.scala:11) at is.hail.utils.package$.fatal(package.scala:78) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:119) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:93) at is.hail.io.bgen.LoadBgen$$anonfun$readState$1.apply(LoadBgen.scala:92) at is.hail.utils.package$.using(package.scala:638) at is.hail.io.bgen.LoadBgen$.readState(LoadBgen.scala:92) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at is.hail.io.bgen.LoadBgen$$anonfun$getFileHeaders$1.apply(LoadBgen.scala:252) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at is.hail.io.bgen.LoadBgen$.getFileHeaders(LoadBgen.scala:252) at is.hail.io.bgen.IndexBgen$.apply(IndexBgen.scala:63) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:460) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1$$anonfun$apply$6.apply(SparkBackend.scala:459) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:46) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:459) at is.hail.backend.spark.SparkBackend$$anonfun$pyIndexBgen$1.apply(SparkBackend.scala:458) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyIndexBgen(SparkBackend.scala:458) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)