# Error when reading a file from a google bucket in hail on a google machine **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [December 20, 2020, 7:45pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863 "2020-12-20T19:45:36Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 20, 2020, 7:45pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/1 "2020-12-20T19:45:36Z")
I am trying to run some code on a google megamem machine, and right now am testing the setup process on a standard machine. I am getting an error when I try to use hail to read in a file stored in a google bucket. This is how I start my machine (the jre file is to install java): ```auto gcloud compute instances create lkptest --machine-type n1-standard-8 gcloud compute scp python_code.py lkptest:. gcloud compute scp jre-8u271-linux-x64.tar.gz lkptest:. gcloud compute ssh lkptest ``` Then I install packages: ```auto sudo apt-get update sudo apt-get install python3.6 sudo apt-get install python3-pip pip3 install -U hail pip3 install numpy scipy matplotlib ipython pandas export PATH=$PATH:/home/lilllianpetersen/.local/bin curl https://broad.io/install-gcs-connector | python # To connect a bucket to hail ``` And I use the following commands to install java: ```auto mkdir software/java/ -p mv jre-8u271-linux-x64.tar.gz software/java/ cd software/java/ tar zxvf jre-8u271-linux-x64.tar.gz cd ~ export JAVA_HOME=~/software/java/jre1.8.0_271 export PATH=$JAVA_HOME/bin:$PATH ``` Then I use the following code in python to read in a block matrix: ```auto grm = hl.linalg.BlockMatrix.read("gs://ukb-gt/grm_file") ``` I get the following error after running the python code: ```auto Initializing Hail with default parameters... 2020-12-20 19:24:38 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2020-12-20 19:24:38 WARN Hail:37 - This Hail JAR was compiled for Spark 2.4.5, running with Spark 2.4.1. Compatibility is not guaranteed. Running on Apache Spark version 2.4.1 SparkUI available at http://lkptest.c.ukbb-pca.internal:4040 Welcome to ____ <>__ / /_/ / ____ / / / __ / _ `/ / / /_/ /_/\_,_/_/_/ version 0.2.61-3c86d3ba497a LOGGING: writing to /home/lilllianpetersen/hail-20201220-1924-0.2.61-3c86d3ba497a.log --------------------------------------------------------------------------- FatalError Traceback (most recent call last) ~/megamem_compute_pcas.py in 36 # Convert to numpy 37 print('\n\nCONVERTING GRM TO NUMPY...\n\n') ---> 38 np_grm = grm.to_numpy() 39 print('\n\nSUCCESSFULLY CONVERTED GRM TO NUMPY!\n\n') 40 in to_numpy(self, _force_blocking) ~/.local/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs) 612 def wrapper(__original_func, *args, **kwargs): 613 args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method) --> 614 return __original_func(*args_, **kwargs_) 615 616 return wrapper ~/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py in to_numpy(self, _force_blocking) 1190 """ 1191 -> 1192 if self.n_rows * self.n_cols > 1 << 31 or _force_blocking: 1193 path = new_temp_file() 1194 self.export_blocks(path, binary=True) ~/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py in n_rows(self) 523 :obj:`int` 524 """ --> 525 return self.shape[0] 526 527 @property ~/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py in shape(self) 552 Number of rows and number of columns. 553 """ --> 554 return tensor_shape_to_matrix_shape(self._bmir) 555 556 @property ~/.local/lib/python3.7/site-packages/hail/ir/blockmatrix_ir.py in tensor_shape_to_matrix_shape(bmir) 379 380 def tensor_shape_to_matrix_shape(bmir): --> 381 shape = bmir.typ.shape 382 is_row_vector = bmir.typ.is_row_vector 383 ~/.local/lib/python3.7/site-packages/hail/ir/base_ir.py in typ(self) 383 def typ(self): 384 if self._type is None: --> 385 self._compute_type() 386 assert self._type is not None, self 387 return self._type ~/.local/lib/python3.7/site-packages/hail/ir/blockmatrix_ir.py in _compute_type(self) 23 24 def _compute_type(self): ---> 25 self._type = Env.backend().blockmatrix_type(self) 26 27 ~/.local/lib/python3.7/site-packages/hail/backend/spark_backend.py in blockmatrix_type(self, bmir) 298 299 def blockmatrix_type(self, bmir): --> 300 jir = self._to_java_blockmatrix_ir(bmir) 301 return tblockmatrix._from_java(jir.typ()) 302 ~/.local/lib/python3.7/site-packages/hail/backend/spark_backend.py in _to_java_blockmatrix_ir(self, ir) 268 269 def _to_java_blockmatrix_ir(self, ir): --> 270 return self._to_java_ir(ir, self._parse_blockmatrix_ir) 271 272 def value_type(self, ir): ~/.local/lib/python3.7/site-packages/hail/backend/spark_backend.py in _to_java_ir(self, ir, parse) 255 r = CSERenderer(stop_at_jir=True) 256 # FIXME parse should be static --> 257 ir._jir = parse(r(ir), ir_map=r.jirs) 258 return ir._jir 259 ~/.local/lib/python3.7/site-packages/hail/backend/spark_backend.py in _parse_blockmatrix_ir(self, code, ref_map, ir_map) 236 237 def _parse_blockmatrix_ir(self, code, ref_map={}, ir_map={}): --> 238 return self._jbackend.parse_blockmatrix_ir(code, ref_map, ir_map) 239 240 @property ~/.local/lib/python3.7/site-packages/py4j/java_gateway.py in __call__ (self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -> 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args: ~/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py in deco(*args, **kwargs) 30 raise FatalError('%s\n\nJava stack trace:\n%s\n' 31 'Hail version: %s\n' ---> 32 'Error summary: %s' % (deepest, full, hail. __version__ , deepest), error_id) from None 33 except pyspark.sql.utils.CapturedException as e: 34 raise FatalError('%s\n\nJava stack trace:\n%s\n' FatalError: IOException: No FileSystem for scheme: gs Java stack trace: java.io.IOException: No FileSystem for scheme: gs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at is.hail.io.fs.HadoopFS.getFileSystem(HadoopFS.scala:98) at is.hail.io.fs.HadoopFS.openNoCompression(HadoopFS.scala:80) at is.hail.io.fs.FS$class.open(FS.scala:139) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.io.fs.FS$class.open(FS.scala:148) at is.hail.io.fs.HadoopFS.open(HadoopFS.scala:70) at is.hail.linalg.BlockMatrix$.readMetadata(BlockMatrix.scala:167) at is.hail.expr.ir.BlockMatrixNativeReader$.apply(BlockMatrixIR.scala:125) at is.hail.expr.ir.BlockMatrixNativeReader$.fromJValue(BlockMatrixIR.scala:132) at is.hail.expr.ir.BlockMatrixReader$.fromJValue(BlockMatrixIR.scala:101) at is.hail.expr.ir.IRParser$.blockmatrix_ir1(Parser.scala:1867) at is.hail.expr.ir.IRParser$$anonfun$blockmatrix_ir$1.apply(Parser.scala:1858) at is.hail.expr.ir.IRParser$$anonfun$blockmatrix_ir$1.apply(Parser.scala:1858) at is.hail.utils.StackSafe$More.advance(StackSafe.scala:64) at is.hail.utils.StackSafe$.run(StackSafe.scala:16) at is.hail.utils.StackSafe$StackFrame.run(StackSafe.scala:32) at is.hail.expr.ir.IRParser$$anonfun$parse_blockmatrix_ir$1.apply(Parser.scala:1963) at is.hail.expr.ir.IRParser$$anonfun$parse_blockmatrix_ir$1.apply(Parser.scala:1963) at is.hail.expr.ir.IRParser$.parse(Parser.scala:1946) at is.hail.expr.ir.IRParser$.parse_blockmatrix_ir(Parser.scala:1963) at is.hail.backend.spark.SparkBackend$$anonfun$parse_blockmatrix_ir$1$$anonfun$apply$24.apply(SparkBackend.scala:614) at is.hail.backend.spark.SparkBackend$$anonfun$parse_blockmatrix_ir$1$$anonfun$apply$24.apply(SparkBackend.scala:613) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend$$anonfun$parse_blockmatrix_ir$1.apply(SparkBackend.scala:613) at is.hail.backend.spark.SparkBackend$$anonfun$parse_blockmatrix_ir$1.apply(SparkBackend.scala:612) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.parse_blockmatrix_ir(SparkBackend.scala:612) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Hail version: 0.2.61-3c86d3ba497a Error summary: IOException: No FileSystem for scheme: gs ``` Are my set up steps correct, and how am I supposed to read the file? Thanks! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 21, 2020, 4:51pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/2 "2020-12-21T16:51:15Z")
What was the output from: ```auto curl https://broad.io/install-gcs-connector | python ``` ? That was the step that should configure access to `gs://`. What is the output of this? ```bash find_spark_home.py echo --------------- ls -al $(find_spark_home.py)/jars/gcs* echo --------------- ls -al $(find_spark_home.py)/conf/spark-defaults.conf cat $(find_spark_home.py)/conf/spark-defaults.conf echo --------------- ls -al ~/.config/gcloud/application_default_credentials.json echo --------------- ls -al ~/.config/gcloud/legacy_credentials/*/adc.json ``` ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 21, 2020, 7:26pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/3 "2020-12-21T19:26:47Z")
No, the response of `curl https://broad.io/install-gcs-connector | python` was: ```auto % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 ``` Here are the other responses: ```auto ~$ find_spark_home.py /home/lilllianpetersen/.local/lib/python3.7/site-packages/pyspark ~$ ls -al $(find_spark_home.py)/jars/gcs* ls: cannot access '/home/lilllianpetersen/.local/lib/python3.7/site-packages/pyspark/jars/gcs*': No such file or directory ``` For all the other lines it says no such file or directory ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 21, 2020, 8:43pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/4 "2020-12-21T20:43:41Z")
Ah, this is my bad, `curl` is not following the redirect. Try this: ```auto curl --show-error --silent --location https://broad.io/install-gcs-connector | python ``` That should print a few messages indicating that it has installed the gcs connector. ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 3:37am UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/5 "2020-12-22T03:37:55Z")
This is what I get: ```auto ~$ curl --show-error --silent --location https://broad.io/install-gcs-connector | python File "", line 22 p.error(f"{args.key_file_path} not found") ^ SyntaxError: invalid syntax ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 7:11am UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/6 "2020-12-22T07:11:14Z")
This means `python` on that computer is Python 2. Trying changing it to `python3` ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 6:41pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/7 "2020-12-22T18:41:21Z")
Hmm. I am confused because the command I use to install python is: ```auto sudo apt-get install python3.6 ``` But then I get; ```auto ~$ python --version Python 2.7.16 ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 8:39pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/8 "2020-12-22T20:39:03Z")
`apt-get` probably installs Python 3 as `python3` not `python`. This avoids breaking packages that assume `python` refers to Python 3. You can change this behavior with [`update-alternatives`](https://linux.die.net/man/8/update-alternatives). There’s a [stack overflow post](https://stackoverflow.com/a/46095470/6823256) recommending against changing the default `python` for a system. Personally, I recommend pervasively using `python3` on every system (Linux or otherwise) as it reliably refers to Python 3, not Python 2. ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 9:30pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/9 "2020-12-22T21:30:48Z")
Ok thanks, I seem to almost have it working. I started the megamem and went through all of the steps. It now seems to start the computation, but I soon get this error: ```auto Initializing Hail with default parameters... 2020-12-22 21:27:25 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2020-12-22 21:27:26 WARN Hail:37 - This Hail JAR was compiled for Spark 2.4.5, running with Spark 2.4.1. Compatibility is not guaranteed. Running on Apache Spark version 2.4.1 SparkUI available at http://lkp-megamem.c.ukbb-pca.internal:4040 Welcome to ____ <>__ / /_/ / ____ / / / __ / _ `/ / / /_/ /_/\_,_/_/_/ version 0.2.61-3c86d3ba497a LOGGING: writing to /home/lilllianpetersen/hail-20201222-2127-0.2.61-3c86d3ba497a.log [Stage 0:> (0 + 80) / 81]Traceback (most recent call last): File "megamem_compute_pcas.py", line 38, in np_grm = grm.to_numpy() File "", line 2, in to_numpy File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper return __original_func(*args_, **kwargs_) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py", line 1199, in to_numpy self.tofile(uri) File "", line 2, in tofile File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper return __original_func(*args_, **kwargs_) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py", line 1171, in tofile Env.backend().execute(BlockMatrixWrite(self._bmir, writer)) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 98, in execute raise e File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 74, in execute result = json.loads(self._jhc.backend().executeJSON(jir)) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 32, in deco 'Error summary: %s' % (deepest, full, hail. __version__ , deepest), error_id) from None hail.utils.java.FatalError: OutOfMemoryError: Java heap space Java stack trace: org.apache.spark.SparkException: Job aborted due to stage failure: Task 53 in stage 0.0 failed 1 times, most recent failure: Lost task 53.0 in stage 0.0 (TID 53, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) at org.apache.spark.rdd.RDD.collect(RDD.scala:944) at is.hail.linalg.BlockMatrix.toBreezeMatrix(BlockMatrix.scala:925) at is.hail.expr.ir.BlockMatrixBinaryWriter.apply(BlockMatrixWriter.scala:116) at is.hail.expr.ir.Interpret$.run(Interpret.scala:827) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16) at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:28) at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$_execute(SparkBackend.scala:354) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:338) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:335) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:335) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:379) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:377) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:377) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: Java heap space at Hail version: 0.2.61-3c86d3ba497a Error summary: OutOfMemoryError: Java heap space ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 9:43pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/10 "2020-12-22T21:43:04Z")
How did you invoke python/ipython? You probably need to tell Hail to [use all the RAM available](https://discuss.hail.is/t/how-do-i-increase-the-memory-or-ram-available-to-the-jvm-when-i-start-hail-through-python/133/2). Change `8g` in that answer to something like `3500g`. ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 9:56pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/11 "2020-12-22T21:56:01Z")
Ok, so in my .bashrc I add: ```auto export PYSPARK_SUBMIT_ARGS="--driver-memory 3500g --executor-memory 3500g pyspark-shell" ``` And when I call: ```auto python3 code.py ``` That will automatically have it use all the RAM? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 10:11pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/12 "2020-12-22T22:11:29Z")
It has to be the .bashrc _on the megamem instance_, but, yes, that should do the trick. (Also, I’m assuming you’re using bash (check with `echo $SHELL`)) ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 10:12pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/13 "2020-12-22T22:12:33Z")
`echo $SHELL` returns /bin/bash I tried adding that line on the megamem bashrc and I got the same error ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 10:17pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/14 "2020-12-22T22:17:16Z")
I did: ```auto PYSPARK_SUBMIT_ARGS="--driver-memory 3500g --executor-memory 3500g pyspark-shell" python3 megamem_compute_pcas.py ``` And it seems to have worked! Thanks for all of your help! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 10:19pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/15 "2020-12-22T22:19:44Z")
I’m not sure if this is clear, but after updating `~/.bashrc`, you have to either restart your session (logout/login) or run `source ~/.bashrc` for the changes to take effect. ---
**Author:** ![lilliank](https://avatars.discourse-cdn.com/v4/letter/l/ecc23a/32.png) [@lilliank](https://discuss.hail.is/u/lilliank)\ **Post date:** [December 22, 2020, 10:23pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/16 "2020-12-22T22:23:14Z")
Ok I did that, and seem to be getting past that error. But am now getting another error: ```auto Initializing Hail with default parameters... 2020-12-22 22:20:27 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2020-12-22 22:20:27 WARN Hail:37 - This Hail JAR was compiled for Spark 2.4.5, running with Spark 2.4.1. Compatibility is not guaranteed. Running on Apache Spark version 2.4.1 SparkUI available at http://lkp-megamem.c.ukbb-pca.internal:4040 Welcome to ____ <>__ / /_/ / ____ / / / __ / _ `/ / / /_/ /_/\_,_/_/_/ version 0.2.61-3c86d3ba497a LOGGING: writing to /home/lilllianpetersen/hail-20201222-2220-0.2.61-3c86d3ba497a.log [Stage 0:========================================================>(80 + 1) / 81]log4j:ERROR Failed to write [2020-12-22 22:22:31 root: ERROR: IOException: No space left on device From org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:248) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.close(FilterOutputStream.java:158) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:413) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at is.hail.io.fs.HadoopFS$$anon$1.close(HadoopFS.scala:39) at java.io.FilterOutputStream.close(FilterOutputStream.java:159) at is.hail.utils.package$.using(package.scala:638) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:29) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:26) at is.hail.utils.richUtils.RichDenseMatrixDouble$.exportToDoubles(RichDenseMatrixDouble.scala:56) at is.hail.expr.ir.BlockMatrixBinaryWriter.apply(BlockMatrixWriter.scala:116) at is.hail.expr.ir.Interpret$.run(Interpret.scala:827) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16) at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:28) at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$_execute(SparkBackend.scala:354) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:338) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:335) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:335) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:379) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:377) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:377) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:246) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.close(FilterOutputStream.java:158) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:413) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at is.hail.io.fs.HadoopFS$$anon$1.close(HadoopFS.scala:39) at java.io.FilterOutputStream.close(FilterOutputStream.java:159) at is.hail.utils.package$.using(package.scala:638) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:29) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:26) at is.hail.utils.richUtils.RichDenseMatrixDouble$.exportToDoubles(RichDenseMatrixDouble.scala:56) at is.hail.expr.ir.BlockMatrixBinaryWriter.apply(BlockMatrixWriter.scala:116) at is.hail.expr.ir.Interpret$.run(Interpret.scala:827) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16) at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:28) at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$_execute(SparkBackend.scala:354) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:338) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:335) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:335) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:379) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:377) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:377) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) ]. java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220) at java.io.Writer.write(Writer.java:157) at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310) at org.apache.log4j.WriterAppender.append(WriterAppender.java:162) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.error(Category.java:305) at is.hail.utils.ErrorHandling$class.handleForPython(ErrorHandling.scala:56) at is.hail.utils.package$.handleForPython(package.scala:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Traceback (most recent call last): File "megamem_compute_pcas.py", line 38, in np_grm = grm.to_numpy() File "", line 2, in to_numpy File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper return __original_func(*args_, **kwargs_) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py", line 1199, in to_numpy self.tofile(uri) File "", line 2, in tofile File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper return __original_func(*args_, **kwargs_) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/linalg/blockmatrix.py", line 1171, in tofile Env.backend().execute(BlockMatrixWrite(self._bmir, writer)) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 98, in execute raise e File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 74, in execute result = json.loads(self._jhc.backend().executeJSON(jir)) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/py4j/java_gateway.py", line 1257, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/home/lilllianpetersen/.local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 32, in deco 'Error summary: %s' % (deepest, full, hail. __version__ , deepest), error_id) from None hail.utils.java.FatalError: IOException: No space left on device Java stack trace: org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:248) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.close(FilterOutputStream.java:158) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:413) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at is.hail.io.fs.HadoopFS$$anon$1.close(HadoopFS.scala:39) at java.io.FilterOutputStream.close(FilterOutputStream.java:159) at is.hail.utils.package$.using(package.scala:638) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:29) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:26) at is.hail.utils.richUtils.RichDenseMatrixDouble$.exportToDoubles(RichDenseMatrixDouble.scala:56) at is.hail.expr.ir.BlockMatrixBinaryWriter.apply(BlockMatrixWriter.scala:116) at is.hail.expr.ir.Interpret$.run(Interpret.scala:827) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16) at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:28) at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$_execute(SparkBackend.scala:354) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:338) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:335) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:335) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:379) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:377) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:377) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:246) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.close(FilterOutputStream.java:158) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:413) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at is.hail.io.fs.HadoopFS$$anon$1.close(HadoopFS.scala:39) at java.io.FilterOutputStream.close(FilterOutputStream.java:159) at is.hail.utils.package$.using(package.scala:638) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:29) at is.hail.utils.richUtils.RichArray$.exportToDoubles(RichArray.scala:26) at is.hail.utils.richUtils.RichDenseMatrixDouble$.exportToDoubles(RichDenseMatrixDouble.scala:56) at is.hail.expr.ir.BlockMatrixBinaryWriter.apply(BlockMatrixWriter.scala:116) at is.hail.expr.ir.Interpret$.run(Interpret.scala:827) at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16) at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53) at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:67) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15) at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13) at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:81) at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13) at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:62) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:14) at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:12) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:12) at is.hail.expr.ir.CompileAndEvaluate$._apply(CompileAndEvaluate.scala:28) at is.hail.backend.spark.SparkBackend.is$hail$backend$spark$SparkBackend$$_execute(SparkBackend.scala:354) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:338) at is.hail.backend.spark.SparkBackend$$anonfun$execute$1.apply(SparkBackend.scala:335) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:25) at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:23) at is.hail.utils.package$.using(package.scala:618) at is.hail.annotations.Region$.scoped(Region.scala:18) at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:23) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:247) at is.hail.backend.spark.SparkBackend.execute(SparkBackend.scala:335) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:379) at is.hail.backend.spark.SparkBackend$$anonfun$7.apply(SparkBackend.scala:377) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.backend.spark.SparkBackend.executeJSON(SparkBackend.scala:377) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Hail version: 0.2.61-3c86d3ba497a Error summary: IOException: No space left on device ``` ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [December 22, 2020, 11:26pm UTC](https://discuss.hail.is/t/error-when-reading-a-file-from-a-google-bucket-in-hail-on-a-google-machine/1863/17 "2020-12-22T23:26:23Z")
You’ve run out of hard drive space. It looks like the current version of `BlockMatrix.to_numpy` serializes the data to disk and then reads it back in as a numpy array. You’ll need to increase the boot disk size to be big enough to store the grm. The size should be roughly n\_rows \* n\_cols \* 8 bytes.