Java error when trying to write a VDS

Help [0.1]
1

Hi there,

I’m trying to get familiar with hail and running a few basic commands to get my feet wet.

I have been able to import_vcf, report() and count() the vds. However when I tried to write out the vds I get a java.FatalError

[eddip@victorchang.edu.au@vclefkas01 /short/software 16:59:01 j:0] $ spark-submit /mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/W_06_hailImport.py
Running on Apache Spark version 2.1.0
SparkUI available at http://129.94.111.229:4040
Welcome to
     __  __     <>__
    / /_/ /__  __/ /
   / __  / _ `/ / /
  /_/ /_/\_,_/_/_/   version 0.1-69ec30f
[Stage 0:============================================>              (3 + 1) / 4]2017-10-25 16:59:57 Hail: INFO: Multiallelic variants detected. Some methods require splitting or filtering multiallelics first.
2017-10-25 16:59:58 Hail: INFO: Coerced sorted dataset
2017-10-25 16:59:58 Hail: INFO: while importing:
    all_17102017_vqsr.normalized.Y.vcf.bgz  import clean
[Stage 2:============================================>              (3 + 1) / 4]364 75577
Traceback (most recent call last):
  File "/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/W_06_hailImport.py", line 10, in <module>
    vds.write('/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/all_17102017_vqsr.normalized.vds') #
  File "<decorator-gen-285>", line 2, in write
  File "/short/software/hail/python/hail/java.py", line 121, in handle_py4j
    'Error summary: %s' % (deepest, full, Env.hc().version, deepest))
hail.java.FatalError

I saw a few other posts about write errors and there were suggestion of --config additions to spark call but didn’t work for me.

Hope you can help.
Thanks.
Eddie

2

Is this the full error message? If so, we need to first fix whatever is causing the real message to get dropped.

3

Looking a bit closer, I bet that what’s happened is the Hail jar isn’t visible to the executors. I definitely want to see the rest of the error message if it exists though!

4

Hi Tim,

That’s the only error message I see back on the console. However, I have found the hail.log, and within that I see an error for changing permissions :

The command - vds.write(‘/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/all_17102017_vqsr.normalized.vds’)
The error in the hail.log

2017-10-26 10:42:20 root: ERROR: ExitCodeException: chmod: changing permissions of ‘/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/all_17102017_vqsr.normalized.vds/metadata.json.gz’: Operation not permitted

From org.apache.hadoop.util.Shell$ExitCodeException: chmod: changing permissions of ‘/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/all_17102017_vqsr.normalized.vds/metadata.json.gz’: Operation not permitted

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
	at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
	at is.hail.utils.richUtils.RichHadoopConfiguration$.is$hail$utils$richUtils$RichHadoopConfiguration$$create$extension(RichHadoopConfiguration.scala:22)
	at is.hail.utils.richUtils.RichHadoopConfiguration$.writeTextFile$extension(RichHadoopConfiguration.scala:243)
	at is.hail.variant.VariantSampleMatrix.writeMetadata(VariantSampleMatrix.scala:2118)
	at is.hail.variant.VariantDatasetFunctions$.write$extension(VariantDataset.scala:730)
	at is.hail.variant.VariantDatasetFunctions.write(VariantDataset.scala:722)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:280)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:745)

I changed the location of where I’m writing the vds to a local directory in my system and the write now completes with no error, vds (folder) created.

So I’m now able to write out my vds.

Eddie

5

Well we’ve never seen this error before. I’m also confused how it didn’t bubble back to the Python error message.

6

It definitely looks like the user that Spark executes as doesn’t have permission to edit:

/mnt/smb/giannoulatou_lab/Eddie_Ip/working/software/hail_data/

What are the permissions on this folder?

7

Hi Dan,

[eddip@victorchang.edu.au@vclefkas01 /mnt/smb/giannoulatou_lab/Eddie_Ip/working/software 09:05:37 j:0] $ ls -l
total 240640
-rwxrwxrwx 1 root root  29797459 Oct 23 18:22 Hail-0.1-69ec30fa6c74-Spark-2.1.0.zip
drwxrwxrwx 2 root root         0 Oct 26 10:46 hail_data
-rwxrwxrwx 1 root root  19741785 Oct 24 12:09 scala-2.12.4.tgz
-rwxrwxrwx 1 root root 195636829 Oct 23 18:27 spark-2.1.0-bin-hadoop2.7.tgz

the write does create a folder “all_17102017_vqsr.normalized.vds”, and I can see an empty “metadata.json.gz” within the vds folder.

Eddie

8

All those directories are owned by root. According to the chmod man page:

Only the owner of a file or the super-user is permitted to change the mode of a file.

I’m fairly confident that your Spark workers are not running as the root user (this is not a good idea anyway). Those directories should really be owned by whichever user your Spark jobs are running as. I think that’s usually your username (you can check your username with whoami). If your username doesn’t work, ask whomever administrates your Spark cluster what user Spark jobs run as.

Aside: Spark really shouldn’t set the file mode like this. Unfortunately, Spark does not make this easy for us to change. I’ll look into mitigation strategies. Sorry about this.

9

Thanks for your response Dan.

Eddie