I cannot start a Hail cluster when using the optional-components Dataproc flag

Hail Query & hailctl
1

I am using the most updated hail version. I put down the following command to initiate the cluster but I am getting the following problem. Kindly help:

hailctl dataproc --beta start hailjupy --vep GRCh37 --optional-components=ANACONDA,JUPYTER --bucket wes_development --project cncd-244400 --region us-central1 --num-preemptible-workers 50 --requester-pays-allow-buckets hail-bucket,big-data

Error:

failed: Initialization action failed. Failed action ‘gs://hail-common/hailctl/dataproc/0.2.46/init_notebook.py’, see output in: gs://wes_development/google-cloud-dataproc-metainfo/3427c8dd-67a7-466f-bbb9-38f22adeff25/hailjupy-m/dataproc-initialization-script-0_output.

In the output, i see the following last lines

Traceback (most recent call last):
File “/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0”, line 64, in
safe_call(*command)
File “/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0”, line 18, in safe_call
print(e.output).decode()
AttributeError: ‘NoneType’ object has no attribute ‘decode’

2

Hail clusters are mostly incompatible with the Dataproc --optional-components flag. What do you plan to use Anaconda for? A Hail cluster always has Jupyter installed, so you don’t need to specify that. Try removing that flag.

hailctl dataproc --beta start hailjupy --vep GRCh37 --bucket wes_development --project cncd-244400 --region us-central1 --num-preemptible-workers 50 --requester-pays-allow-buckets hail-bucket,big-data
1 Like
3

Trying that

4

Did this work, @Haseeb1?

5

yes it did