GCP jupyter error

Hail Batch & General Cloud
1

Hello

I’ve updated to the latest version of hail, by running pip uninstall followed by pip-install hail.

hail version: 0.2.108-fc03e9d5dc08

I create a cluster using the following command:

hailctl dataproc start my-hail-cluster --optional-components=JUPYTER --enable-component-gateway --master-machine-type n1-standard-8 --worker-machine-type n1-standard-4 --bucket my_bucket --project myproject --region us-central1 --requester-pays-allow-all --properties spark:spark.driver.maxResultSize=8g,spark:spark.executor.memory=4g

When I open jupyter link: I get the following error:
Server error: Internal server error: module ‘google.auth.credentials’ has no attribute ‘CredentialsWithTokenUri’

I have used hail like this multiple times so I’m not sure why this has come up. The bucket and cluster are in the same region. If I initiate a non-hail cluster this issue does not occur.

Any help on this would be highly appreciated.

2

Hello again
Any help with this would be appreciated.

Thank you

3

Don’t use --optional-components=JUPYTER. You probably also do not need the --enable-component-gateway.

hailctl automatically configures a version of Jupyter which works properly with Hail. You’re adding a different version of Jupyter which is, apparently, incompatible.

4

Thanks for the clarification.

How can I then connect to jupyter? I’ve tried running external_ip:8132 and internal_ip:8132. But it doesn’t seem to connect.

5

Take a look at these docs.

6

Thank you

I ran the following command on the master node of my cluster:
hailctl dataproc connect my-cluster notebook --zone us-central1-c

And I get the following error:

ERROR: (gcloud.compute.ssh) Could not fetch resource:

  • Request had insufficient authentication scopes.

I tried authenticating using google auth login however, the authentication doesn’t go through.

Apologies for the continuous headache, your support is greatly appreciated!

7

Your user account lacks sufficient privilege to SSH to VMs which is how hailctl connects you to your notebook. Take a look at the suggestion in this stack overflow answer.