Best practices for setting up GCP billing for Hail for the first time

Hail Batch & General Cloud
1

Can you please clarify what is best practice for a new user using Hail with GCP for the first time, how to get billing set up so that we have the ability to use dataproc and requester pays buckets?

I am a new Broadie and my group is using Hail for the first time. We just set up a GCP billing account MYBILLING and created a project MYPROJECTNAME. I have confirmed that I am logged in with gcloud auth list and that I have the following permissions with gcloud projects get-iam-policy MYPROJECTNAME:

roles/dataproc.admin
roles/dataproc.worker
roles/storage.admin

And yet when I try to run Hail, for instance with hailctl dataproc start cluster-1 --packages gnomad, I get this error:

ERROR: (gcloud.projects.get-iam-policy) User [XXX] does not have permission to access projects instance [XXX:getIamPolicy] (or it may not exist): The caller does not have permission

I have already consulted this: Using Hail on the Google Cloud Platform but it says it is “old and no longer works”. Meanwhiel the current “hail on the cloud” documentation: Hail | Hail on the Cloud does not specify how to grant the necessary permissions so that one can run the commands indicated.

1 Like
2

Hi @Yuan_Lian!
I’m sorry to hear you’re having difficulty using hail with dataproc. While I can’t answer your general question about best practices, I can try to help unblock you. My colleague may be able to help with your general question when he returns.

The following documentation pages may be of help:

Regarding your issue with permissions, can you confirm if the redacted user “XXX” is your user or the service account associated with your user? You might need to configure your project’s IAM policies to allow access to that user/service account. See the link below for a reference of google’s predefined roles that you can use
https://cloud.google.com/iam/docs/understanding-roles#predefined

Unfortunately I don’t have a list of every permission required so you might need to try a couple of times until you get all of them. Try to keep permissions granted to service accounts to a minimum.

I hope this helps. Let me know if you need futher assistance.
Ed