Open notebooks causes 500 Internal Server Error in Dataproc

Hi Hail team,

I am writing to get advice on Dataproc issue. I run Hail on Dataproc on GCP.
Today, I got “500 internal server error”, which is that I have never seen before until yesterday when I try to open a Jupyter notebook. If you have any idea to solve it, please share with me. Thank you.

Here is my code to launch Dataproc and open the jupyter notebook

hailctl dataproc start ${cluster_name} --vep GRCh38 \
        --requester-pays-allow-annotation-db \
        --packages gnomad --requester-pays-allow-buckets gnomad-public-requester-pays \
        --master-machine-type=n1-highmem-8 --worker-machine-type=n1-highmem-8 \
        --num-workers=2 --num-secondary-workers=0 \
hailctl dataproc connect ${cluster_name} notebook


Can you replicate this by creating a fresh cluster and trying to connect, or is it a particular cluster that has this problem?

Hi Tim.

Thank you for your supports. The issue has been resolved by updating Hail from v0.2.62 to 0.2.78. :grinning:


0.2.79 also made the hail error messages from dataproc better, no reason not to upgrade one more :+1: