How to pass gcloud parameters through hailctl?

Hail Query & hailctl
1

Hi,

I’m trying to run the following command

hailctl dataproc submit \
  --files=${my_files} \
  ${cluster_name} \
  ${script.py} \
  ${jobarg1} \
  ...

which results in this gcloud command being run:

gcloud dataproc jobs submit pyspark ${my_script.py} \
    --files=${my_files} \
    --py-files=/tmp/pyscripts_3bv_9tw7.zip \
    --properties= \
    -- \
    ${jobarg1}

My issue is that I also want to run gcloud dataproc submit with the --driver-log-levels root=WARN parameter to get something like this:

gcloud dataproc jobs submit pyspark ${my_script.py} \
    --files=${my_files} \
    --driver-log-levels root=WARN \
    --py-files=/tmp/pyscripts_3bv_9tw7.zip \
    --properties= \
    -- \
    ${jobarg1}

I cannot figure out the hailctl dataproc submit command which would successfully pass the --driver-log-levels root=WARN paramter to gcloud dataproc

2

Theoretically, I’m pretty sure you’re just supposed to be able to pass extra arguments to submit and they get passed along to gcloud. Try just adding --driver-log-levels root=WARN at the end of your submit command and see if that generates the gcloud command you want. Note that you can add --dry-run argument to hailctl submit in order to just have it print the gcloud command without executing it.