Billing project not working and trouble with output files

Hail Batch & General Cloud
1

Hi,

I am trying to learn how to use hail batch and I am trying to configure my billing project and bucket to the hail batch system. In my terminal, I used the “hailctl config set batch/billing_project” and “hailctl config set batch/bucket” and it give no error. Then when I try run something w/ batch on my notebook, it says that billing project doesn’t exist. The billing project that you give as part of the hail batch tutorial works for me, and I made sure that the billing project name is correct on my google console page.

Also, I am having trouble getting the output files to get into my bucket. I was using this set of the commands from the tutorial, adding my bucket to the output path. Nothing shows up in my bucket when I run this, though it says “Batch completed successfully!”

b = hb.Batch(name=‘hello-input’)
j = b.new_job(name=‘hello’)
j.command(f’echo “hello” > {j.ofile}')
b.write_output(j.ofile, ‘gs://my_bucket/output/hello.txt’)
b.run()

2

Do you see anything on the Batch UI (https://batch.hail.is) for this batch?

This looks like you’re doing the right thing; I’ll ping someone on the services team to take a look.

3

I have one idea about the billing project though – the Batch billing project and the Google Cloud billing project are totally separate.

I think you have a Batch billing project, so make sure you’re using that one instead of the Google project ID.

4

Jackie thinks the second issue (no file created) is fixed by this open pull request:

https://github.com/hail-is/hail/pull/10716

So hold tight until Monday or so and it should be resolved.

1 Like
5

Oh, yes I was using the Google Cloud billing project ID. I didn’t realize I had a Batch billing project. Where can I find that?

6

I’m looking at your message more carefully and I think part of the problem is you’re actually running your batches using the LocalBackend (default). This is where the bug that should get fixed early next week occurs.

To use the ServiceBackend, you’ll want to find your billing project name which you can get by clicking on your username in the top right hand corner of the header menu. As Tim mentioned, this billing project name is not the same as your google project. Then use hailctl config set batch/billing_project <BILLING_PROJECT_NAME> with the trial billing project name. Once you have set the billing project name, you can do the following to submit batches to the service:

import hailtop.batch as hb

backend = hb.ServiceBackend()
b = hb.Batch(backend=backend)

The simple hello world example should work now. However, if you want to write files to your google bucket, then make sure you follow the directions for setting up your new Batch service account.

Please let us know if you have additional questions.