# Gcloud job failure **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [February 1, 2023, 10:10pm UTC](https://discuss.hail.is/t/gcloud-job-failure/3099 "2023-02-01T22:10:37Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![arm](https://avatars.discourse-cdn.com/v4/letter/a/ce73a5/32.png) [@arm](https://discuss.hail.is/u/arm)\ **Post date:** [February 1, 2023, 10:10pm UTC](https://discuss.hail.is/t/gcloud-job-failure/3099/1 "2023-02-01T22:10:37Z")
Hi Hail Team! I recently ran the following script using hailctl dataproc. It took 19 hours to run and all of the steps seemed to be completed, but at the end I received an error message that was not very informative and nothing was actually written to the file. Is there something wrong with the code? Are there other potential sources of error? ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [d7070dc3db81471a8be2582d82ecc92e] failed with error: Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at: > **[Google Cloud Platform](https://accounts.google.com/v3/signin/identifier?continue=https%3A%2F%2Fconsole.cloud.google.com%2Fdataproc%2Fjobs%2Fd7070dc3db81471a8be2582d82ecc92e%3Fproject%3Dbrave-cubist-363320%26region%3Dus-east1&dsh=S-1171140304%3A1764196042711061&followup=https%3A%2F%2Fconsole.cloud.google.com%2Fdataproc%2Fjobs%2Fd7070dc3db81471a8be2582d82ecc92e%3Fproject%3Dbrave-cubist-363320%26region%3Dus-east1&ifkv=ARESoU1dN30mrU69BYpOstCOwgnwtn7-z0A02f5Fiw7RlL5wU4lTA1-gqiZ6mDtDUrLTrmnNyN78iA&osid=1&passive=1209600&service=cloudconsole&flowName=GlifWebSignIn&flowEntry=ServiceLogin)** > > Google Cloud Platform lets you build, deploy, and scale applications, websites, and services on the same infrastructure as Google. gcloud dataproc jobs wait ‘d7070dc3db81471a8be2582d82ecc92e’ --region ‘us-east1’ --project ‘brave-cubist-363320’ > **[Google Cloud Platform](https://accounts.google.com/v3/signin/identifier?continue=https%3A%2F%2Fconsole.cloud.google.com%2Fstorage%2Fbrowser%2Fdataproc-staging-us-east1-942231253036-bw4veo0a%2Fgoogle-cloud-dataproc-metainfo%2F9e941d43-e4d7-43c5-a07a-3084830814df%2Fjobs%2Fd7070dc3db81471a8be2582d82ecc92e%2F&dsh=S286974933%3A1764196043967659&followup=https%3A%2F%2Fconsole.cloud.google.com%2Fstorage%2Fbrowser%2Fdataproc-staging-us-east1-942231253036-bw4veo0a%2Fgoogle-cloud-dataproc-metainfo%2F9e941d43-e4d7-43c5-a07a-3084830814df%2Fjobs%2Fd7070dc3db81471a8be2582d82ecc92e%2F&ifkv=ARESoU2zbpHTudegatZShCLwQ0OUOyqFxSAlG6z8GKny-pydLjiQGfBGbD8JDP-curNY1aUhysYo&osid=1&passive=1209600&service=cloudconsole&flowName=GlifWebSignIn&flowEntry=ServiceLogin)** > > Google Cloud Platform lets you build, deploy, and scale applications, websites, and services on the same infrastructure as Google. gs://dataproc-staging-us-east1-942231253036-bw4veo0a/google-cloud-dataproc-metainfo/9e941d43-e4d7-43c5-a07a-3084830814df/jobs/d7070dc3db81471a8be2582d82ecc92e/driveroutput Traceback (most recent call last): File “/Users/xyz123/Library/Python/3.7/bin/hailctl”, line 8, in sys.exit(main()) File “/Users/xyz123/Library/Python/3.7/lib/python/site-packages/hailtop/hailctl/ **main**.py”, line 107, in main cli.main(args) File “/Users/xyz123/Library/Python/3.7/lib/python/site-packages/hailtop/hailctl/dataproc/cli.py”, line 124, in main jmp[args.module].main(args, pass\_through\_args)) File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/base\_events.py”, line 584, in run\_until\_complete return future.result() File “/Users/xyz123/Library/Python/3.7/lib/python/site-packages/hailtop/hailctl/dataproc/submit.py”, line 88, in main gcloud.run(cmd) File “/Users/xyz123/Library/Python/3.7/lib/python/site-packages/hailtop/hailctl/dataproc/gcloud.py”, line 9, in run return subprocess.check\_call([“gcloud”] + command) File “/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py”, line 347, in check\_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command ‘[‘gcloud’, ‘dataproc’, ‘jobs’, ‘submit’, ‘pyspark’, ‘/Users/xyz123/Desktop/AUTUMN/AMP-PD\_Code/densify\_filter.py’, ‘–cluster=cluster1’, ‘–files=’, ‘–py-files=/var/folders/qg/q8g70wfd4493bfmkmjgcb3qh0000gn/T/pyscripts\_xi5m2d68.zip’, ‘–properties=’, ‘–region=us-east1’, ‘–’, ‘-i’, ‘gs://filepath/v1-2-amp-pd-mgrb.sparse.mt/’, ‘-o’, ‘gs://file\_path/test\_start.mt/’]’ returned non-zero exit status 1. import hail as hl import argparse # Arguements parser = argparse.ArgumentParser() parser.add\_argument(“-f”, “–full\_run”, action=“store\_true”, help=“Runs on chr22 and chrX only by default. If full\_run is set, it runs on the whole matrix. WARNING: This will be VERY expensive”) parser.add\_argument(“-w”, “–overwrite”, action=‘store\_true’, help=“If set will overwrite output matrix if it already exists”) requiredNamed = parser.add\_argument\_group(‘required named arguments’) requiredNamed.add\_argument(“-i”, “–input\_mt\_path”, required=True) requiredNamed.add\_argument(“-o”, “–output\_mt\_path”, required=True) #requiredNamed.add\_argument(“-p”, “–requester\_pays\_project\_id”, help=“Project ID to bill to when accessing requester pays bucket, needed to access hail annotationDB”) args = parser.parse\_args() # Store Inputs input\_mt\_path = args.input\_mt\_path output\_mt\_path = args.output\_mt\_path #requester\_pays\_project\_id = args.requester\_pays\_project\_id # read mt mt = hl.read\_matrix\_table(input\_mt\_path) mt = hl.experimental.densify(mt) # Save mt densified and filtered to CHR22/PPMI mt.write(output\_mt\_path, overwrite=True)