Hailctl autoscaling issue

Hail Query & hailctl
1

This seems never happen before,
first the error shows auto scale was not imported, then I imported the following file,
hailautoscale.yaml

1 workerConfig:
2 minInstances: 2
3 maxInstances: 50
4 secondaryWorkerConfig:
5 maxInstances: 50
6 basicAlgorithm:
7 cooldownPeriod: 4m
8 yarnConfig:
9 scaleUpFactor: 0.05
10 scaleDownFactor: 1.0
11 gracefulDecommissionTimeout: 1h
~

Then start the job like,

hailctl dataproc start ukbbwesqc --autoscaling-policy=hailautoscale
no errors thrown,
but when I run the following, there are errors
hailctl dataproc submit ukbbwesqc UKBB_WES_import_split_multi.py

ERROR: (gcloud.dataproc.jobs.submit.pyspark) Job [0fd863203b89439db21080cd78e97480] failed with error:

Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at:

https://console.cloud.google.com/dataproc/jobs/0fd863203b89439db21080cd78e97480?project=qualitycontrol-330617&region=us-east1

gcloud dataproc jobs wait ‘0fd863203b89439db21080cd78e97480’ --region ‘us-east1’ --project ‘qualitycontrol-330617’

https://console.cloud.google.com/storage/browser/dataproc-staging-us-east1-467882637532-17e3ovf7/google-cloud-dataproc-metainfo/21ecc286-3ee9-4a35-81e9-2e4cc7eb025c/jobs/0fd863203b89439db21080cd78e97480/

gs://dataproc-staging-us-east1-467882637532-17e3ovf7/google-cloud-dataproc-metainfo/21ecc286-3ee9-4a35-81e9-2e4cc7eb025c/jobs/0fd863203b89439db21080cd78e97480/driveroutput