# Error: "Bucket is a requester pays bucket but no user project provided." **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [October 6, 2022, 4:21pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887 "2022-10-06T16:21:25Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 4:21pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/1 "2022-10-06T16:21:25Z")
Hi! I am trying to annotate a matrix with CADD scores. db = hl.experimental.DB(region=‘us’, cloud=‘gcp’) mt = db.annotate\_rows\_db(mt, ‘CADD’) Tried to manually specify as this post suggests ([I'm encountering "Bucket is a requester pays bucket but no user project provided."](https://discuss.hail.is/t/im-encountering-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2536)), but the error persists. import hail as hl hl.init(spark\_conf={ ‘spark.hadoop.fs.gs.requester.pays.mode’: ‘CUSTOM’, ‘spark.hadoop.fs.gs.requester.pays.buckets’: ‘hail-datasets-us’, ‘spark.hadoop.fs.gs.requester.pays.project.id’: ‘caddprojectnew’ }) Any help would be greatly appreciated! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 6, 2022, 4:37pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/2 "2022-10-06T16:37:16Z")
Hey @laf , What version of Hail is installed? Are you using Google Dataproc? ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 4:49pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/4 "2022-10-06T16:49:25Z")
Yes using dataproc and Hail version 0.2.99-57537fea08d4, thank you! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 6, 2022, 4:54pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/5 "2022-10-06T16:54:14Z")
When starting a dataproc cluster, you have to tell `hailctl` to use enable requester pays: ```python --requester-pays-allow-all Allow reading from all requester-pays buckets. --requester-pays-allow-buckets REQUESTER_PAYS_ALLOW_BUCKETS Comma-separated list of requester-pays buckets to allow reading from. --requester-pays-allow-annotation-db Allows reading from any of the requester-pays buckets that hold data for the annotation database. ``` For dataproc clusters, you can’t do this with `hl.init` because Spark is already running on the cluster. `hl.init` only works when running locally or when initializing a Spark cluster from scratch (not using Dataproc, EMR, or HDInsight). ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 4:58pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/6 "2022-10-06T16:58:12Z")
@danking Ok thank you, I do think I did this when I originally started the cluster like this hailctl dataproc start clutstercadd --requester-pays-allow-annotation-db --region=‘us-east1’ ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 6, 2022, 6:51pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/7 "2022-10-06T18:51:04Z")
Can you share the full stack trace and error message you’re getting? If you have the hail log file that will help as well. ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 7:07pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/8 "2022-10-06T19:07:23Z")
Here it is, thank you so much for the help! FatalError Traceback (most recent call last) in ----\> 1 y2 = db.annotate\_rows\_db(inty, ‘CADD’) in annotate\_rows\_db(self, rel, \*names) ~/lanefitz/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) 575 def wrapper(_original\_func, \*args, \*\*kwargs): 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) → 577 return _original\_func(\*args, \*\*kwargs_) 578 579 return wrapper ~/lanefitz/lib/python3.7/site-packages/hail/experimental/db.py in annotate\_rows\_db(self, rel, \*names) 534 rel = rel.annotate(\*\*{dataset.name: genes.index(rel.key)[dataset.name]}) 535 else: → 536 indexed\_value = dataset.index\_compatible\_version(rel.key) 537 if isinstance(indexed\_value.dtype, hl.tstruct) and len(indexed\_value.dtype) == 0: 538 indexed\_value = hl.is\_defined(indexed\_value) ~/lanefitz/lib/python3.7/site-packages/hail/experimental/db.py in index\_compatible\_version(self, key\_expr) 266 compatible\_indexed\_values = [ 267 (version.maybe\_index(key\_expr, all\_matches), version.version) → 268 for version in self.versions 269 if version.maybe\_index(key\_expr, all\_matches) is not None 270 ] ~/lanefitz/lib/python3.7/site-packages/hail/experimental/db.py in (.0) 267 (version.maybe\_index(key\_expr, all\_matches), version.version) 268 for version in self.versions → 269 if version.maybe\_index(key\_expr, all\_matches) is not None 270 ] 271 if len(compatible\_indexed\_values) == 0: ~/lanefitz/lib/python3.7/site-packages/hail/experimental/db.py in maybe\_index(self, indexer\_key\_expr, all\_matches) 153 Struct of compatible indexed values, if they exist. 154 “”" → 155 return hl.read\_table(self.url).\_maybe\_flexindex\_table\_by\_expr( 156 indexer\_key\_expr, all\_matches=all\_matches) 157 in read\_table(path, \_intervals, \_filter\_intervals, \_n\_partitions, \_assert\_type, \_load\_refs) ~/lanefitz/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) 575 def wrapper(_original\_func, \*args, \*\*kwargs): 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) → 577 return _original\_func(\*args, \*\*kwargs_) 578 579 return wrapper ~/lanefitz/lib/python3.7/site-packages/hail/methods/impex.py in read\_table(path, \_intervals, \_filter\_intervals, \_n\_partitions, \_assert\_type, \_load\_refs) 2923 “”" 2924 if \_load\_refs: → 2925 for rg\_config in Env.backend().load\_references\_from\_dataset(path): 2926 hl.ReferenceGenome.\_from\_config(rg\_config) 2927 ~/lanefitz/lib/python3.7/site-packages/hail/backend/spark\_backend.py in load\_references\_from\_dataset(self, path) 326 327 def load\_references\_from\_dataset(self, path): → 328 return json.loads(self.hail\_package().variant.ReferenceGenome.fromHailDataset(self.fs.\_jfs, path)) 329 330 def from\_fasta\_file(self, name, fasta\_file, index\_file, x\_contigs, y\_contigs, mt\_contigs, par): ~/lanefitz/lib/python3.7/site-packages/py4j/java\_gateway.py in **call** (self, \*args) 1303 answer = self.gateway\_client.send\_command(command) 1304 return\_value = get\_return\_value( → 1305 answer, self.gateway\_client, self.target\_id, self.name) 1306 1307 for temp\_arg in temp\_args: ~/lanefitz/lib/python3.7/site-packages/hail/backend/py4j\_backend.py in deco(\*args, \*\*kwargs) 29 tpl = Env.jutils().handleForPython(e.java\_exception) 30 deepest, full, error\_id = tpl.\_1(), tpl.\_2(), tpl.\_3() —\> 31 raise fatal\_error\_from\_java\_error\_triplet(deepest, full, error\_id) from None 32 except pyspark.sql.utils.CapturedException as e: 33 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’ FatalError: GoogleJsonResponseException: 400 Bad Request GET [https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata](https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata) { “code” : 400, “errors” : [ { “domain” : “global”, “message” : “Bucket is a requester pays bucket but no user project provided.”, “reason” : “required” } ], “message” : “Bucket is a requester pays bucket but no user project provided.” } Java stack trace: java.io.IOException: Error accessing gs://hail-datasets-us/CADD/v1.4/GRCh37/table.ht at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:2221) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:2108) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfoInternal(GoogleCloudStorageFileSystem.java:1091) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfo(GoogleCloudStorageFileSystem.java:1065) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:955) at is.hail.io.fs.HadoopFS.fileStatus(HadoopFS.scala:166) at is.hail.io.fs.FS.isDir(FS.scala:364) at is.hail.io.fs.FS.isDir$(FS.scala:362) at is.hail.io.fs.HadoopFS.isDir(HadoopFS.scala:72) at is.hail.expr.ir.RelationalSpec$.readMetadata(AbstractMatrixTableSpec.scala:31) at is.hail.expr.ir.RelationalSpec$.readReferences(AbstractMatrixTableSpec.scala:74) at is.hail.variant.ReferenceGenome$.fromHailDataset(ReferenceGenome.scala:581) at is.hail.variant.ReferenceGenome.fromHailDataset(ReferenceGenome.scala) at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:834) com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request GET [https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata](https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metageneration,size,contentType,contentEncoding,md5Hash,crc32c,metadata) { “code” : 400, “errors” : [ { “domain” : “global”, “message” : “Bucket is a requester pays bucket but no user project provided.”, “reason” : “required” } ], “message” : “Bucket is a requester pays bucket but no user project provided.” } at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455) at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getObject(GoogleCloudStorageImpl.java:2215) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfo(GoogleCloudStorageImpl.java:2108) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfoInternal(GoogleCloudStorageFileSyst em.java:1091) at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfo(GoogleCloudStorageFileSystem.java: 1065) at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getFileStatus(GoogleHadoopFileSystemBase.java:955) at is.hail.io.fs.HadoopFS.fileStatus(HadoopFS.scala:166) at is.hail.io.fs.FS.isDir(FS.scala:364) at is.hail.io.fs.FS.isDir$(FS.scala:362) at is.hail.io.fs.HadoopFS.isDir(HadoopFS.scala:72) at is.hail.expr.ir.RelationalSpec$.readMetadata(AbstractMatrixTableSpec.scala:31) at is.hail.expr.ir.RelationalSpec$.readReferences(AbstractMatrixTableSpec.scala:74) at is.hail.variant.ReferenceGenome$.fromHailDataset(ReferenceGenome.scala:581) at is.hail.variant.ReferenceGenome.fromHailDataset(ReferenceGenome.scala) at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:834) Hail version: 0.2.99-57537fea08d4 Error summary: GoogleJsonResponseException: 400 Bad Request GET [https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metage](https://storage.googleapis.com/storage/v1/b/hail-datasets-us/o/CADD%2Fv1.4%2FGRCh37%2Ftable.ht?fields=bucket,name,timeCreated,updated,generation,metage) neration,size,contentType,contentEncoding,md5Hash,crc32c,metadata { “code” : 400, “errors” : [ { “domain” : “global”, “message” : “Bucket is a requester pays bucket but no user project provided.”, “reason” : “required” } ], “message” : “Bucket is a requester pays bucket but no user project provided.” } [https://o2portal.rc.hms.harvard.edu/node/compute-e-16-233.o2.rc.hms.harvard.edu/17606/edit/hail-20221006-1242-0.2.99-57537fea08d4.log](https://o2portal.rc.hms.harvard.edu/node/compute-e-16-233.o2.rc.hms.harvard.edu/17606/edit/hail-20221006-1242-0.2.99-57537fea08d4.log) ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 6, 2022, 7:38pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/9 "2022-10-06T19:38:28Z")
OK, and just to be totally clear, you started the cluster like this: ```python hailctl dataproc start clutstercadd --requester-pays-allow-annotation-db --region='us-east1' ``` And the Python code you’re executing is this: ```python import hail as hl hl.init(spark_conf={ 'spark.hadoop.fs.gs.requester.pays.mode': 'CUSTOM', 'spark.hadoop.fs.gs.requester.pays.buckets': 'hail-datasets-us', 'spark.hadoop.fs.gs.requester.pays.project.id': 'caddprojectnew' }) db = hl.experimental.DB(region=‘us’, cloud=‘gcp’) mt = db.annotate_rows_db(mt, ‘CADD’) ``` Have you also tried without the `spark_conf` argument? Let’s confirm that your spark configuration is correct. Can you navigate to the spark UI page? ```python hailctl dataproc connect clustercadd spark-ui ``` Then click “environment”. I believe in here it should tell you the configuration of things like “hadoop.fs.gs.requester.pays.buckets”. Can you confirm that those are set to the values you provided? ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 8:19pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/10 "2022-10-06T20:19:38Z")
Yes that all looks correct. Have tried with and without the spark\_conf. Ran that spark-ui command, but it needs chromium so I’m trying to get it now. Sorry I’m new to this! ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 6, 2022, 11:53pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/11 "2022-10-06T23:53:09Z")
I’m on terminal so having a hard time connecting browser. If there’s another way to check this I’ll try that too! ---
**Author:** ![laf](https://avatars.discourse-cdn.com/v4/letter/l/13edae/32.png) [@laf](https://discuss.hail.is/u/laf)\ **Post date:** [October 17, 2022, 4:55pm UTC](https://discuss.hail.is/t/error-bucket-is-a-requester-pays-bucket-but-no-user-project-provided/2887/12 "2022-10-17T16:55:50Z")
![Screen Shot 2022-10-17 at 12.53.50 PM](https://canada1.discourse-cdn.com/flex036/uploads/hail/original/1X/97fe3da8994fc439e315b3386cbc4aa85186cc80.png) Hi! This is what I get when I navigate to the spark UI page by running the command you sent, please let me know if that gives you any insight as to what I am doing wrong – thank you!