Error when running count after filtering MT

Sorry…I cannot for the life of me figure out how to attach a file on here. Google has failed me. How should I send it?

This is prehaps non-intuitive or just aggressively intuitive, but you should be able to drag and drop? I just fixed the file extensions to permit .log files.

foo.log (9 Bytes)

Yup now I can. Before it would only accept image extensions. Thank you.
gnomad_clinvar.log (403.8 KB)


(TableRead Table{global:Struct{},key:[],row:Struct{s:String}} False “{“name”:“TableNativeReader”,“path”:“gs://gnomad/hardcalls/hail-0.2/mt/exomes/”,”_spec":{“name”:“TableSpec”,“file_version”:65536,“hail_version”:“devel-a23032101373”,“references_rel_path”:"…/references",“table_type”:“Table{global:Struct{},key:[s],row:Struct{s:String}}”,“components”:{“globals”:{“name”:“RVDComponentSpec”,“rel_path”:"…/globals/rows"},“rows”:{“name”:“RVDComponentSpec”,“rel_path”:“rows”},“partition_counts”:{“name”:“PartitionCountsComponentSpec”,“counts”:[164332]}}}}"))))))

This is a super old file. I bet it has the required-globals problem.

I’m fairly confident I back-patched (read: manually edited the metadata.json.gz) all the gnomAD files, so unless there’s another issue, I would think it should work.

Maybe try a different file (the exomes, or non-split hardcalls) to double check, maybe I missed one, but this one’s a pretty big workhorse, so I’d be surprised. mt = mt.select_globals() can also work to check if that’s the issue.

I fixed the error message here to give us more information. Can you try running on latest master?

(if you need help building from source, let me know)

mt = get_gnomad_data('exomes', adj=True, release_annotations=True, release_samples=True, split=True)
mt = mt.select_globals()

Fails with the same error.

I attempted to build from source but received this error when running ./

ERROR: (gcloud.iam.service-accounts.keys.create) RESOURCE_EXHAUSTED: Maximum number of keys on account reached.
  retryDelay: 86401s

I haven’t done this before so it is likely it was done incorrectly. However, when I initialize hail locally in the hail/hail directory I am running version 0.2.24-e3e63a2f9856. I think it’s just I can’t hook up to gcs?

This code works for me with my home-spun 9c44fc9e7c2b (or maybe 22f6defd17d who knows with my rig anymore, anyway relatively recent) so I don’t think it’s the old requiredness problem.

@mwilson I mean try on the cloud with a custom build – to do this you do (from hail/hail):

HAILCTL_BUCKET_BASE="gs://a-bucket-you-can-write-to" make install-hailctl

With or without mt = mt.select_globals(), I now see this error when running the code above. Would the full log be helpful?

FatalError: RuntimeException: globals mismatch:
typ: Struct{}
val: +Struct{}

    Java stack trace:
java.lang.RuntimeException: globals mismatch:
  typ: Struct{}
  val: +Struct{}
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableLike$
	at scala.collection.mutable.ArrayOps$
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:24)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:57)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:57)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.backend.Backend.execute(Backend.scala:57)
	at is.hail.backend.Backend.executeJSON(Backend.scala:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(
	at java.lang.reflect.Method.invoke(
	at py4j.reflection.MethodInvoker.invoke(
	at py4j.reflection.ReflectionEngine.invoke(
	at py4j.Gateway.invoke(
	at py4j.commands.AbstractCommand.invokeMethod(
	at py4j.commands.CallCommand.execute(

Hail version: 0.2.24-e3e63a2f9856
Error summary: RuntimeException: globals mismatch:
  typ: Struct{}
  val: +Struct{}

@konradjk you didn’t get all the +s :frowning:

I think there must be some buried in other metadata.json.gz that weren’t used then, but are now.

ah that makes sense. i think i only had to do the overall metadata.json.gz one before, but maybe now we need to do more. sigh.

I can fix this in Hail though

oh that’d be great. the previous fix was pretty nervewracking

The code above now runs, i.e. filtering the columns only, but when I run

mt = mt.filter_rows(hl.agg.any(mt.GT.is_non_ref()))

I am seeing the same global mismatch error.

the log file might help here, then

hail-20191008-2047-0.2.24-aca727e8d9c9.log (1.1 MB)

oops… this should fix…