# Error when running count after filtering MT

**URL:** <https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114>\
**Category:** Hail Query & hailctl\
**Created:** [October 7, 2019, 6:05pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114 "2019-10-07T18:05:20Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 7, 2019, 6:05pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/1 "2019-10-07T18:05:21Z")

</div>

Hello Hail team,

I am running into an error when running count after filtering on either columns or rows.

```
mt = get_gnomad_data('genomes', adj=True, release_annotations=True, split=True)
mt = hl.filter_intervals(mt, hl.experimental.get_gene_intervals(gene_symbols=['AP4B1','AP4E1', 'AP4M1', 'AP4S1']))
mt.rows().count()

```

will throw the error

```
FatalError: IllegalArgumentException: requirement failed

Java stack trace:
java.lang.IllegalArgumentException: requirement failed
	at scala.Predef$.require(Predef.scala:212)
	at is.hail.expr.ir.TableValue.<init>(TableValue.scala:47)
	at is.hail.expr.ir.TableNativeZippedReader.apply(TableIR.scala:245)
	at is.hail.expr.ir.TableRead.execute(TableIR.scala:295)
	at is.hail.expr.ir.TableFilterIntervals.execute(TableIR.scala:1714)
	at is.hail.expr.ir.Interpret$$anonfun$apply$2.apply$mcJ$sp(Interpret.scala:730)
	at is.hail.expr.ir.Interpret$$anonfun$apply$2.apply(Interpret.scala:730)
	at is.hail.expr.ir.Interpret$$anonfun$apply$2.apply(Interpret.scala:730)
	at scala.Option.getOrElse(Option.scala:121)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:730)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:89)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:59)
	at is.hail.expr.ir.InterpretNonCompilable$$anonfun$7.apply(InterpretNonCompilable.scala:19)
	at is.hail.expr.ir.InterpretNonCompilable$$anonfun$7.apply(InterpretNonCompilable.scala:19)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
	at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:19)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$2.apply(CompileAndEvaluate.scala:37)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$2.apply(CompileAndEvaluate.scala:37)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:24)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:37)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:55)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:55)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:8)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:7)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:7)
	at is.hail.backend.Backend.execute(Backend.scala:55)
	at is.hail.backend.Backend.executeJSON(Backend.scala:61)
	at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.24-9cd88d97bedd
Error summary: IllegalArgumentException: requirement failed

```

mt.count() will run if I do not filter to the gene intervals. However, if I filter to gnomad release samples using mt = mt.filter\_cols(mt.meta.release). I see the same error. I am using version 0.2.24-9cd88d97bedd .

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 7, 2019, 6:08pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/2 "2019-10-07T18:08:02Z")

</div>

can we have the log file?

---

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 7, 2019, 6:21pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/3 "2019-10-07T18:21:55Z")

</div>

Sorry…I cannot for the life of me figure out how to attach a file on here. Google has failed me. How should I send it?

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [October 7, 2019, 6:25pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/4 "2019-10-07T18:25:25Z")

</div>

This is prehaps non-intuitive or just aggressively intuitive, but you should be able to drag and drop? I just fixed the file extensions to permit .log files.

[foo.log](https://discuss.hail.is/uploads/short-url/nrirPNmENYtslFxFogvwDYYD7sL.log) (9 Bytes)

---

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 7, 2019, 6:26pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/5 "2019-10-07T18:26:07Z")

</div>

Yup now I can. Before it would only accept image extensions. Thank you.  
[gnomad\_clinvar.log](https://discuss.hail.is/uploads/short-url/2RuJ2b4syZmd03L9o79Tygczv7p.log) (403.8 KB)

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 7, 2019, 6:40pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/6 "2019-10-07T18:40:08Z")

</div>

aha:

> (TableRead Table{global:Struct{},key:,row:Struct{s:String}} False “{"name":"TableNativeReader","path":"gs://gnomad/hardcalls/hail-0.2/mt/exomes/gnomad.exomes.mt/cols","\_spec":{"name":"TableSpec","file\_version":65536,"hail\_version":"devel-a23032101373","references\_rel\_path":"…/references","table\_type":"Table{global:Struct{},key:[s],row:Struct{s:String}}","components":{"globals":{"name":"RVDComponentSpec","rel\_path":"…/globals/rows"},"rows":{"name":"RVDComponentSpec","rel\_path":"rows"},"partition\_counts":{"name":"PartitionCountsComponentSpec","counts":[164332]}}}}”))))))

This is a super old file. I bet it has the required-globals problem.

---

<div class="post-metadata">

**Author:** ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)\
**Post date:** [October 7, 2019, 10:53pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/7 "2019-10-07T22:53:03Z")

</div>

I’m fairly confident I back-patched (read: manually edited the metadata.json.gz) all the gnomAD files, so unless there’s another issue, I would think it should work.

---

<div class="post-metadata">

**Author:** ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)\
**Post date:** [October 7, 2019, 10:54pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/8 "2019-10-07T22:54:24Z")

</div>

Maybe try a different file (the exomes, or non-split hardcalls) to double check, maybe I missed one, but this one’s a pretty big workhorse, so I’d be surprised. `mt = mt.select_globals()` can also work to check if that’s the issue.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 1:10pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/9 "2019-10-08T13:10:56Z")

</div>

I fixed the error message here to give us more information. Can you try running on latest master?

(if you need help building from source, let me know)

---

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 8, 2019, 3:23pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/10 "2019-10-08T15:23:22Z")

</div>

```
mt = get_gnomad_data('exomes', adj=True, release_annotations=True, release_samples=True, split=True)
mt = mt.select_globals()
mt.count() 

```

Fails with the same error.

I attempted to build from source but received this error when running ./install-gcs-connector.sh

```
ERROR: (gcloud.iam.service-accounts.keys.create) RESOURCE_EXHAUSTED: Maximum number of keys on account reached.
'@type': type.googleapis.com/google.rpc.RetryInfo
  retryDelay: 86401s

```

I haven’t done this before so it is likely it was done incorrectly. However, when I initialize hail locally in the hail/hail directory I am running version 0.2.24-e3e63a2f9856. I think it’s just I can’t hook up to gcs?

---

<div class="post-metadata">

**Author:** ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)\
**Post date:** [October 8, 2019, 3:30pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/11 "2019-10-08T15:30:05Z")

</div>

This code works for me with my home-spun `9c44fc9e7c2b` (or maybe `22f6defd17d` who knows with my rig anymore, anyway relatively recent) so I don’t think it’s the old requiredness problem.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 4:51pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/12 "2019-10-08T16:51:41Z")

</div>

@mwilson I mean try on the cloud with a custom build – to do this you do (from `hail/hail`):

```auto
HAILCTL_BUCKET_BASE="gs://a-bucket-you-can-write-to" make install-hailctl

```

---

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 8, 2019, 5:22pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/13 "2019-10-08T17:22:00Z")

</div>

With or without ` mt = mt.select_globals()`, I now see this error when running the code above. Would the full log be helpful?

> FatalError: RuntimeException: globals mismatch:  
> typ: Struct{}  
> val: +Struct{}
> 
> ```
> Java stack trace:
> 
> ```

```
java.lang.RuntimeException: globals mismatch:
  typ: Struct{}
  val: +Struct{}
	at is.hail.expr.ir.TableValue.<init>(TableValue.scala:50)
	at is.hail.expr.ir.TableNativeReader.apply(TableIR.scala:181)
	at is.hail.expr.ir.TableRead.execute(TableIR.scala:295)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:748)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:89)
	at is.hail.expr.ir.Interpret$.apply(Interpret.scala:59)
	at is.hail.expr.ir.InterpretNonCompilable$$anonfun$5.apply(InterpretNonCompilable.scala:16)
	at is.hail.expr.ir.InterpretNonCompilable$$anonfun$5.apply(InterpretNonCompilable.scala:16)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
	at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:16)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$2.apply(CompileAndEvaluate.scala:37)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$2.apply(CompileAndEvaluate.scala:37)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:24)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:37)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:57)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:57)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:8)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:7)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:7)
	at is.hail.backend.Backend.execute(Backend.scala:57)
	at is.hail.backend.Backend.executeJSON(Backend.scala:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2.24-e3e63a2f9856
Error summary: RuntimeException: globals mismatch:
  typ: Struct{}
  val: +Struct{}

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 6:02pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/14 "2019-10-08T18:02:40Z")

</div>

@konradjk you didn’t get all the `+`s ☹

I think there must be some buried in other metadata.json.gz that weren’t used then, but are now.

---

<div class="post-metadata">

**Author:** ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)\
**Post date:** [October 8, 2019, 6:05pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/15 "2019-10-08T18:05:15Z")

</div>

ah that makes sense. i think i only had to do the overall metadata.json.gz one before, but maybe now we need to do more. sigh.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 6:06pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/16 "2019-10-08T18:06:31Z")

</div>

I can fix this in Hail though

---

<div class="post-metadata">

**Author:** ![konradjk](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/konradjk/32/12_2.png) [@konradjk](https://discuss.hail.is/u/konradjk)\
**Post date:** [October 8, 2019, 6:07pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/17 "2019-10-08T18:07:13Z")

</div>

oh that’d be great. the previous fix was pretty nervewracking

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 6:13pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/18 "2019-10-08T18:13:46Z")

</div>

> <https://github.com/hail-is/hail/pull/7227>

---

<div class="post-metadata">

**Author:** ![mwilson](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/mwilson/32/376_2.png) [@mwilson](https://discuss.hail.is/u/mwilson)\
**Post date:** [October 8, 2019, 8:51pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/19 "2019-10-08T20:51:39Z")

</div>

The code above now runs, i.e. filtering the columns only, but when I run

```
mt = mt.filter_rows(hl.agg.any(mt.GT.is_non_ref()))
mt.count()

```

I am seeing the same global mismatch error.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [October 8, 2019, 8:54pm UTC](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114/20 "2019-10-08T20:54:00Z")

</div>

the log file might help here, then

[Next page](https://discuss.hail.is/t/error-when-running-count-after-filtering-mt/1114.md?page=2)