# Hail 0.2 - Method code too large error in the latest build

**URL:** <https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497>\
**Category:** Hail Query & hailctl\
**Created:** [May 11, 2018, 4:04pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497 "2018-05-11T16:04:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bjo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/bjo/32/149_2.png) [@bjo](https://discuss.hail.is/u/bjo)\
**Post date:** [May 11, 2018, 4:04pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/1 "2018-05-11T16:04:53Z")

</div>

Has there been a change with the from\_pandas() method?

This part of my pipeline was working until last weekend, but now it gives me the Method Code too large error (I’ll remind you that I have to use pandas to invert my original data, since the original data has samples in rows):

chrom\_expr\_ht = hl.Table.from\_pandas(expr\_df).key\_by(‘ID’)

I’m attaching the entry from the log file that may be relevant (I’ve masked the field names and gene names).

2018-05-11 05:20:41 MemoryStore: INFO: Block broadcast\_32 stored as values in memory (estimated size 437.7 KB, free 21.7 GB)  
2018-05-11 05:20:41 MemoryStore: INFO: Block broadcast\_32\_piece0 stored as bytes in memory (estimated size 30.0 KB, free 21.7 GB)  
2018-05-11 05:20:41 BlockManagerInfo: INFO: Added broadcast\_32\_piece0 in memory on 10.128.0.54:42791 (size: 30.0 KB, free: 21.7 GB)  
2018-05-11 05:20:41 SparkContext: INFO: Created broadcast 32 from textFile at RichSparkContext.scala:16  
2018-05-11 05:20:41 FileInputFormat: INFO: Total input files to process : 1  
2018-05-11 05:20:41 root: INFO: in Table.value: execute:  
(TableMapRows  
(TableLiteral)  
(MakeStruct  
(ID  
(GetField ID  
(Ref row)))  
(field\_1  
(GetField field\_1  
(Ref row)))  
(field\_2  
(GetField field\_2  
(Ref row)))  
(field\_3  
(GetField field\_3  
(Ref row)))

…

```
Select[ENSG00000XXXXX](
  SymRef[__uid_58]
)
Select[ENSG00000XXXXX](
  SymRef[__uid_58]
)
Select[ENSG00000XXXXX](
  SymRef[__uid_58]
)

```

)  
)  
2018-05-11 05:22:01 root: INFO: is/hail/codegen/generated/C75. instruction count: 3  
2018-05-11 05:22:01 root: INFO: is/hail/codegen/generated/C75.apply instruction count: 36158  
2018-05-11 05:22:01 root: INFO: is/hail/codegen/generated/C75.apply instruction count \> 8000  
2018-05-11 05:22:01 root: INFO: is/hail/codegen/generated/C75.apply instruction count: 9  
2018-05-11 05:22:02 root: ERROR: RuntimeException: Method code too large!  
From java.lang.RuntimeException: Method code too large!  
at is.hail.relocated.org.objectweb.asm.MethodWriter.a(Unknown Source)  
at is.hail.relocated.org.objectweb.asm.ClassWriter.toByteArray(Unknown Source)  
at is.hail.asm4s.FunctionBuilder.classAsBytes(FunctionBuilder.scala:256)  
at is.hail.asm4s.FunctionBuilder.result(FunctionBuilder.scala:288)  
at is.hail.expr.CM.runWithDelayedValues(CM.scala:80)  
at is.hail.expr.Parser$.is$hail$expr$Parser$$evalNoTypeCheck(Parser.scala:60)  
at is.hail.expr.Parser$.eval(Parser.scala:73)  
at is.hail.expr.Parser$.parseExpr(Parser.scala:88)  
at is.hail.table.Table.select(Table.scala:606)  
at is.hail.table.Table.select(Table.scala:594)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:498)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:280)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:214)  
at java.lang.Thread.run(Thread.java:748)

2018-05-11 05:22:02 SparkContext: INFO: Invoking stop() from shutdown hook  
2018-05-11 05:22:02 AbstractConnector: INFO: Stopped Spark@e685744{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [May 11, 2018, 4:07pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/2 "2018-05-11T16:07:35Z")

</div>

The “method code too large” errors come from the new, faster code path we’re working on. We haven’t touched to\_pandas, but we’ve been working to make all the internal transformations take the fast path, so seeing this error pop up doesn’t surprise me too much.

@danking @wang do we have thoughts?

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [May 11, 2018, 4:24pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/3 "2018-05-11T16:24:22Z")

</div>

also, I’d really recommend writing to disk and importing to Hail, instead of `from_pandas`. It’ll probably be way faster.

---

<div class="post-metadata">

**Author:** ![bjo](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/bjo/32/149_2.png) [@bjo](https://discuss.hail.is/u/bjo)\
**Post date:** [May 19, 2018, 12:07am UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/4 "2018-05-19T00:07:48Z")

</div>

@danking @wang

Any plans to look into this issue? I encounter this error even without using from\_pandas.

Another case where I encounter this error in the latest build (5d2bfa56e252):

```
expr_tp_ht = hl.import_table(expr_path + tissue + suffix, impute = True).key_by('gene_id')
expr_tp_ht = expr_tp_ht.join(gene_anno_ht.select(gene_anno_ht.mappability), how = 'left')
expr_tp_ht = expr_tp_ht.filter(expr_tp_ht.mappability >= 0.8)
expr_tp_ht = expr_tp_ht.transmute(chrom = hl.int(expr_tp_ht['#chr'][3:]))
expr_tp_ht = expr_tp_ht.transmute(TSS = expr_tp_ht.end).drop('start')

```

I tried writing (or running show()) expr\_tp\_ht after each line of this code block, and it seems like the ‘Method Code too Large’ error occurs after the last line. I’m pretty sure the syntax is correct - actually, this error is triggered in only a subset of my runs, which are different tissues, which I find very bizarre (for example, 6 out of 15 tissues trigger this error while the other 9 are running fine). These 6 don’t necessarily have the largest number of rows/columns either. I also tried repartition() to have more partitions, and also tried running annotate() instead of trasmute(), but to no avail.

For a little more information:  
2018-05-18 23:29:40 root: INFO: in Table.value: execute:  
(TableJoin  
(TableMapRows  
(TableMapRows  
(TableMapRows  
(TableKeyBy  
(TableLiteral))  
(Let \_\_uid\_4  
(SelectFields  
(ApplyIR annotate  
(SelectFields  
(Ref row))  
(MakeStruct  
(TSS  
(GetField end  
(Ref row))))))  
(ApplyIR annotate  
(MakeStruct  
(gene\_id  
(GetField gene\_id  
(Ref row))))  
(MakeStruct  
(start  
(GetField start  
(Ref \_\_uid\_4)))  
(SUBJ-1  
(GetField SUBJ-1  
(Ref \_\_uid\_4)))  
…  
(TSS  
(GetField TSS  
(Ref \_\_uid\_5)))))))  
(Let \_\_uid\_6  
(MakeStruct  
(TSS  
(GetField TSS  
(Ref row))))  
(ApplyIR annotate  
(MakeStruct  
(gene\_id  
(GetField gene\_id  
(Ref row))))  
(MakeStruct  
(TSS  
(GetField TSS  
(Ref \_\_uid\_6)))))))  
(TableRead gs://gtex-hail/Data/Annotation/gencode\_26/gencode\_v26\_autosomal\_genes.ht))  
2018-05-18 23:29:41 root: INFO: is/hail/codegen/generated/C21. instruction count: 3  
2018-05-18 23:29:41 root: INFO: is/hail/codegen/generated/C21.apply instruction count: 56963  
2018-05-18 23:29:41 root: INFO: is/hail/codegen/generated/C21.apply instruction count \> 8000  
2018-05-18 23:29:41 root: INFO: is/hail/codegen/generated/C21.apply instruction count: 22  
2018-05-18 23:29:41 root: ERROR: RuntimeException: Method code too large!

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [May 22, 2018, 1:57pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/5 "2018-05-22T13:57:21Z")

</div>

Hi @bjo,

Sorry for the delay in my response. Can you share the output of `expr_tp_ht.describe()` and `gene_anno_ht.describe()` or more relevantly, how many row fields does each table have?

To get you unblocked, I would recommend dropping any fields that are unnecessary for your downstream computations. I realize this is not an ideal solution.

We have a long term plan (stop using the JVM, which enforces these method size limits) and a medium term plan (be smarter about the generated code so it isn’t too big). I’m exploring some short term fixes, I’ll get back to you later today if I have any ideas.

Also: Can you share the full hail.log?

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [May 22, 2018, 9:25pm UTC](https://discuss.hail.is/t/hail-0-2-method-code-too-large-error-in-the-latest-build/497/6 "2018-05-22T21:25:39Z")

</div>

> [@bjo](#):
>
> gene\_anno\_ht

One more thing, I suspect the issue is actually in the definition of `gene_anno_ht` not in the code you’ve posted. Can you share the definition of `gene_anno_ht`?