Method code too large error

When I run the following lines of code:

table1.row.show()
table2.row.show()

For table1 it will show the data, but for table2 it will not.

Error is:

---------------------------------------------------------------------------
FatalError                                Traceback (most recent call last)
<ipython-input-80-bf4d6c719c23> in <module>()
  1 table1.row.show()
----> 2 table2.row.show()

/home/hail/hail.zip/hail/typecheck/check.py in wrapper(*args, **kwargs)
545         def wrapper(*args, **kwargs):
546             args_, kwargs_ = check_all(f, args, kwargs, checkers, is_method=is_method)
--> 547             return f(*args_, **kwargs_)
548 
549         update_wrapper(wrapper, f)

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in show(self, n, width, truncate, types)
677             Print an extra header line with the type of each field.
678         """
--> 679         print(self._show(n, width, truncate, types))
680 
681     def _show(self, n=10, width=90, truncate=None, types=True):

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in _show(self, n, width, truncate, types)
684         if isinstance(source, hl.Table):
685             if self is source.row:
--> 686                 return source._show(n, width, truncate, types)
687             elif self is source.key:
688                 return source.select()._show(n, width, truncate, types)

/home/hail/hail.zip/hail/table.py in _show(self, n, width, truncate, types)
   1201 
   1202     def _show(self, n=10, width=90, truncate=None, types=True):
-> 1203         return self._jt.showString(n, joption(truncate), types, width)
   1204 
   1205     def index(self, *exprs):

/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1131         answer = self.gateway_client.send_command(command)
   1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
   1134 
   1135         for temp_arg in temp_args:

/home/hail/hail.zip/hail/utils/java.py in deco(*args, **kwargs)
194             raise FatalError('%s\n\nJava stack trace:\n%s\n'
195                              'Hail version: %s\n'
--> 196                              'Error summary: %s' % (deepest, full, hail.__version__, deepest)) from None
197         except pyspark.sql.utils.CapturedException as e:
198             raise FatalError('%s\n\nJava stack trace:\n%s\n'

FatalError: RuntimeException: Method code too large!

Java stack trace:
java.lang.RuntimeException: Method code too large!
	at is.hail.relocated.org.objectweb.asm.MethodWriter.a(Unknown Source)
	at is.hail.relocated.org.objectweb.asm.ClassWriter.toByteArray(Unknown Source)
	at is.hail.asm4s.FunctionBuilder.classAsBytes(FunctionBuilder.scala:306)
	at is.hail.expr.ir.EmitFunctionBuilder.result(EmitFunctionBuilder.scala:284)
	at is.hail.expr.ir.Compile$.apply(Compile.scala:50)
	at is.hail.expr.ir.Compile$.apply(Compile.scala:31)
	at is.hail.expr.ir.Compile$.apply(Compile.scala:72)
	at is.hail.expr.ir.TableMapRows.execute(TableIR.scala:413)
	at is.hail.expr.ir.TableKeyBy.execute(TableIR.scala:161)
	at is.hail.table.Table.value$lzycompute(Table.scala:237)
	at is.hail.table.Table.value(Table.scala:232)
	at is.hail.table.Table.x$5$lzycompute(Table.scala:240)
	at is.hail.table.Table.x$5(Table.scala:240)
	at is.hail.table.Table.rvd$lzycompute(Table.scala:240)
	at is.hail.table.Table.rvd(Table.scala:240)
	at is.hail.table.Table.take(Table.scala:875)
	at is.hail.table.Table.showString(Table.scala:911)
	at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:280)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:748)

Hail version: devel-eb1e04205793
Error summary: RuntimeException: Method code too large!

What am I doing wrong?

what is table2? How many fields does it have? what is your pipeline?

Need this info to debug.

Table2 has 96 rows and 5960 columns.

I have dont nothing to the data accept read it in. I read is the data like such:

#New annotations
table2 = (hl.import_table('gs://nneka/Supplementary_Table_1.csv', delimiter=',', impute=True, missing='NA').key_by('Individual ID'))

Nice. That’s a ton of columns.

Hail is clearly running into some limitations. Can you share the hail.log file? It should be in the same directory that you started python in. The information in the log file will help us understand how we can avoid this limitation.

In the meantime, is it possible for you to use only some of the columns?

@danking
I am not sure how to find this, could you please direct me? I am running a Jupyter Notebook in google cloud.

I will try to run only a few columns, thank you.

If you’re running a jupyter notebook, open a terminal like this (top right from the jupyter home screen)

You should see hail.log if you ls from that directory. Next. copy it to your bucket: gsutil cp hail.log gs://(path here)

Then you can download and share with us! Thanks.

https://github.com/hail-is/hail/issues/3922

The log file is at the end.

thanks!! This is very useful and already points out potential problems.