Error with mt.s.show()

Hail Query & hailctl
1

Hi Tim,

I got this error when I tried to run mt.s.show(5) :

FatalError Traceback (most recent call last)
in ()
----> 1 mt.s.show(5)

in show(self, n, width, truncate, types, handler)

/home/hail/hail.zip/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
558 def wrapper(original_func, *args, **kwargs):
559 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 560 return original_func(*args, **kwargs)
561
562 return wrapper

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in show(self, n, width, truncate, types, handler)
686 Print an extra header line with the type of each field.
687 “”"
–> 688 handler(self._show(n, width, truncate, types))
689
690 def _show(self, n=10, width=90, truncate=None, types=True):

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in _show(self, n, width, truncate, types)
709 if source is not None:
710 name = source._fields_inverse.get(self, name)
–> 711 t = self._to_table(name)
712 if name in t.key:
713 t = t.key_by(name).select()

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in _to_table(self, name)
627 else:
628 m = source.select_cols(field_name)
–> 629 m = m.rename({field_name: name})
630 else:
631 m = source.select_cols(**{name: self})

in rename(self, fields)

/home/hail/hail.zip/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
558 def wrapper(original_func, *args, **kwargs):
559 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 560 return original_func(*args, **kwargs)
561
562 return wrapper

/home/hail/hail.zip/hail/matrixtable.py in rename(self, fields)
3240 global_map[k] = v
3241
-> 3242 return MatrixTable(self._jvds.renameFields(row_map, col_map, entry_map, global_map))
3243
3244 def distinct_by_row(self):

/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in call(self, *args)
1131 answer = self.gateway_client.send_command(command)
1132 return_value = get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name)
1134
1135 for temp_arg in temp_args:

/home/hail/hail.zip/hail/utils/java.py in deco(*args, **kwargs)
208 raise FatalError(’%s\n\nJava stack trace:\n%s\n’
209 ‘Hail version: %s\n’
–> 210 ‘Error summary: %s’ % (deepest, full, hail.version, deepest)) from None
211 except pyspark.sql.utils.CapturedException as e:
212 raise FatalError(’%s\n\nJava stack trace:\n%s\n’

FatalError: AssertionError: assertion failed

Java stack trace:
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:156)
at is.hail.variant.MatrixTable.value$lzycompute(MatrixTable.scala:418)
at is.hail.variant.MatrixTable.value(MatrixTable.scala:415)
at is.hail.variant.MatrixTable.x$9$lzycompute(MatrixTable.scala:422)
at is.hail.variant.MatrixTable.x$9(MatrixTable.scala:422)
at is.hail.variant.MatrixTable.rvd$lzycompute(MatrixTable.scala:422)
at is.hail.variant.MatrixTable.rvd(MatrixTable.scala:422)
at is.hail.variant.MatrixTable.renameFields(MatrixTable.scala:968)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2-29fbaeaf265e
Error summary: AssertionError: assertion failed

Do you have any idea, please?

Best,

Laura

2

Hi Laura,
could you try updating to the latest version first? It’s quite possible we’ve fixed this in the last 2 weeks.

Sorry for the trouble.

3

Hi Tim,

I updated Hail with the last version: version 0.2-6866448d481e
I still have an error:

FatalError Traceback (most recent call last)
in ()
----> 1 mt.s.show(5)

in show(self, n, width, truncate, types, handler)

/home/hail/hail.zip/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
558 def wrapper(original_func, *args, **kwargs):
559 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 560 return original_func(*args, **kwargs)
561
562 return wrapper

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in show(self, n, width, truncate, types, handler)
686 Print an extra header line with the type of each field.
687 “”"
–> 688 handler(self._show(n, width, truncate, types))
689
690 def _show(self, n=10, width=90, truncate=None, types=True):

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in _show(self, n, width, truncate, types)
709 if source is not None:
710 name = source._fields_inverse.get(self, name)
–> 711 t = self._to_table(name)
712 if name in t.key:
713 t = t.key_by(name).select()

/home/hail/hail.zip/hail/expr/expressions/base_expression.py in _to_table(self, name)
627 else:
628 m = source.select_cols(field_name)
–> 629 m = m.rename({field_name: name})
630 else:
631 m = source.select_cols(**{name: self})

in rename(self, fields)

/home/hail/hail.zip/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
558 def wrapper(original_func, *args, **kwargs):
559 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
–> 560 return original_func(*args, **kwargs)
561
562 return wrapper

/home/hail/hail.zip/hail/matrixtable.py in rename(self, fields)
3240 global_map[k] = v
3241
-> 3242 return MatrixTable(self._jvds.renameFields(row_map, col_map, entry_map, global_map))
3243
3244 def distinct_by_row(self):

/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in call(self, *args)
1131 answer = self.gateway_client.send_command(command)
1132 return_value = get_return_value(
-> 1133 answer, self.gateway_client, self.target_id, self.name)
1134
1135 for temp_arg in temp_args:

/home/hail/hail.zip/hail/utils/java.py in deco(*args, **kwargs)
208 raise FatalError(’%s\n\nJava stack trace:\n%s\n’
209 ‘Hail version: %s\n’
–> 210 ‘Error summary: %s’ % (deepest, full, hail.version, deepest)) from None
211 except pyspark.sql.utils.CapturedException as e:
212 raise FatalError(’%s\n\nJava stack trace:\n%s\n’

FatalError: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}

Java stack trace:
java.lang.AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}
at scala.Predef$.assert(Predef.scala:170)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:42)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:129)
at is.hail.expr.ir.LowerMatrixIR$$anonfun$matrixRules$1.applyOrElse(LowerMatrixIR.scala:91)
at is.hail.expr.ir.LowerMatrixIR$.is$hail$expr$ir$LowerMatrixIR$$lower(LowerMatrixIR.scala:40)
at is.hail.expr.ir.LowerMatrixIR$.apply(LowerMatrixIR.scala:16)
at is.hail.expr.ir.Interpret$.apply(Interpret.scala:68)
at is.hail.variant.MatrixTable.x$9$lzycompute(MatrixTable.scala:419)
at is.hail.variant.MatrixTable.x$9(MatrixTable.scala:419)
at is.hail.variant.MatrixTable.rvd$lzycompute(MatrixTable.scala:419)
at is.hail.variant.MatrixTable.rvd(MatrixTable.scala:419)
at is.hail.variant.MatrixTable.renameFields(MatrixTable.scala:965)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)

Hail version: 0.2-6866448d481e
Error summary: AssertionError: assertion failed: Table{global:Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}
Table{global:+Struct{__cols:Array[Struct{s:String}]},key:[locus,alleles],row:Struct{locus:Locus(GRCh37),alleles:Array[String],rsid:String,qual:Float64,filters:Set[String],info:Struct{AC:Array[Int32],AF:Array[Float64],AN:Int32,DP:Int32,AVGDP:Float64,GC:Array[Int32],GN:Int32,HWEAF:Array[Float64],HWDGF:Array[Float64],IBC:Float64,HWE_SLP:Float64,ABE:Float64,ABZ:Float64,NS_NREF:Int32,BQZ:Float64,CYZ:Float64,STZ:Float64,NMZ:Float64,IOR:Float64,NM0:Float64,NM1:Float64,SVM:Float64,OVERLAP:Array[String],ANN:Array[String],LOF:Array[String],NMD:Array[String],IS_HWE_SLP:Float64,IS_IBC:Float64},the entries! [877f12a8827e18f61222c6c8c5fb04a8]:Array[Struct{GT:Call,DP:Int32,PL:Array[+Int32],GQ:Int32,AD:Array[+Int32]}]}}

I have other errors with mt.count(), hl.summarize_variants(mt).

Best,

Laura

4

Thanks for updating!

Ah… I think I see the problem. Was this table written a while ago (~6 months)? We had an issue where we wrote files with an unintended property for a few weeks, then fixed the problem. However, these files cause trouble with some newer infrastructure.

Could you try putting .select_globals() right after your hl.read_matrix_table and see if downstream things work after that? If so, I’ll give you a way to edit the file to fix the problem permanently.

5

I still have the same error.
Do we have to write a new mt file?

6

to clarify you have:

mt = hl.read_matrix_table(...).select_globals()

(or the equivalent)

When Konrad has had similar errors, this fixes it!

7

yes, I did that

8

can you do the following and paste the result:

mt = hl.read_matrix_table(...)
print(mt._jvds.globalType().parsableString())
mt = mt.select_globals()
print(mt._jvds.globalType().parsableString())
9

+Struct{}
Struct{}

10

ah, darn, a recent change broke this hack.

This pull request will fix the problem:

https://github.com/hail-is/hail/pull/4778

11

Hi Tim,

It works! Thanks!