Qq_plot error FatalError: ClassNotFoundException: is.hail.expr.ir.agg.ApproxCDFCombiner

Hail Query & hailctl
1

hail_log.log (2.3 MB) Hi hail team,
I have run gwas on a modest sized matrixtable and I have successfully ploted the manhattan plot but when I try to create the qq plot I get a strange error that I have never seen before nor know how to deal with. Any ideas please? I attach the log (part of as it was too big to upload here)
Thank you again for your help!

gwas = hl.linear_regression_rows(
        y=mt.phenotype.disease,
        x=mt.GT.n_alt_alleles(), covariates=[1.0], pass_through=[mt.rsid])
    gwas = gwas.checkpoint(f"{tmp_dir}/ukbb-plink/gwas/ukbb-plink-checkpoint", overwrite=True)
    gwas.export(f"{tmp_dir}/ukbb-plink/gwas/ukbb-plink-export.tsv.bgz", header=True)


    print("Plotting ")
    
    p = hl.plot.manhattan(gwas.p_value, title="UKBB plink Manhattan plot")
    output_file(f"{temp_dir}/UKBB-manhattan.html")
    save(p)
        #hl.hadoop_copy(f"{i}.WGS-manhattan-{covariates[i]}.html", f"{BUCKET}/gwas/plots/")
    p = hl.plot.qq(gwas.p_value, title="UKBB plink QQ plot")
    output_file(f"{temp_dir}/UKBB-QQplot.html")
    save(p)

The error is:

FatalError Traceback (most recent call last)
in
----> 1 q = hl.plot.qq(gwas.p_value,collect_all=False, title=“UKBB plink QQ plot”)
FatalError: ClassNotFoundException: is.hail.expr.ir.agg.ApproxCDFCombiner

Java stack trace:
java.lang.RuntimeException: error while applying lowering ‘InterpretNonCompilable’
at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:26)
at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:18)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:18)
at is.hail.expr.ir.CompileAndEvaluate$$anonfun$apply$1.apply(CompileAndEvaluate.scala:16)
at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:14)
at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:56)
at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:56)
at is.hail.utils.package$.using(package.scala:596)
at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:10)
at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:9)
at is.hail.utils.package$.using(package.scala:596)
at is.hail.annotations.Region$.scoped(Region.scala:18)
at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:9)
at is.hail.backend.Backend.execute(Backend.scala:56)
at is.hail.backend.Backend.executeJSON(Backend.scala:62)
at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2

Paging Dr. @patrick-schultz

3

have seen this before: https://hail.zulipchat.com/#narrow/stream/123010-Hail-0.2E2.20support/topic/ApproxCDFCombiner.20class.20not.20found

No solution…I have no idea what’s going on.

4

@pavlos is this reliably reproducible for you? And what version of Spark and Hail are you using? It looks like your log is just the tail of the log.

Also, what is your spark configuration and with what parameters did you invoke pyspark (or what is PYSPARK_SUBMIT_ARGS if you’re using python/ipython/jupyter)?

5

I also met the same issue when I was running GWAS tutorial using jupyter notebook on GCP. Does anyone have some solution?

6

This problem (which we still don’t understand) should be solved by this change, which will land in 0.2.31. I plan to make that release on Tuesday, Jan 21.

1 Like