Fresh install fails eigSymDSuite and LinearMixedRegressionSuite tests

[CanWood]

Hi folks,

I’ve just done a fresh install of spark-2.0.2 and hail from today’s git. On executing the Hail tests, all but two passed: LinearMixedRegressionSuite.lmmLargeExampleTest and eigSymDSuite.eigSymTest. On review of both, /hail/build/reports/tests/classes/is.hail.methods.LinearMixedRegressionSuite.html and /hail/build/reports/tests/classes/is.hail.stats.eigSymDSuite.html show the same (verbatim):

breeze.linalg.MatrixNotSymmetricException: Matrix is not symmetric
	at breeze.linalg.package$$anonfun$requireSymmetricMatrix$1$$anonfun$apply$mcVI$sp$1.apply$mcVI$sp(package.scala:122)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at breeze.linalg.package$$anonfun$requireSymmetricMatrix$1.apply$mcVI$sp(package.scala:120)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at breeze.linalg.package$.requireSymmetricMatrix(package.scala:120)
	at breeze.linalg.cholesky$ImplCholesky_DM$.apply(cholesky.scala:25)
	at breeze.linalg.cholesky$ImplCholesky_DM$.apply(cholesky.scala:16)
	at breeze.generic.UFunc$class.apply(UFunc.scala:48)
	at breeze.linalg.cholesky$.apply(cholesky.scala:15)
	at is.hail.methods.LinearMixedRegressionSuite.lmmLargeExampleTest(LinearMixedRegressionSuite.scala:200)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:696)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
	at org.testng.TestNG.run(TestNG.java:1057)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.runTests(TestNGTestClassProcessor.java:133)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:83)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:120)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)```

Any troubleshooting tips?

Cheers!
CanWood

[danking]

Hey CanWood, sorry to hear that you ran into this error! I’m not entirely sure why you’re receiving this error. It’s possible that there’s an issue with the Breeze native libraries that your system is providing. In order to best help us reproduce this issue can you tell us:

• your operating system and version
• the version of Scala you’re using (scalac -version)

Can you also try running the tests with this patch, which will disable the breeze natives?

git am /path/to/0001-use-breeze-non-natives.patch
gradle test --tests '*LinearMixedRegressionSuite*' --tests '*eigSymDSuite*'

in your hail repository directory?

[jbloom]

For the record, the matrix at fault is literally of the form X^T * X, so symmetric. Without disabling natives, I’m also curious whether increasing the Java stack size will fix this issue:

Fixing logreg, lmmreg error when using many sample covariates on Dataproc

We were surprised to find that using more than 9 sample covariates in logistic or linear mixed regression on Google Dataproc would throw an error. We’ve engaged Google support on a superior fix, but in the meantime they’ve suggested and we’ve verified the work around of including the properties spark.driver.extraJavaOptions=-Xss4M and spark.executor.extraJavaOptions=-Xss4M in the cluster creation command to increase the Java stack size, e.g.: --properties="spark:spark.driver.extraJavaOptions=-X…

[CanWood]

Thanks, danking and jbloom,

Sorry for the delay getting back to you. I was afk for days.

We are running on a CentOS 7.2.1511 system, with a kernel pinned to 3.10.0-123.20.1.el7.x86_64, R/3.3.1, Java/1.8.0_65-b17, plink/1.90b3.40, R/3.3.1, and python/2.7.10. I don’t currently have a version of scala in my path but from my reading of the spark install docs and hail install docs, I didn’t think we needed one.

jbloom: I’ve run with the stack size cranked way up and and without disabling the breeze natives with Dan’s patch. That didn’t get us over the hump. Same errors. Moving on to Dan’s tips…

danking: First, I dropped a scala 2.11.8 in to my path, which also upped my java version to 1.8.0_74, just due to our environment modules. I re-ran the tests and they failed, similarly.

Next, I applied your patch. I did not have a gradle in my path, as the docs said to use ./gradlew, so I opted to, for consistency, use ./gradlew again. On running those tests, the tests were successful! Thanks for that.

So, now that I have the tests succeeding, is the take home message here that breeze natives are an issue? Might users run in to similar issues as your tests did?

Again, thanks both for yout tips and time.

Cheers,
CanWood

[danking]

CanWood,

Yeah, using ./gradlew handles the compiler dependencies and such for you, so my question about versions is irrelevant.

Regarding the patch causing the tests to succeed, yes, this implies that the breeze natives are related to the cause of the error. I suspect you’re experiencing exactly the error discussed in this breeze issue. This issue is not caused directly by breeze natives. I suspect the breeze natives are producing a “less symmetric” matrix than the Scala reference implementations. With this issue in mind, I suspect the breeze natives are correct (enough) and the real problem is an over zealous breeze symmetry check.

The solution is for Hail to increase our breeze dependency to at least v0.12. This version was the first release to include a relaxation of the symmetry constraints. I will look into this, unfortunately, I believe we must depend on the same version of breeze that Spark provides.

[danking]

@CanWood,

If you can use version 2.1.0 of Spark, then I suggest using that version. After this PR is merged, Hail will use version Breeze 0.12 when compiled for Spark 2.1.0 (using ./gradlew -Dspark.version=2.1.0 shadowJar, which should not trigger the error you are receiving.

I’ve tested this locally on my machine, but I was never able to trigger the issue you’re having. Please let me know if this resolves your issue!

[CanWood]

Thanks, Dan! It’ll be a bit before I get to installing and testing. I’ll let you know how I go when I do.

Cheers