Local Hail Install issues

Help [0.1]
1

Hi there, I was in Boulder last week and am excited to use Hail more. I’m having issues installing it locally, namely:

VIPBGB1L01130AB:hail jalexander4$ ./gradlew shadowJar
:compileJava UP-TO-DATE
:nativeLib
(cd libsimdpp-2.0-rc2 && cmake .)
/bin/sh: cmake: command not found
make: *** [libsimdpp] Error 127
:nativeLib FAILED

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ‘:nativeLib’.

Process ‘command ‘make’’ finished with non-zero exit value 2

  • Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 9.439 secs

I have Java 8 (build 1.8.0_121-b13) and here is c++ -v

VIPBGB1L01130AB:hail jalexander4$ c++ -v
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

and sysctl -a | grep machdep.cpu

VIPBGB1L01130AB:hail jalexander4$ sysctl -a | grep machdep.cpu
machdep.cpu.max_basic: 20
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core™ i7-5557U CPU @ 3.10GHz
machdep.cpu.family: 6
machdep.cpu.model: 61
machdep.cpu.extmodel: 3
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 4
machdep.cpu.feature_bits: 9221959987971750911
machdep.cpu.leaf7_feature_bits: 35399595
machdep.cpu.extfeature_bits: 1241984796928
machdep.cpu.signature: 198356
machdep.cpu.brand: 0
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID SMAP RDSEED ADX IPT FPU_CSDS
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI
machdep.cpu.logical_per_package: 16
machdep.cpu.cores_per_package: 8
machdep.cpu.microcode_version: 33
machdep.cpu.processor_flag: 6
machdep.cpu.mwait.linesize_min: 64
machdep.cpu.mwait.linesize_max: 64
machdep.cpu.mwait.extensions: 3
machdep.cpu.mwait.sub_Cstates: 286531872
machdep.cpu.thermal.sensor: 1
machdep.cpu.thermal.dynamic_acceleration: 1
machdep.cpu.thermal.invariant_APIC_timer: 1
machdep.cpu.thermal.thresholds: 2
machdep.cpu.thermal.ACNT_MCNT: 1
machdep.cpu.thermal.core_power_limits: 1
machdep.cpu.thermal.fine_grain_clock_mod: 1
machdep.cpu.thermal.package_thermal_intr: 1
machdep.cpu.thermal.hardware_feedback: 0
machdep.cpu.thermal.energy_policy: 1
machdep.cpu.xsave.extended_state: 7 832 832 0
machdep.cpu.xsave.extended_state1: 1 0 0 0
machdep.cpu.arch_perf.version: 3
machdep.cpu.arch_perf.number: 4
machdep.cpu.arch_perf.width: 48
machdep.cpu.arch_perf.events_number: 7
machdep.cpu.arch_perf.events: 0
machdep.cpu.arch_perf.fixed_number: 3
machdep.cpu.arch_perf.fixed_width: 48
machdep.cpu.cache.linesize: 64
machdep.cpu.cache.L2_associativity: 8
machdep.cpu.cache.size: 256
machdep.cpu.tlb.inst.large: 8
machdep.cpu.tlb.data.small: 64
machdep.cpu.tlb.data.small_level1: 64
machdep.cpu.address_bits.physical: 39
machdep.cpu.address_bits.virtual: 48
machdep.cpu.core_count: 2
machdep.cpu.thread_count: 4
machdep.cpu.tsc_ccc.numerator: 0
machdep.cpu.tsc_ccc.denominator: 0

I’m new to this, so it may be something very simple.

Thanks,
Jeff

2

Hi Jeff, glad you liked the workshop! The error says you’re missing a C compiler, e.g. cmake. Have you installed this as described in Getting Started?

3

Thanks for getting back to me. I have installed both XCode and CMake (just downloaded and installed, not using Homebrew).

4

Do you need to add it to your path? What do you get from cmake --version?

6

Yup, just needed to add cmake to the path. Thanks!

7

Glad things are working!

We’ll be putting the practicals from Boulder onto the website in the next week or two, once we can pull out the IBG-specific jargon/references. We’re also going to start posting precompiled Hail binaries very soon – lots of people have had issues compiling, for lots of reasons!

8

I’m having the same issue as above, cmake is installed and in the path

jpriest@blalock:~/priest_apps/hail$ ./gradlew shadowJar
:compileJava UP-TO-DATE
:generateBuildInfo
:nativeLib
(cd libsimdpp-2.0-rc2 && cmake .)
CMake Error: Could not open file for write in copy operation /home/jpriest/priest_apps/hail/src/main/c/libsimdpp-2.0-rc2/.pc.tmp
CMake Error: : System Error: Permission denied
CMake Error at CMakeLists.txt:39 (configure_file):
configure_file Problem configuring file

– Configuring incomplete, errors occurred!
CMake Error: Unable to open cache file for save. /home/jpriest/priest_apps/hail/src/main/c/libsimdpp-2.0-rc2/CMakeCache.txt
CMake Error: : System Error: Permission denied
make: *** [libsimdpp] Error 1
:nativeLib FAILED

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ‘:nativeLib’.

Process ‘command ‘make’’ finished with non-zero exit value 2

  • Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 6.558 secs

09:00:34.553 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process ‘command ‘make’’
09:00:34.557 [QUIET] [system.out] (cd libsimdpp-2.0-rc2 && cmake .)
09:00:34.579 [ERROR] [system.err] CMake Error: Could not open file for write in copy operation /home/jpriest/priest_apps/hail/src/main/c/libsimdpp-2.0-rc2/.pc.tmp
09:00:34.579 [ERROR] [system.err] CMake Error: : System Error: Permission denied
09:00:34.579 [ERROR] [system.err] CMake Error at CMakeLists.txt:39 (configure_file):
09:00:34.579 [ERROR] [system.err] configure_file Problem configuring file
09:00:34.580 [ERROR] [system.err]
09:00:34.580 [ERROR] [system.err]
09:00:35.623 [QUIET] [system.out] – Configuring incomplete, errors occurred!
09:00:35.624 [ERROR] [system.err] CMake Error: Unable to open cache file for save. /home/jpriest/priest_apps/hail/src/main/c/libsimdpp-2.0-rc2/CMakeCache.txt
09:00:35.624 [ERROR] [system.err] CMake Error: : System Error: Permission denied
09:00:35.632 [ERROR] [system.err] make: *** [libsimdpp] Error 1
09:00:35.634 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
09:00:35.634 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process ‘command ‘make’’ finished with exit value 2 (state: FAILED)
09:00:35.636 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ‘:nativeLib’
09:00:35.636 [LIFECYCLE] [class org.gradle.TaskExecutionLogger] :nativeLib FAILED
09:00:35.636 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :nativeLib (Thread[main,5,main]) completed. Took 1.086 secs.
09:00:35.636 [DEBUG] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] Task worker [Thread[main,5,main]] finished, busy: 1.128 secs, idle: 0.011 secs
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter]
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] FAILURE: Build failed with an exception.
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter]
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] * What went wrong:
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] Execution failed for task ‘:nativeLib’.
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] > Process ‘command ‘make’’ finished with non-zero exit value 2
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter]
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] * Try:
09:00:35.643 [ERROR] [org.gradle.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.
09:00:35.644 [LIFECYCLE] [org.gradle.BuildResultLogger]
09:00:35.644 [LIFECYCLE] [org.gradle.BuildResultLogger] BUILD FAILED
09:00:35.644 [LIFECYCLE] [org.gradle.BuildResultLogger]
09:00:35.644 [LIFECYCLE] [org.gradle.BuildResultLogger] Total time: 7.472 secs
09:00:35.645 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache fileSnapshotsToTreeSnapshotsIndex.bin (/home/jpriest/priest_apps/hail/.gradle/2.14.1/taskArtifacts/fileSnapshotsToTreeSnapshotsIndex.bin)
09:00:35.646 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache fileSnapshots.bin (/home/jpriest/priest_apps/hail/.gradle/2.14.1/taskArtifacts/fileSnapshots.bin)
09:00:35.646 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache taskArtifacts.bin (/home/jpriest/priest_apps/hail/.gradle/2.14.1/taskArtifacts/taskArtifacts.bin)
09:00:35.646 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on task history cache (/home/jpriest/priest_apps/hail/.gradle/2.14.1/taskArtifacts).
09:00:35.647 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 1 (avg: 0.032 secs, total: 0.032 secs)
09:00:35.648 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
09:00:35.648 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.002 secs
09:00:35.648 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache Generated Gradle JARs cache (/home/jpriest/.gradle/caches/2.14.1/generated-gradle-jars) was closed 0 times.
09:00:35.649 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-artifacts.bin (/home/jpriest/.gradle/caches/modules-2/metadata-2.16/module-artifacts.bin)
09:00:35.649 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache artifact-at-repository.bin (/home/jpriest/.gradle/caches/modules-2/metadata-2.16/artifact-at-repository.bin)
09:00:35.649 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-metadata.bin (/home/jpriest/.gradle/caches/modules-2/metadata-2.16/module-metadata.bin)
09:00:35.649 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache module-versions.bin (/home/jpriest/.gradle/caches/modules-2/metadata-2.16/module-versions.bin)
09:00:35.649 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on artifact cache (/home/jpriest/.gradle/caches/modules-2).
09:00:35.649 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.memcache.InMemoryCachedRepositoryFactory] In-memory dependency metadata cache closed. Repos cached: 7, cache instances: 5, modules served from cache: 0, artifacts: 0
09:00:35.649 [DEBUG] [org.gradle.cache.internal.btree.BTreePersistentIndexedCache] Closing cache plugin-use-metadata.bin (/home/jpriest/.gradle/caches/2.14.1/plugin-resolution/plugin-use-metadata.bin)
09:00:35.649 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on Plugin Resolution Cache (/home/jpriest/.gradle/caches/2.14.1/plugin-resolution).
09:00:35.650 [DEBUG] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopping 0 compiler daemon(s).
09:00:35.650 [INFO] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Stopped 0 compiler daemon(s).

9

Hi James,
Sorry to leave this hanging for a few days!

I’m happy to report that we’re now compiling prebuilt distributions of Hail. These are publicly available in our Google bucket (gs://hail-common/distributions).

There are now links to the latest versions of Hail 0.1 for Spark 2.0.2 and 2.1.0 in our getting started page. Downloading one of these (corresponding to the version of Spark that you have) will let you run Hail without needing to worry about the compile dependencies yourself. Please let us know if there are parts of the updated page that aren’t clear!

1 Like
10

I second tim’s approach of using a pre-compiled binary. However, if you really must compile from source, it seems that whichever user is executing ./gradlew shadowJar does not own the quoted directory. I would check that

whoami

returns the same user as

ls -l /home/jpriest/priest_apps/hail/src/main/c/libsimdpp-2.0-rc2/ | awk '{print $3}'

before executing gradle.

11

Hi tpoterba
Are the precompiled binaries for Spark 2.1.0.2 on the website?

12

We provide distributions for Spark 2.0.2 and Spark 2.1.0.

The latest links are embedded here:
https://hail.is/docs/stable/getting_started.html