TypeError: 'JavaPackage' object is not callable

Hail Query & hailctl
1

Everything seems to be working except the python. Have you ever seen this error? Clearly I have missed something.

$ python
Python 2.7.13 (default, Jul 7 2017, 13:50:00)
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.

import hail
hc = hail.HailContext()
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File “”, line 1, in
File “”, line 2, in init
File “hail/typecheck/check.py”, line 202, in _typecheck
return f(*args, **kwargs)
File “hail/context.py”, line 83, in init
parquet_compression, min_block_size, branching_factor, tmp_dir)
TypeError: ‘JavaPackage’ object is not callable

Kevin

2

Hi Kevin,
This usually means that the jar file isn’t set correctly. The way to pass it in is with the SPARK_CLASSPATH environment variable. What’s yours set to?

3

Nothing and that would be the problem.

4

Set it to SPARK_CLASSPATH=$HAIL_HOME/build/libs/hail-all-spark.jar and now it works.

5

Fantastic! I’ll change the title of the post to make it more searchable for the “JavaPackage object is not callable” error.

6

NB: Beryl and I just noticed this also seems to happen if you use Spark 2.2.0 - make sure you’re using 2.0.2 or 2.1.0 (at least until Hail is updated to use 2.2).

2 Likes
7

Thanks for the observation.
First I encountered problems caused by Python3
and then I was trying to deal with this :frowning:
thanks for saving the time (Y)

8

I encountered this problem with Python3 / Spark 2.2.0 and the reason is that the SPARK_CLASSPATH environment variable doesn’t work with Spark 2.2.0 any more.
Thanks to Tim for giving me the solution: setting the following environment variable let me put the Hail jar in Spark path:
export PYSPARK_SUBMIT_ARGS="--conf spark.driver.extraClassPath=$JAR_PATH --conf spark.executor.extraClassPath=$JAR_PATH pyspark-shell"

1 Like
9

Hi, I am trying to install hail and follow the tutorial on a server running on CentOS Linux 7, but I get the following error when I run the example code on the document.

TypeError Traceback (most recent call last)
in
1 import hail as hl
----> 2 mt= hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
3 mt.count()

</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1471> in balding_nichols_model(n_populations, n_samples, n_variants, n_partitions, pop_dist, fst, af_dist, reference_genome, mixture)

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
582 @decorator
583 def wrapper(original_func, *args, **kwargs):
→ 584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
585 return original_func(*args, **kwargs)
586

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in check_all(f, args, kwargs, checks, is_method)
510 if i < len(args):
511 arg = args[i]
→ 512 args_.append(checker.check(arg, name, arg_name))
513 # passed as keyword
514 else:

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in check(self, x, caller, param)
54 for tc in self.checkers:
55 try:
—> 56 return tc.check(x, caller, param)
57 except TypecheckFailure:
58 pass

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in check(self, x, caller, param)
301 for tc, f in self.fs:
302 try:
→ 303 return f(tc.check(x, caller, param))
304 except TypecheckFailure:
305 pass

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/genetics/reference_genome.py in (x)
8
9 rg_type = lazy()
—> 10 reference_genome_type = oneof(transformed((str, lambda x: hl.get_reference(x))), rg_type)
11
12

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/context.py in get_reference(name)
356 :class:.ReferenceGenome
357 “”"
→ 358 Env.hc()
359 if name == ‘default’:
360 return default_reference()

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/utils/java.py in hc()
61 import sys
62 sys.stderr.write(“Initializing Spark and Hail with default parameters…\n”)
—> 63 init()
64 assert Env._hc is not None
65 return Env._hc

</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1189> in init(sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, _optimizer_iterations, _backend)

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
583 def wrapper(original_func, *args, **kwargs):
584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
→ 585 return original_func(*args, **kwargs)
586
587 return wrapper

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/context.py in init(sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, _optimizer_iterations, _backend)
262 min_block_size, branching_factor, tmp_dir,
263 default_reference, idempotent, global_seed,
→ 264 _optimizer_iterations,_backend)
265
266

</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1187> in init(self, sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, optimizer_iterations, _backend)

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
583 def wrapper(original_func, *args, **kwargs):
584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
→ 585 return original_func(*args, **kwargs)
586
587 return wrapper

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/context.py in init(self, sc, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmp_dir, default_reference, idempotent, global_seed, optimizer_iterations, _backend)
97 self._jhc = self._hail.HailContext.apply(
98 jsc, app_name, joption(master), local, log, True, append,
—> 99 min_block_size, branching_factor, tmp_dir, optimizer_iterations)
100
101 self._jsc = self._jhc.sc()

TypeError: ‘JavaPackage’ object is not callable

following is what I’ve added to the .bash_profile

export SPARK_HOME=/home/him/spark-2.4.0-bin-hadoop2.7/
export HAIL_HOME=/home/him/hail/
export PYTHONPATH=“${PYTHONPATH:+$PYTHONPATH:}$HAIL_HOME/build/distributions/hail-python.zip”
export PYTHONPATH=“$PYTHONPATH:$SPARK_HOME/python”
export PYTHONPATH=“$PYTHONPATH:$SPARK_HOME/python/lib/py4j-*-src.zip”
export SPARK_CLASSPATH=$HAIL_HOME/build/libs/hail-all-spark.jar #added this due to jar calling error
#PYSPARK_SUBMIT_ARGS is used by ipython and jupyter
export PYSPARK_SUBMIT_ARGS="
–jars $HAIL_HOME/build/libs/hail-all-spark.jar
–conf spark.driver.extraClassPath="$HAIL_HOME/build/libs/hail-all-spark.jar"
–conf spark.executor.extraClassPath=$HAIL_HOME/build/libs/hail-all-spark.jar
–conf spark.serializer=org.apache.spark.serializer.KryoSerializer
–conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator
pyspark-shell"

10

Hi @him26

The Hail log contains the original error, what you see above is simply saying there is no Java process running. Can you please attach the hail log file?

Can you also include the command line you used to compile Hail?

11

Thank you for the reply.
I did not do anything special. I just followed the installation steps in quick start section of the document with slight mod on installing jupyter before the hail install.

conda create - n hail python==3.6
pip install jupyter
pip install hail

After this command I run is following. Import runs find. but the second line throws an error.

import hail as hl
mt= hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
mt.count()

looking at the lines added to the .bash_profile I do not have a directory hail directory on my machine.
Where do I get this from?

Where do I find the hail log file from?
Thank you

12

While waiting for response I reinstalled with pip hail and installed java-jdk 8.0.92 with conda and ran the following command

import hail as hl
hl.init()

2019-08-06 15:52:54 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
2019-08-06 15:52:54 WARN DependencyUtils:66 - Local jar /build/libs/hail-all-spark.jar does not exist, skipping.
Traceback (most recent call last):
File “”, line 1, in
File “</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1064>”, line 2, in init
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/hail/context.py”, line 264, in init
_optimizer_iterations,_backend)
File “</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1062>”, line 2, in init
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py”, line 585, in wrapper
return original_func(*args, **kwargs)
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/hail/context.py”, line 99, in init
min_block_size, branching_factor, tmp_dir, optimizer_iterations)
TypeError: ‘JavaPackage’ object is not callable

before reinstalling hail I deleted a folder called hail (which I don’t remember where I got it from) and it looks like now my system does not know where hail is…:frowning:

Any suggestions on what I can try ? thank you

13

Hmm. Something to try: delete all the environment variables you set and run the following:

HAIL_COMPILE_NATIVES=1 make -C hail/ install

This should create and install a local wheel (pip distribution)

14

@him26, to be clear, Tim and I suspect your machine’s CPU is missing features on which the pip-installed-version of Hail depends. Can you share the output of cat /proc/cpuinfo?

Tim’s suggestion to download the hail repository and compile from source is your best bet.

15

@danking Here is the output you asked for it has ~1400 lines and below is first 25 lines. After which is repeat of the same.

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
stepping : 1
microcode : 0xb00002a
cpu MHz : 1200.195
cache size : 35840 KB
physical id : 0
siblings : 28
core id : 0
cpu cores : 14
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 20
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts spec_ctrl intel_stibp
bogomips : 3991.39
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

will try Tim’s suggestion and report back. Thank you.

I downloaded hail 0.2.19 release version, erased all environment variables and I ran Tim’s suggested commend and the resulting output is as below

fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
make: Entering directory `/home/him/hail/hail’
echo ‘[Build Metadata]’ > src/main/resources/build-info.properties
echo ‘user=him’ >> src/main/resources/build-info.properties
echo ‘revision=’ >> src/main/resources/build-info.properties
echo ‘branch=’ >> src/main/resources/build-info.properties
echo ‘date=2019-08-07T00:59:52Z’ >> src/main/resources/build-info.properties
echo ‘url=’ >> src/main/resources/build-info.properties
echo ‘sparkVersion=2.4.0’ >> src/main/resources/build-info.properties
echo ‘hailPipVersion=0.2.19’ >> src/main/resources/build-info.properties
echo 0.2.19- > python/hail/hail_version
echo 0.2.19 > python/hail/hail_pip_version
cp -f python/hail/hail_version python/hailtop/hail_version
cp -f python/hail/hail_pip_version python/hailtop/hail_pip_version
./gradlew releaseJar
Downloading (edited out due to user limit) gradle-5.4.1-bin.zip
…
Unzipping /home/him/.gradle/wrapper/dists/gradle-5.4.1-bin/e75iq110yv9r9wt1a6619x2xm/gradle-5.4.1-bin.zip to /home/him/.gradle/wrapper/dists/gradle-5.4.1-bin/e75iq110yv9r9wt1a6619x2xm
Set executable permissions for: /home/him/.gradle/wrapper/dists/gradle-5.4.1-bin/e75iq110yv9r9wt1a6619x2xm/gradle-5.4.1/bin/gradle

Welcome to Gradle 5.4.1!

Here are the highlights of this release:

  • Run builds with JDK12
  • New API for Incremental Tasks
  • Updates to native projects, including Swift 5 support

For more details see (edited out due to user limit)

Starting a Gradle Daemon (subsequent builds will be faster)

Task :nativeLibPrebuilt FAILED
g++: error: unrecognized command line option ‘-std=c++14’
make[1]: Entering directory /home/him/hail/hail/src/main/c' g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Upcalls.cpp -MG -M -MF build/Upcalls.d -MT build/Upcalls.o g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux testutils/unit-tests.cpp -MG -M -MF build/testutils/unit-tests.d -MT build/testutils/unit-tests.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux test.cpp -MG -M -MF build/test.d -MT build/test.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Region_test.cpp -MG -M -MF build/Region_test.d -MT build/Region_test.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Region.cpp -MG -M -MF build/Region.d -MT build/Region.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux PartitionIterators.cpp -MG -M -MF build/PartitionIterators.d -MT build/PartitionIterators.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux ObjectArray.cpp -MG -M -MF build/ObjectArray.d -MT build/ObjectArray.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativeStatus.cpp -MG -M -MF build/NativeStatus.d -MT build/NativeStatus.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativePtr.cpp -MG -M -MF build/NativePtr.d -MT build/NativePtr.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativeModule.cpp -MG -M -MF build/NativeModule.d -MT build/NativeModule.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativeLongFunc.cpp -MG -M -MF build/NativeLongFunc.d -MT build/NativeLongFunc.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativeCodeSuite.cpp -MG -M -MF build/NativeCodeSuite.d -MT build/NativeCodeSuite.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux NativeBoot.cpp -MG -M -MF build/NativeBoot.d -MT build/NativeBoot.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Logging.cpp -MG -M -MF build/Logging.d -MT build/Logging.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux ibs.cpp -MG -M -MF build/ibs.d -MT build/ibs.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux FS.cpp -MG -M -MF build/FS.d -MT build/FS.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Encoder.cpp -MG -M -MF build/Encoder.d -MT build/Encoder.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux Decoder.cpp -MG -M -MF build/Decoder.d -MT build/Decoder.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux davies.cpp -MG -M -MF build/davies.d -MT build/davies.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux cache-tests.cpp -MG -M -MF build/cache-tests.d -MT build/cache-tests.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux ApproximateQuantiles_test.cpp -MG -M -MF build/ApproximateQuantiles_test.d -MT build/ApproximateQuantiles_test.o g++: error: unrecognized command line option ‘-std=c++14’ g++ -o build/NativeBoot.o -march=corei7-avx -O3 -std=c++14 -Ilibsimdpp-2.1 -Wall -Wextra -fPIC -ggdb -fno-strict-aliasing -I../resources/include -I/home/him/miniconda3/envs/hail/include -I/home/him/miniconda3/envs/hail/include/linux -MD -MF build/NativeBoot.d -MT build/NativeBoot.o -c NativeBoot.cpp g++: error: unrecognized command line option ‘-std=c++14’ make[1]: *** [build/NativeBoot.o] Error 1 make[1]: Leaving directory /home/him/hail/hail/src/main/c’

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ‘:nativeLibPrebuilt’.

Process ‘command ‘make’’ finished with non-zero exit value 2

  • Try:
    Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

  • Get more help at (edited out due to user limit)

BUILD FAILED in 1m 3s
1 actionable task: 1 executed
make: *** [shadowJar] Error 1
make: Leaving directory `/home/him/hail/hail’Preformatted text

16

Ok I think I understand and now. When you use pip to install hail, you should not set any of those environment variables. In fact, if any of them are set, Hail might not start correctly.

Your CPU has the flags I expect. Compiling from source might fix the problem but only because the environment variables are for the non-pip-installed version of Hail.

17

Erased all environment variable. uninstalled and reinstall hail.
Retrying the first tutorial commend. And I get the following some other error… :frowning:

Initializing Spark and Hail with default parameters…
Running on Apache Spark version 2.4.0
SparkUI available at (removed)
Welcome to
__ __ <>__
/ // /__ __/ /
/ __ / _ `/ / /
// //_,/// version 0.2.19-c6ec8b76eb26
LOGGING: writing to /home/him/DATA/hail-20190807-1044-0.2.19-c6ec8b76eb26.log
2019-08-07 10:44:32 Hail: INFO: balding_nichols_model: generating genotypes for 3 populations, 50 samples, and 100 variants…
ERROR:root:Exception while sending command.

Traceback (most recent call last):
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/py4j/java_gateway.py”, line 1159, in send_command
raise Py4JNetworkError(“Answer from Java side is empty”)
py4j.protocol.Py4JNetworkError: Answer from Java side is empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/py4j/java_gateway.py”, line 985, in send_command
response = connection.send_command(command)
File “/home/him/miniconda3/envs/hail/lib/python3.6/site-packages/py4j/java_gateway.py”, line 1164, in send_command
“Error while receiving”, e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving

Py4JError Traceback (most recent call last)
in
1 mt= hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
----> 2 mt.count()

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/matrixtable.py in count(self)
2418 Number of rows, number of cols.
2419 “”"
→ 2420 return (self.count_rows(), self.count_cols())
2421
2422 @typecheck_method(output=str,

</home/him/miniconda3/envs/hail/lib/python3.6/site-packages/decorator.py:decorator-gen-1127> in count_rows(self, _localize)

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
583 def wrapper(original_func, *args, **kwargs):
584 args, kwargs = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
→ 585 return original_func(*args, **kwargs)
586
587 return wrapper

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/matrixtable.py in count_rows(self, _localize)
2373 ir = TableCount(MatrixRowsTable(self._mir))
2374 if _localize:
→ 2375 return Env.backend().execute(ir)
2376 else:
2377 return construct_expr(ir, hl.tint64)

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/backend/backend.py in execute(self, ir, timed)
106
107 def execute(self, ir, timed=False):
→ 108 result = json.loads(Env.hc()._jhc.backend().executeJSON(self._to_java_ir(ir)))
109 value = ir.typ._from_json(result[‘value’])
110 timings = result[‘timings’]

~/miniconda3/envs/hail/lib/python3.6/site-packages/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
→ 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

~/miniconda3/envs/hail/lib/python3.6/site-packages/hail/utils/java.py in deco(*args, **kwargs)
207 import pyspark
208 try:
→ 209 return f(*args, **kwargs)
210 except py4j.protocol.Py4JJavaError as e:
211 s = e.java_exception.toString()

~/miniconda3/envs/hail/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334 raise Py4JError(
335 “An error occurred while calling {0}{1}{2}”.
→ 336 format(target_id, “.”, name))
337 else:
338 type = answer[1]

Py4JError: An error occurred while calling o50.executeJSON

18

Can you share this log file?

The true source of the error should be present there.

19

Here is the link to the log file:
hail.log

20

The log indicates that Spark was told to shutdown. Sometimes this is caused by an out of memory error. How much RAM does this machine have? Can you try increasing the memory available to Spark with export PYSPARK_SUBMIT_ARGS="--driver-memory 8g pyspark-shell"?