Whenever I try to use PySpark within Python code:
from pyspark import SparkConf, SparkContext
conf = (SparkConf().set("spark.executor.memory", "5g").set("spark driver.memory", "5g"))
sc = SparkContext(conf= conf)
import hail as hl
def file2hailMTX(bed, bim, fam, vcf, paramSM, outputMTX):
hl.init(min_block_size=128)
if paramSM == 'vcf':
hl.import_vcf(vcf, force_bgz=True).write(outputMTX, overwrite=True)
else:
hl.import_plink(bed=bed, bim=bim, fam=fam).write(outputMTX, overwrite=True)
file2hailMTX(snakemake.input[0], snakemake.input[1], snakemake.input[2], snakemake.input[3], snakemake.params[0], snakemake.output[0])
I am facing the error:
Warning: Ignoring non-Spark config property: spark driver.memory
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/ani/opt/anaconda3/envs/snakemake/lib/python3.10/site-packages/pyspark/jars/spark-unsafe_2.12-3.1.3.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
22/03/14 12:28:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Traceback (most recent call last):
File "/Users/ani/PycharmProjects/HailStuff/.snakemake/scripts/tmp9oofbn4p.NewHailScript.py", line 17, in <module>
file2hailMTX(snakemake.input[0], snakemake.input[1], snakemake.input[2], snakemake.input[3], snakemake.params[0], snakemake.output[0])
File "/Users/ani/PycharmProjects/HailStuff/.snakemake/scripts/tmp9oofbn4p.NewHailScript.py", line 11, in file2hailMTX
hl.init(min_block_size=128)
File "<decorator-gen-1700>", line 2, in init
File "/Users/ani/opt/anaconda3/envs/snakemake/lib/python3.10/site-packages/hail/typecheck/check.py", line 577, in wrapper
return __original_func(*args_, **kwargs_)
File "/Users/ani/opt/anaconda3/envs/snakemake/lib/python3.10/site-packages/hail/context.py", line 290, in init
backend = SparkBackend(
File "/Users/ani/opt/anaconda3/envs/snakemake/lib/python3.10/site-packages/hail/backend/spark_backend.py", line 181, in __init__
self._jbackend = hail_package.backend.spark.SparkBackend.apply(
TypeError: 'JavaPackage' object is not callable
[Mon Mar 14 12:28:32 2022]
Error in rule vcf_or_plink_hail:
jobid: 4
output: MTX/mtx_from_chr4.mt
RuleException:
CalledProcessError in line 12 of /Users/ani/PycharmProjects/HailStuff/rules/vcf_or_plink.smk:
Command 'set -euo pipefail; /Users/ani/opt/anaconda3/envs/snakemake/bin/python3.10 /Users/ani/PycharmProjects/HailStuff/.snakemake/scripts/tmp9oofbn4p.NewHailScript.py' returned non-zero exit status 1.
File "/Users/ani/PycharmProjects/HailStuff/rules/vcf_or_plink.smk", line 12, in __rule_vcf_or_plink_hail
File "/Users/ani/opt/anaconda3/envs/snakemake/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /Users/ani/PycharmProjects/HailStuff/.snakemake/log/2022-03-14T122823.285810.snakemake.log
As far as I get it, the main issue’s:TypeError: 'JavaPackage' object is not callable I have looked through the other errors, but I guess my issue’s a bit different. Possibly I have done smth wrong in my Python code, but I have no idea what is the error actually.