Hello,
version 0.2.130-bea04d9c79b5 within All of Us.
I have a piece of hail code that produces this warning. Everything still works fine, but I wanted to report it just incase it’s an issue. I realize that it says to report this to spark, but I have no idea where within the hail code the issue exists. Included below is the relevant hail code and log file. I did kill the run so you might see that at the end of the log file.
mt = hl.read_matrix_table(mt_gwas_ready_file)
mt = mt.annotate_cols(ancestry=ancestry_table[mt.s], phenotype=phenotype_table[mt.s])
mt = mt.naive_coalesce(1000)
pcs = [mt.ancestry.pca_features[i] for i in range(10)]
rest_covariates = [
mt.phenotype.x1,
mt.phenotype.x2,
mt.phenotype.x3,
mt.phenotype.x4,
]
covariates = [1] + pcs + rest_covariates
gwas_results = hl.linear_regression_rows(
y=mt.phenotype.y,
x=mt.n_alt_alleles,
covariates=covariates,
)
2024-07-09 14:22:22.016 Hail: INFO: linear_regression_rows: running on 903 samples for 1 response variable y,
with input variable x, and 19 additional covariates...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.util.SizeEstimator$ (file:/usr/lib/spark/jars/spark-core_2.12-3.3.0.jar) to field java.util.concurrent.locks.ReentrantReadWriteLock.readerLock
WARNING: Please consider reporting this to the maintainers of org.apache.spark.util.SizeEstimator$
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[Stage 3:> (0 + 8) / 1000]
hail-20240709-1421-0.2.130-bea04d9c79b5.log (847.3 KB)
-Andrew