LD pruning error

Hail Query & hailctl
1

Hello

I’m running LD-prune step on a (344031, 37702) matrix, on a cluster with 10 n1-highmem-8 worker nodes. I get an error: Executor heartbeat timed out after 137902 ms.
The following parameters are set in my spark-defaults conf file:
spark.executor.memory=10117m
spark.executor.memoryOverhead=15175m

I was able to run a smaller dataset (~90k, 20k) individuals, which took about an hour to run. Although the command succeeded in the spark UI I could see jobs failing with the same error message

What would I need to do here?

Thank you very much

2

Hey @sk4919 !

Can you try adding block_size=2048 to your call to ld_prune?

3

Thank you, it’s working now.

What does block size do exactly, and which functions is it relevant to?

Thanks