The answer here appears to be that we didn’t set the executor memory high enough. N1-highmem-8 machines have 52GB of memory so we’ll need to take up more than half of that in order for YARN to pack only one executor per node. I set spark.executor.memory=40g and spark.executor.memoryOverhead=4000 and am re-running your workload. It appears to be working now.
1 Like