Dataproc error: java.io.IOException: Failed to create local dir

If Spark 2.3.0 seems to fix things, then that does seem like a good solution, yeah.

Here’s the dependency versions for that:

You’ll also need to compile on a Linux VM so the native libraries are built properly.