UKBiobank Research Analysis Platform (RAP) MatrixTable Write Issues

[tpoterba]

What is the runtime? I’m assuming this is running on a cluster?

I think the core issue here is that the file system you’re writing to / reading from is a local file system, not network-visible. The issue is similar to the one described here:

Cannot load a Hail Table in Terra notebook

The core problem is that you’re loading a file on a local file system (visible only to the cluster driver node) across a cluster of multiple machines. This is a bad error message (should probably say “file doesn’t exist”), but this is happening because if you don’t supply the file scheme (scheme://) then the default is used, which is HDFS for Dataproc clusters. There’s no folder in HDFS in the default user directory with the name training_pca.ht. You can read this file because the read_table …

The solution is that you should write to a file system that’s network visible – a cloud object store like Google Storage / S3 / Azure Blob Storage, or a network file system like Lustre/HDFS.