Using the export VCF function seems to yield an empty VCF file but the logs look good – any idea why or what to try next?
Export VCF empty output
My first guess is that your touch command made an empty one, and that the real one is written somewhere else that you don’t expect.
Try doing this again without doing the touch first and with a name of a file that doesn’t already exist. It’s possible that hail is just refusing to overwrite a file that already exists and not throwing an error message. If that’s the case, we should change it so it does throw an error message.
My guess here is that it’s actually writing to an HDFS filesystem - is this on Google Cloud?
@konradjk it is on Google Cloud – whats the best way to export in that case?
@johnc1231 A different failure when I don’t have a touch statement:
You’ll see it if you !hdfs dfs -ls - since you’re on google cloud, you could also just put it on google cloud storage directly by specifying a gs:// path like: hl.export_vcf(mt, 'gs://bucket/path/to/file.vcf.bgz') - it will (b)gzip it for free if you add that extension so that you can move around less data.
1 Like