I am using Hail to do QC and exporting phenotype data into VCF file, using the following scripts
101 print(“writing out whole vcf with GT only”)
102 vdsGT = vds.select_entries(vds.GT)
103 hl.export_vcf(vdsGT, qced_vcf_gt)
Outputing above VCF has no problem for many small chromosmes chr22~chr18 (file size approximate 50 GB)
but when exporting following VCF I always have issue, and had never been successful, wondering if there is any way to resolve this issue?
105 print(“writing out whole vcf with GT, AD, PL…”)
106 vdsTHREE = vds.select_entries(vds.GT, vds.AD, vds.PL)
107 hl.export_vcf(vdsTHREE, qced_vcf_three)
We need to know more information about this issue. Are you getting an error? If so, what’s the full stack trace?
There has been no error message, and the program seems to be stopped at some point, where the CPU usage (2 nodes) drops from 60% (when exporting GT-only VCF)to less 0.1%, but no error message shows up, waited for more than 7 days, still no output of GTADPL- VCF
Where can I get the what’s the full stack trace?
On the terminal the program shows in the process of running, but never produced the results.
I imagine that the VCF you’re trying to export is very large. When you use export_vcf(..., parallel=None) (which is the default argument), then you generate a single VCF file. The way Hail generates this file is by writing out chunks of the VCF in parallel, then doing a single-threaded concatenation step. We do not recommend doing this for large VCFs, especially since object stores like Google Cloud Storage have a maximum object size that caps the size of any single file.
I highly recommend using one of the parallel output modes.If you really really need to generate just one file, you should use the new parallel='composable' mode and use gsutil compose to concatenate chunks. We probably can’t offer detailed walkthroughs on how to do the compose step, though.