Importing multiple vcfs from multiple Terra buckets to Hail

Hail Batch & General Cloud
1

Hi,

I need to import multiple vcf files from multiple Terra buckets into Hail. I am using hl.import_vcf and specify the path to vcfs with a wildcard, it looks like this:
vcfs='gs://bucket-name/submissions/folder-name/ReblockGVCF/**/*.vcf.gz'

I get the error that “arguments refer to no files”. It seems that Hail treats the entry as a vector.

Overall, I’ve got a manifest with a list of vcf files and paths to them (in various buckets and various folders within those buckets) and I am looking for the best way to import them. For now, I’ve tried one bucket and put the path common to the files I need, replacing the different parts of the path with wild card(s). It did not work (see the error above). Any suggestions on how to address the error/ best way to import the files are very much appreciated! Thank you!

2

Hi @Tetyana_Zayats, thanks for reaching out. The issue here is that unlike gcloud storage ls or shell expansion, Hadoop Glob Patterns don’t expand ** into any number of path segments separated by /.

You’ll need to build the file list manually. My recommendation is to use gcloud storage ls with those patterns, saving the list to a file.

# remove the list so we don't keep appending to it, if something went
# wrong and it needs to be regenerated
rm filelist.txt
prefixes="gs://bucket-name0/...
gs://bucket-name1/..."
for path in $prefixes; do
  # use this funny quoting so the glob gets passed along
  # to gcloud and not the shell
  gcloud storage ls "$path"'ReblockGVCF/**/*.vcf.gz'  \
    >> filelist.txt
done

Then, in python:

with open('filelist.txt') as filelist:
    vcfs = [file.strip() for file in filelist]
3

Thank you! I will give it a go!

Because there are several buckets with files I need, can I generate one file list and then ask hail to copy those from different buckets? Or Do I need to do this for each bucket individually?

4

No, you should be able to list all the buckets and get everything into one file, then assuming your user/service accounts have appropriate access, handle everything without extra copies.

5

Thank you!
I’ll give it a go, although I’m not entirely sure I understand everything just yet. Hopefully, things will become clearer as I work through the suggested steps. I’ll follow up, if I run into any trouble.
Thank you for your help!