ZipException: File does not conform to block gzip format

Hail Query & hailctl
1

I am running a VariantDatasetCombiner with Hail version:

Hail version: 0.2.137-1fc852d5db9a

I am combining approximately 1,120 GVCF files stored in Google Cloud Storage.

The combiner fails with the following error:

Error summary:
ZipException: File does not conform to block gzip format.

The stack trace indicates that the error occurs while Hail is reading a tabix-indexed VCF/GVCF through the BGZF reader, e.g.:

BGzipInputStream$BGzipHeader.<init>
BGzipInputStream.decompressNextBlock
BGzipInputStream.virtualSeek
BGzipLineReader.virtualSeek
TabixLineIterator.next
TabixReadVCFIterator

I performed several checks on the input GVCFs. The BGZF header of the files I checked is valid, and the standard 28-byte BGZF EOF marker is also present. I also checked some of the previously suspected files using bgzip -t, and they passed.

I also verified that the required .tbi indexes are present for files that were previously missing their indexes.

Interestingly, one GVCF that had previously caused the VDS Combiner to stall also passed:



bgzip -t: PASS
tabix index: PASS

The GVCFs are stored in GCS and the VDS Combiner is also running on Google Cloud.

My questions are:

  1. What could cause BGzipInputStream$BGzipHeader / virtualSeek to throw ZipException: File does not conform to block gzip format when the BGZF header and EOF marker are valid?
  2. Could this indicate a problem with a specific BGZF block in the middle of the file, rather than the beginning or end of the file?
  3. Could there be an issue related to random access / virtual offsets / Tabix indexing when reading BGZF files from GCS?
  4. Is there a recommended way to identify which GVCF and/or BGZF block is causing the error?
  5. Would you recommend any additional validation beyond bgzip -t and tabix to diagnose this issue?

The combiner is configured approximately as follows:

branch_factor = 50
gvcf_batch_size = 25
target_records = 24000