# General advice for dealing with large unordered vcf files **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [September 15, 2021, 11:12pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246 "2021-09-15T23:12:18Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 15, 2021, 11:12pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/1 "2021-09-15T23:12:18Z")
I have a fairly large vcf (430 MB compressed), and it contains unsorted variants. It was produced as part of a WGS analysis pipeline that I’m only vaguely familiar with. Not sure if this is relevant but it also contains many alternate contig ID’s such as: ##contig=\ ##contig=\ ##contig=\ ##contig=\ ##contig=\ ##contig=\ ##contig=\ ##contig=\ ##contig=\ So far, I’ve successfully imported this vcf into a matrix table object, but I cant seem to run any useful queries on it without crashing the memory. For example, when I try to display the first 5 lines [mt.show(5)], it gives me a long java stack trace error that ends with: > Error summary: IOException: No space left on device I eventually want to annotate this vcf with gnomAD popmax values, but my inability to even display the first 5 rows is not inspiring confidence. Can anyone give me some advice on how to handle this table without overloading my memory? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 15, 2021, 11:59pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/2 "2021-09-15T23:59:38Z")
We fixed a bug that may have led to unintended “ordering unsorted dataset with network shuffle” behavior – could you try rerunning with 0.2.76? > [@mrdata](#): > > So far, I’ve successfully imported this vcf into a matrix table object, but I cant seem to run any useful queries on it without crashing the memory The shuffle (distributed sort) happens on import, so all queries are gated by that. Once we make that work (or avoid it), downstream work should be pretty painless. ---
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 16, 2021, 4:57am UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/3 "2021-09-16T04:57:18Z")
Thanks for your response. I was able to upgrade the version to 0.2.76, but I still ran into the same error. At first, I was able to successfully run > mt.rows().show(5) > … > [it displayed the first 5 lines of my vcf beautifully] But then I ran this and it gave me the same error I was getting before: > x=mt.rows() > x.select(“info”).show(5) > … > [very] > [long] > [java] > [stacktrace] > Hail version: 0.2.76-d470e2fea3f0 > Error summary: IOException: No space left on device Each time I compiled, the output indicated that it’s still doing that shuffling behavior you mentioned. Is there any way to disable that? Are there any other ways to work around this issue? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [September 16, 2021, 4:40pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/4 "2021-09-16T16:40:44Z")
Hmm. Just to be clear your code is exactly: ```auto mt = hl.read_matrix_table(...) x = mt.rows() x.select('info').show(5) ``` with no other intervening operations on `mt`? You should only experience a shuffle if you perform order changing operations such as `key_by`, `order_by`, and `group_by` (and their column-wise and row-wise equivalents). I strongly suspect there is either another Hail bug or something in your code inadvertently triggered a shuffle. * * * You can’t disable shuffling. Hail’s data model fundamentally depends on ordering. We implement SQL-like joins using ordering, not hashing. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 16, 2021, 4:44pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/5 "2021-09-16T16:44:36Z")
The shuffle is happening on import\_vcf. The VCF isn’t sorted according to Hail. ---
**Author:** ![mrdata](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@mrdata](https://discuss.hail.is/u/mrdata)\ **Post date:** [September 16, 2021, 5:35pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/6 "2021-09-16T17:35:24Z")
> import hail as hl > hl.init(spark\_conf={‘spark.driver.memory’:‘200g’}) ## maybe this line doesn’t do what I think it does? > rg = hl.get\_reference(‘GRCh38’) > vcf = ‘/path/to/raw.vcf.gz’ > mt=hl.import\_vcf(vcf, reference\_genome=rg, force=True)#, force\_bgz? > ht=hl.read\_table(“gnomad.genomes.v3.1.1.sites.ht”) > ht=ht.filter(~hl.is\_missing(ht.popmax.AF)) #remove all popmax=NA > n=mt.annotate\_rows(popmax=ht[mt.locus, mt.alleles].popmax.AF) > n=n.filter\_rows(n.popmax \< 0.01, keep=True) > hl.export\_vcf(n, ‘path/to/wgsTester-filtered.vcf.gz’) > … > … > [long Java stack] > Error summary: IOException: No space left on device I found that spark\_conf option on a forum somewhere and threw that in, thinking it would expand the memory, but I’m not at all sure that’s whats happening. Am I on the right track with this? I was also considering trying to parallelize it somehow? Would repartitioning or increasing n\_partitions on the import have any effect? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 16, 2021, 5:37pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/7 "2021-09-16T17:37:42Z")
The “no space left on device” is a disk limitation, not memory – the VCF is sorted using an out-of-core (on-disk) sorting algorithm. This is slow and space-intensive. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [September 16, 2021, 5:38pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/8 "2021-09-16T17:38:06Z")
We need to figure out why the VCF isn’t sorted, I think. That’s the top priority. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [September 17, 2021, 2:31pm UTC](https://discuss.hail.is/t/general-advice-for-dealing-with-large-unordered-vcf-files/2246/9 "2021-09-17T14:31:42Z")
@mrdata , could you possibly post the first, say, 100 loci and variants from your VCF? Can you take a look at the rest of the VCF (e.g. using gzcat & less) and see if there’s some obvious (unordered) pattern to the loci and variants?