# Filter all variants which belong to a sample ID

**URL:** <https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215>\
**Category:** Hail Query & hailctl\
**Created:** [December 12, 2019, 10:49am UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215 "2019-12-12T10:49:11Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![toandd](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/toandd/32/337_2.png) [@toandd](https://discuss.hail.is/u/toandd)\
**Post date:** [December 12, 2019, 10:49am UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/1 "2019-12-12T10:49:11Z")

</div>

Hello Hail Dev team,

```
CHR POS REF ALT INFO FORMAT SAMPLE1 SAMPLE2 ... SAMPLEn
chr1	1100	C	T	AF=0.3,GQ=20,... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1101	G	T	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1102	A	T	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1103	C	G	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1104	C	T	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1105	C	T	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
chr1	1106	C	T	AF=...,GQ=... GT,AF,DP...	0/1,0.5,...	0/1,0.5,... 0/1,0.5,...
...

```

I want to filter all variants of a specific sample to create a structured data and export it to elasticsearch database. Data looks like in below example:

```
{
	"sample_ID": "SAMPLE1",
	"AF": 0.5,
	"GQ": 10,
	...
	variant_filter: [
		{
			"locus": {
			    "contig": "chr1",
			    "position": 1100
			  },
			  "alleles": [
			    "C",
			    "T"
			  ],
			"variant_class": "SNV",
			"consequences": ["intron_variant", ...],
			"population_allele_freq": 0,3,
			"population_genotype_quality": 20,
		},
		{
			"locus": {
			    "contig": "chr2",
			    "position": 1101
			  },
			  "alleles": [
			    "G",
			    "T"
			  ],
			"variant_class": "indel",
			"consequences": ["downstrean_gene_variant", ...],
			"population_allele_freq": 0,125,
			"population_genotype_quality": 50,
		},
		...	

	]
}

```

Does Hail support an easy way to do above work?

---

<div class="post-metadata">

**Author:** ![toandd](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/toandd/32/337_2.png) [@toandd](https://discuss.hail.is/u/toandd)\
**Post date:** [December 12, 2019, 10:52am UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/2 "2019-12-12T10:52:20Z")

</div>

I can filter all subjects which have a specific variant by using following lines:

```
    subject_id_list = hl.agg.filter(mt.GT.is_het(), hl.agg.collect(mt.s))
    mt = mt.annotate_rows(carrier=subject_id_list)

```

But I can not find a similar way to filter all variants of a subject

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [December 12, 2019, 12:42pm UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/3 "2019-12-12T12:42:31Z")

</div>

You can do the same thing with annotate\_cols. Don’t do this, though – you’ll blow memory limits.

It somewhat looks like you want the data transposed (sample-major) going into elasticsearch. Is that right?

---

<div class="post-metadata">

**Author:** ![toandd](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/toandd/32/337_2.png) [@toandd](https://discuss.hail.is/u/toandd)\
**Post date:** [December 12, 2019, 2:36pm UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/4 "2019-12-12T14:36:25Z")

</div>

I still can not figure out a way to filter all variants of a sample before annotate\_cols, could you suggest by some lines of code. I will test memory consumsion after that.

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [December 12, 2019, 6:42pm UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/5 "2019-12-12T18:42:20Z")

</div>

```python
variant_list = hl.agg.filter(mt.GT.is_het(), hl.agg.collect(mt.row_key))
mt = mt.annotate_cols(variants = variant_list)

```

---

<div class="post-metadata">

**Author:** ![toandd](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/toandd/32/337_2.png) [@toandd](https://discuss.hail.is/u/toandd)\
**Post date:** [December 13, 2019, 2:16am UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/6 "2019-12-13T02:16:19Z")

</div>

Thank you @tpoterba. As my understanding, This code will require spark to load all lines in vcf file, and consume lots of memory? Can hail transpose data when import vcf file or I have to write a tool to do this work before import data to hail?

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [December 13, 2019, 2:48pm UTC](https://discuss.hail.is/t/filter-all-variants-which-belong-to-a-sample-id/1215/7 "2019-12-13T14:48:37Z")

</div>

We currently do not have a transpose operation.