# Improve writing time for GWAS results

**URL:** <https://discuss.hail.is/t/improve-writing-time-for-gwas-results/1794>\
**Category:** Hail Query & hailctl\
**Created:** [November 18, 2020, 5:06pm UTC](https://discuss.hail.is/t/improve-writing-time-for-gwas-results/1794 "2020-11-18T17:06:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Liverpool](https://avatars.discourse-cdn.com/v4/letter/l/e99b99/32.png) [@Liverpool](https://discuss.hail.is/u/Liverpool)\
**Post date:** [November 18, 2020, 5:06pm UTC](https://discuss.hail.is/t/improve-writing-time-for-gwas-results/1794/1 "2020-11-18T17:06:42Z")

</div>

Hi, I’m running linear regression with about 100K samples and 90K variants. After GWAS, I will also export the results. The whole process took about 2 hours which seems to be too long. Below are my codes and may I kindly ask how can I improve it? Thanks a lot!

```auto
import hail as hl
hl.init(min_block_size=128)  

### prepare the inputs:
chr='22'  
Input='phenotype.txt'
y = 'Diabetes' # name of the phenotype in the text file

### import the covariates and phenotype info:
sample_info = hl.import_table(paths=Input, types={'IID':hl.tstr, y:hl.tint32}, impute=True).key_by('IID')
sample_info = sample_info.rename({y: 'phenotype'}) # rename the regression Y to be "phenotype"

### Import PLINK bfiles: 
mt = hl.import_plink(bed='genotype/chr'+chr+'.bed',
                     bim='genotype/chr'+chr+'.bim',
                     fam='genotype/chr'+chr+'.fam',
                     quant_pheno=True, missing='-9', 
                     reference_genome='GRCh37')

### add phenotypes to genotype matrix and keep only GWAS sample
mt = mt.annotate_cols(pheno = sample_info[mt.s]) # add phenotype and covariate info
mt = mt.filter_cols(~hl.is_nan(mt.pheno.phenotype)) # filter NA phenotypes

### variant QC
mt = hl.variant_qc(mt)  
mt = mt.filter_rows((mt.variant_qc.AF[1] >= 0.01)&
                    (mt.variant_qc.p_value_hwe >= 1e-6)&
                    (mt.variant_qc.call_rate >= 0.99))       

### run linear regressions
gwas = mt.annotate_rows(linreg = hl.agg.linreg(y = mt.pheno.phenotype, 
                                                x = [1, mt.GT.n_alt_alleles()])) 
gwas = gwas.rows().key_by()  
gwas = gwas.select(CHR = gwas.locus.contig, SNP = gwas.rsid, BP = gwas.locus.position, 
                A1 = gwas.alleles[1], A2 = gwas.alleles[0], EAF = gwas.variant_qc.AF[1],
                BETA = gwas.linreg.beta[1], SE = gwas.linreg.standard_error[1], 
                P = gwas.linreg.p_value[1], N = gwas.linreg.n)

### export result
# first write to Hail’s efficient and fast on-disk format, then read back in and convert to a text file
gwas.write('chr'+chr+'linear.ht') 
hl.read_table('chr'+chr+'linear.ht').export('chr'+chr+'linear.tsv.bgz', header=True)

```

---

<div class="post-metadata">

**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\
**Post date:** [November 20, 2020, 9:17pm UTC](https://discuss.hail.is/t/improve-writing-time-for-gwas-results/1794/2 "2020-11-20T21:17:25Z")

</div>

One possible code change is to use the [linear\_regression\_rows](https://hail.is/docs/0.2/methods/stats.html#hail.methods.linear_regression_rows) method instead of `hl.agg.linreg` – it is purpose-built for efficient GWAS execution, and we’re still working on improving the performance of the linreg aggregator.

---

<div class="post-metadata">

**Author:** ![Liverpool](https://avatars.discourse-cdn.com/v4/letter/l/e99b99/32.png) [@Liverpool](https://discuss.hail.is/u/Liverpool)\
**Post date:** [November 20, 2020, 9:32pm UTC](https://discuss.hail.is/t/improve-writing-time-for-gwas-results/1794/3 "2020-11-20T21:32:03Z")

</div>

thanks!