Hi Akhil, I was hoping to respond today but need until tomorrow to post an update. Thanks for your patience!

# Code review of lmm for multi phenotypes

Hi Akhil,

It looks like you’re doing the right thing by computing the kinship once and reusing it across all phenotypes.

Are all these phenotypes on exactly the same samples? If so, after you’ve run LMM once with `pa_t_path`

set as well, use the approach documented in linear_mixed_regression_rows under the heading **Efficiently varying the response or set of fixed effects** in the full-rank case to run it all subsequent times. This will avoid recomputing `pa_t`

each time, which is the most intensive step by far, so you should see a huge speed-up on all subsequent runs.

Other questions once you’ve made this change:

For each loop:

How long does the fit step take? (should be very quick)

How much time is spent on linear_mixed_regression_rows step?

Do the loops each take the same amount of time or so they get substantially slower with time?

600 million is a LOT of variants. I’m confused how you could have that many after filtering at 5% minor allele frequency. Can you explain?

And can you test how long it takes to run a single (unmixed) linear_regression_rows for comparison so we know where the bottleneck is?

—

If it’s still slow after the major fix above, a smaller speed up would come from us upgrading our infrastructure to accommodate testing multiple phenotypes with LMM in one pass, similar to what we do for linear regression on UKBB. Knowing these numbers will help us evaluate how helpful that would be.

Aside: With so few samples, I suspect you are using more variants to estimate kinship than necessary. 50k should be plenty. If you’re using millions, you could downsample to get there. But if you already have a kinship matrix you’re happy with, this won’t speed you up since all the time is in the loop.

Thanks!

Hello Jon

Are all these phenotypes on exactly the same samples?

`Yes,all the phenotypes have same samples and i will try the method you suggested above with fewer phenotypes.`

Other questions once you’ve made this change:

For each loop:

How long does the fit step take? (should be very quick)

`i am thinking it was quick as i dint checked the time before`

How much time is spent on linear_mixed_regression_rows step?

`I think it took more time but will check after changing according to your suggestion.`

`I will respond back once i check these.`

Do the loops each take the same amount of time or so they get substantially slower with time?

`this is also one thing i will have a look into.`

600 million is a LOT of variants(its for initial dataset). I’m confused how you could have that many after filtering at 5% minor allele frequency. Can you explain?

`Subsetted version have less variants compared to initial set`

And can you test how long it takes to run a single (unmixed)

linear_regression_rows for comparison so we know where the bottleneck is?

`ok will have a look into it and check too`

Aside: With so few samples, I suspect you are using more variants to estimate kinship than necessary. 50k should be plenty. If you’re using millions, you could downsample to get there. But if you already have a kinship matrix you’re happy with, this won’t speed you up since all the time is in the loop.

`I am using only 200k variants for 2000 samples`

Thanks!

This is the staging gfile from before iteration. If u want to have a look into. I think it have 3 to 4 phenotypes from start of running and outputing

Hey @jon, i tried as you suggested and i am getting this error, eventhough there are no missing values.

```
list = [prune.pheno.SL003310,prune.pheno.SL004140,prune.pheno.SL004146]
list1 = ['SL003310','SL004140','SL004146']
x = [1.0,prune.pheno.male,prune.pheno.age]
y = [prune.pheno.SL004140,prune.pheno.SL004146]
model,p = hl.linear_mixed_model(y=list[0],x=[1.0,prune.pheno.male,prune.pheno.age],k=rrm,p_path='gs://jhs_data_topmed/lmm_phewas/'+list1[0]+'.bm')
for i in range(1,3):
name = list[i]
print(name)
out = list1[i]
print(out)
#model,p = hl.linear_mixed_model(y=name,x=[1.0,prune.pheno.male,prune.pheno.age],k=rrm,p_path='gs://jhs_data_topmed/lmm_phewas/'+out+'.bm')
print(time.time())
model1 = hl.stats.LinearMixedModel(p @ y, p @ x, s)
print(time.time())
model1.fit()
print(time.time())
result_ht = model1.fit_alternatives(p_path='gs://jhs_data_topmed/lmm_phewas/'+list1[0]+'.bm')
print(time.time())
model1.h_sq
print(time.time())
result_table = hl.linear_mixed_regression_rows(mt.GT.n_alt_alleles(), model1)
print(time.time())
result_table.export('gs://jhs_data_topmed/lmm_phewas/gwas.'+out+'.tsv.gz')
print(time.time())
```

Hi Akhil, when you run `linear_mixed_regression_rows`

the first time, you need to set the `pa_t_path`

to store this block matrix once and for all. Then to GWAS other phenotypes, you want to pass in `pa_t_path`

, not `p_path`

, to `fit_alternatives`

as in the documentation.

This may be what’s causing the error message far above, which is not about missing values; it’s a dimension mismatch for matrix multiplication. If you’re still getting this error, check the dimensions of `p`

, `y`

, and `x`

in your case with `.shape`

Oh got it. I thought i though p_path and pa_t_path are same. Thank you will change and check.

.shape u mean x.shape,y.shape,p.shape?

I tried and still got the same error as above.

Can you post your new code and tell me what shapes you’re getting?

You should be running `linear_mixed_regression_rows`

once, saving out `pa_t_path`

. You can then do the other phenotypes in a loop with `LinearMixedModel`

and `fit_alternatives`

. But before you do the rest in a loop, make sure that the latter works on the same phenotype, giving the same results as `linear_mixed_regression_rows`

but much faster because it doesn’t recompute the product of `p`

and `a_t`

.

The picture of the google cloud screen you posted is cutoff in a way that makes it impossible to read the output. Posting text is also generally better than posting pictures.

Oh I see, the point was just to show the shapes? Nevermind then.

hello @snashraf , i did similar thing what i did at top but i got some errors and its taking to long to run so i stopped running it.

Akhil, could you post text instead of screenshots? It means that the next time someone has this problem, they’ll be able to find this thread.

Sure I will keep that in mind. Actually i got answer for this so no problem. i am keeping link here for answer and thank you i will copy paste text form next time? https://hail.zulipchat.com/#narrow/stream/123010-Hail-0.2E2.20support/topic/lmm.20for.20multiple.20phenotypes