# Regression with multiple phenotypes with varying degrees of missingness **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [March 20, 2020, 5:50pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347 "2020-03-20T17:50:35Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Mark](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Mark](https://discuss.hail.is/u/Mark)\ **Post date:** [March 20, 2020, 5:50pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/1 "2020-03-20T17:50:35Z")
A basic coding question. I’m trying to run a phewas with hundreds of phenotypes. My plan is to use linear\_regression\_rows. Some of the phenotypes have many missing values; others do not. Thus, I want my outcome variable to be a list of lists, not a single list, so that each phenotype is treated independently. If I had three phenotypes, I’d probably write the code as follows: ```auto phewas = hl.linear_regression_rows(y=[[mt.pheno.A],[mt.pheno.B],[mt.pheno.C]], x=hl.gp_dosage(mt.GP),covariates=[1.0,mt.pheno.age,mt.pheno.sex]) ``` With hundreds of phenotypes, however, it is difficult to manually list them all out. How can I efficiently create a list of lists that I can use as the y parameter in linear\_regression\_rows? Thanks! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [March 20, 2020, 6:00pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/2 "2020-03-20T18:00:32Z")
First, let me clarify that if two phenotypes appear in the same inner list, e.g. A and B in `[[A, B], [C]]`, they are treated as independent regressions (the same as if we had written `[[A],[B],[C]]`), but are computed in a more computationally efficient way. The only requirement for appearing in the same inner list is a shared missingness pattern. Essentially, this expression should evaluate to true: ```auto hl.is_defined(mt.pheno.A).collect() == hl.is_defined(mt.pheno.B).collect() ``` * * * If every phenotype has a different missingness pattern, you can easily produce a list of independent lists with this python expression: ```auto [[mt.pheno[x]] for x in mt.pheno] ``` This works because any “struct” (aka nested) annotation in Hail can be used as if it is a list of its children: ```auto In [11]: mt = hl.balding_nichols_model(3,10,10) ...: mt.bn.describe() 2020-03-20 13:59:59 Hail: INFO: balding_nichols_model: generating genotypes for 3 populations, 10 samples, and 10 variants... -------------------------------------------------------- Type: struct { n_populations: int32, n_samples: int32, n_variants: int32, n_partitions: int32, pop_dist: array, fst: array, mixture: bool } -------------------------------------------------------- Source: Index: [] -------------------------------------------------------- In [12]: list(mt.bn) Out[12]: ['n_populations', 'n_samples', 'n_variants', 'n_partitions', 'pop_dist', 'fst', 'mixture'] ``` ---
**Author:** ![Mark](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Mark](https://discuss.hail.is/u/Mark)\ **Post date:** [March 20, 2020, 6:04pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/3 "2020-03-20T18:04:37Z")
Thanks, Dan. Each phenotype has a unique pattern of missingness, so the python expression you mention is just what I was trying to come up with. ---
**Author:** ![Akhil\_Pampana](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/akhil_pampana/32/295_2.png) [@Akhil\_Pampana](https://discuss.hail.is/u/Akhil_Pampana)\ **Post date:** [March 21, 2020, 4:46am UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/4 "2020-03-21T04:46:19Z")
I dont know i can buzz in or not but when u run linear regression one at a time. it will consider missingness for each phenotype seperately. say if a phenotype have n values out of m samples, while running regression it will filter samples which doesnt have values in phenotypes and do regression based on n values. Thats what i observer for my analysis. How many penotypes are you running on? @Mark . ps: sorry if i buzz with wrong answer. U can use R to get all missing values for n number of phenotypes columns. ---
**Author:** ![Mark](https://avatars.discourse-cdn.com/v4/letter/m/e9c0ed/32.png) [@Mark](https://discuss.hail.is/u/Mark)\ **Post date:** [March 24, 2020, 2:42pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/5 "2020-03-24T14:42:20Z")
Here’s what I tried: > phens = [[mt.pheno] for x in mt.pheno] > phewas = hl.linear\_regression\_rows(y=phens, x=hl.gp\_dosage(mt.GP),covariates=[1.0,mt.covariates.age,mt.covariates.sex]) Unfortunately, I got the following error: > ERROR:root:Exception while sending command. > Traceback (most recent call last): > File “/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py”, line 1159, in send\_command > raise Py4JNetworkError(“Answer from Java side is empty”) > py4j.protocol.Py4JNetworkError: Answer from Java side is empty > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File “/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py”, line 985, in send\_command > response = connection.send\_command(command) > File “/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py”, line 1164, in send\_command > “Error while receiving”, e, proto.ERROR\_ON\_RECEIVE) > py4j.protocol.Py4JNetworkError: Error while receiving It also said: > Py4JError: An error occurred while calling o102.pyPersist. Any idea why this might be happening? I don’t know if it’s relevant, but all my column names in the phenotype matrix are just numbers. Could that be causing a problem? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [April 27, 2020, 1:37pm UTC](https://discuss.hail.is/t/regression-with-multiple-phenotypes-with-varying-degrees-of-missingness/1347/6 "2020-04-27T13:37:20Z")
Hey @Mark, Sorry for the delay, I had some bad notifications settings. This usually indicates that you don’t have BLAS libraries appropriately installed and configured. If you’re on Ubuntu or Debian, you might try: `apt-get install libopenblas liblapack3`.