# Missing value and logistic regression **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [September 19, 2020, 6:15pm UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670 "2020-09-19T18:15:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Stephen](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Stephen](https://discuss.hail.is/u/Stephen)\ **Post date:** [September 19, 2020, 6:15pm UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/1 "2020-09-19T18:15:05Z")
Hello! I’m trying to use Hail to run a logistic regression GWAS on a case-control(0 vs 1) phenotype. Here are my codes: > import hail as hl > hl.init() > mt = hl.import\_plink(bed=‘chr22.bed’,bim=‘chr22.bim’,fam=‘chr22.fam’,missing=‘.’,quant\_pheno=False) > covar = (hl.import\_table(‘pheno\_covariate.txt’,types={‘IID’:hl.tstr,‘mypheno’:hl.tfloat64},impute=True,missing=‘.’).key\_by(‘IID’)) > mt = mt.annotate\_cols(covar=covar[mt.s]) > mt\_logistic = hl.logistic\_regression\_rows(test=‘wald’,y=mt.covar.mypheno,x=mt.GT.n\_alt\_alleles(), > covariates=[1,mt.covar.IsMale,mt.covar.PC1,mt.covar.PC2,mt.covar.PC3,mt.covar.PC4,mt.covar.PC5,mt.covar.PC6,mt.covar.PC7,mt.covar.PC8,mt.covar.PC9,mt.covar.PC10]) > mt\_logistic.export(‘chr22\_hail.txt’) I have two questions: 1. How will Hail deal with the missing value in import\_table() function? Will the missing value be deleted or imputed in some way? 2. When I ran this code, I got an error “OutOfMemoryError: GC overhead limit exceed”. Are there any mistakes in my code? Thanks so much! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [September 21, 2020, 12:57pm UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/2 "2020-09-21T12:57:56Z")
Hi @Stephen, You probably need to [increase the memory available to Spark](https://discuss.hail.is/t/how-do-i-increase-the-memory-or-ram-available-to-the-jvm-when-i-start-hail-through-python/133/2), a library Hail uses. Hail treats missing data as missing. If you execute, ```auto covar.filter(hl.is_missing(cover.mypheno)).show() ``` you’ll see the missing data represented by a special value `NA`. If you are curious how `logistic_regression_rows` deals with missing y-values, check out the docs for [`logistic_regression_rows`](https://hail.is/docs/0.2/methods/stats.html#hail.methods.logistic_regression_rows), specifically the warning box. ---
**Author:** ![Stephen](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Stephen](https://discuss.hail.is/u/Stephen)\ **Post date:** [September 27, 2020, 5:57am UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/3 "2020-09-27T05:57:39Z")
Hi @danking , Thank you for your reply. If I use Hail by runing .py file on a linux server, where should I add this command? Is it right to add it to the first line of my .py file? > export PYSPARK\_SUBMIT\_ARGS=“–driver-memory 8g --executor-memory 8g pyspark-shell” ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [September 28, 2020, 1:09am UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/4 "2020-09-28T01:09:47Z")
That code sets an environment variable in a shell like bash or zsh. You have to run that l, in your shell, before you run Python or pyspark or spark-submit. ---
**Author:** ![Stephen](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Stephen](https://discuss.hail.is/u/Stephen)\ **Post date:** [October 2, 2020, 4:31am UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/5 "2020-10-02T04:31:23Z")
Hi @danking, Thank you for your reply. Now I have another error: > Traceback (most recent call last): > File “/z/Comp/logi/1.py”, line 3, in > hl.init() > File “”, line 2, in init > File “/ua/zwang2547/.local/lib/python3.6/site-packages/hail/typecheck/check.py”, line 614, in wrapper > return _original\_func(\*args, \*\*kwargs_) > File “/ua/zwang2547/.local/lib/python3.6/site-packages/hail/context.py”, line 231, in init > skip\_logging\_configuration, optimizer\_iterations) > File “/ua/zwang2547/.local/lib/python3.6/site-packages/hail/backend/spark\_backend.py”, line 165, in **init** > pyspark.SparkContext.\_ensure\_initialized(conf=conf) > File “/ua/zwang2547/.local/lib/python3.6/site-packages/pyspark/context.py”, line 316, in \_ensure\_initialized > SparkContext.\_gateway = gateway or launch\_gateway(conf) > File “/ua/zwang2547/.local/lib/python3.6/site-packages/pyspark/java\_gateway.py”, line 46, in launch\_gateway > return \_launch\_gateway(conf) > File “/ua/zwang2547/.local/lib/python3.6/site-packages/pyspark/java\_gateway.py”, line 108, in \_launch\_gateway > raise Exception(“Java gateway process exited before sending its port number”) > Exception: Java gateway process exited before sending its port number Do you know how to fix it? ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [October 2, 2020, 12:36pm UTC](https://discuss.hail.is/t/missing-value-and-logistic-regression/1670/6 "2020-10-02T12:36:32Z")
Look for a hail log file. There’s more detail there. This almost certainly means you have an error in PYSPARK\_SUBMIT\_ARGS. Make sure it’s exactly as described at the other post and make sure you have enough memory.