# Hail 0.2 help for warning message **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [March 14, 2019, 9:00pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904 "2019-03-14T21:00:58Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 14, 2019, 9:00pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/1 "2019-03-14T21:00:58Z")
Hi, I was trying to trouble shoot why the varaint\_qc and to\_pandas step takes very long time (I was trying to filter variants to 5 positions -\> filter samples to a list of 20K -\> variant qc -\> export). There’s a warning message I don’t quite get it: 2019-03-14 20:29:58 Hail: WARN: Found differences between requested and initialized parameters. Ignoring requested parameters. Param: tmpDir, Provided value: /tmp, Existing value: dbfs:/tmp/hail.MfvZ1EpfplvG Param: minBlockSize, Provided value: 1, Existing value: 0 2019-03-14 20:30:00 Hail: INFO: Number of BGEN files parsed: 1 2019-03-14 20:30:00 Hail: INFO: Number of samples in BGEN files: 487409 2019-03-14 20:30:00 Hail: INFO: Number of variants across all BGEN files: 5751712 2019-03-14 20:30:39 Hail: INFO: interval filter loaded 8 of 18000 partitions 2019-03-14 20:30:41 Hail: INFO: Coerced sorted dataset My code: vds\_result = hl.filter\_intervals(mt,target\_intervals\_parsed).cache() merged = vds\_result.filter\_cols(hl.literal(subject\_array).contains(vds\_result.s)) merged\_variant\_qc = hl.variant\_qc(merged) merged\_variant\_qc\_pd = merged\_variant\_qc.rows().to\_pandas() Thanks. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 18, 2019, 11:58pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/2 "2019-03-18T23:58:24Z")
Where are you running Hail? You can see this error message when `hl.init()` is called with the `idempotent=True` argument, with different parameters otherwise. How long is this taking? I’d expect it to take a minute or so. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 18, 2019, 11:59pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/3 "2019-03-18T23:59:25Z")
> [@trptyrphe](#): > > merged = vds\_result.filter\_cols(hl.literal(subject\_array).contains(vds\_result.s)) ah, wait: ```python hl.literal(subject_array).contains(vds_result.s)) ``` should probably be: ```python hl.literal(set(subject_array)).contains(vds_result.s)) ``` ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 19, 2019, 1:07pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/4 "2019-03-19T13:07:52Z")
I am running Hail using databricks (runtime HLS 5.3 beta). The filter\_cols step is fast, it’s the next step: variant\_qc took forever to complete (5 variants, 20K subjects). Thanks. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 19, 2019, 1:35pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/5 "2019-03-19T13:35:49Z")
What hail version are you using? `hl. __version__ ` ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 19, 2019, 1:37pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/6 "2019-03-19T13:37:15Z")
hail 0.2: [https://docs.databricks.com/applications/genomics/hail.html#hail-02](https://docs.databricks.com/applications/genomics/hail.html#hail-02) ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 19, 2019, 1:37pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/7 "2019-03-19T13:37:46Z")
I need the minor version – before we dig in at all, I want to make sure it’s recent. ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 19, 2019, 1:44pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/8 "2019-03-19T13:44:56Z")
0.2.5-44045f72f11c ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 19, 2019, 1:50pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/9 "2019-03-19T13:50:18Z")
Did you try my change to the literal? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 19, 2019, 1:50pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/10 "2019-03-19T13:50:38Z")
note that even if the filter\_cols line is fast, that could still be the problem – Hail is lazy and executes the entire pipeline together at the end. ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 19, 2019, 2:06pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/11 "2019-03-19T14:06:48Z")
The list array only contains unique elements. I understand it’s the variant\_qc step to execute, so does it meet the expectation to be slow (hours not complete) for 5 variants x 20K samples? I just need a feasible way to perform the variant qc and export the result. Thanks. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 19, 2019, 3:53pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/12 "2019-03-19T15:53:25Z")
It needs to be a set not for uniqueness, but for performance – `.contains` on an array is O(n), while .contains on a set is O(log(n)). See the warning here: [https://hail.is/docs/0.2/expressions.html#hail.expr.expressions.ArrayExpression.contains](https://hail.is/docs/0.2/expressions.html#hail.expr.expressions.ArrayExpression.contains) ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 21, 2019, 3:29pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/13 "2019-03-21T15:29:34Z")
Still very slow with the set(). And if I break down the steps, it is step (B) the bottleneck, is there more efficient way to look up the variant qc results than the rows().to\_pandas()? Thanks. A. merged\_variant\_qc = hl.variant\_qc(merged) B. merged\_variant\_qc\_pd = merged\_variant\_qc.rows().to\_pandas() ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 21, 2019, 3:31pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/14 "2019-03-21T15:31:13Z")
to\_pandas uses a Spark protocol that is known to be very slow. If you try `merged_variant_qc.write('...')` (and read and to\_pandas that result) then that’s the best measurement. Can you run again and give us the Hail log file? You don’t need to let it run to completion, just let it get to the slow part. ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 21, 2019, 3:33pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/15 "2019-03-21T15:33:35Z")
where can I find the hail log? ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 21, 2019, 3:34pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/16 "2019-03-21T15:34:39Z")
It should be echoed in initialization: ```auto In [1]: hl.init() Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Running on Apache Spark version 2.2.0 SparkUI available at http://10.1.0.166:4040 Welcome to ____ <>__ / /_/ / ____ / / / __ / _ `/ / / /_/ /_/\_,_/_/_/ version 0.2.11-e8bbc49d0ae2 LOGGING: writing to /Users/tpoterba/hail/hail/hail-20190321-1134-0.2.11-e8bbc49d0ae2.log ``` ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 21, 2019, 3:34pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/17 "2019-03-21T15:34:53Z")
oh, wait, you’re using Databricks, and I think they do some weird stuff related to setup ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 21, 2019, 3:36pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/18 "2019-03-21T15:36:00Z")
try this: ```auto In [3]: hl.utils.java.Env.hc()._log Out[3]: '/Users/tpoterba/hail/hail/hail-20190321-1134-0.2.11-e8bbc49d0ae2.log' `` ``` ---
**Author:** ![trptyrphe](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/trptyrphe/32/263_2.png) [@trptyrphe](https://discuss.hail.is/u/trptyrphe)\ **Post date:** [March 21, 2019, 3:45pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/19 "2019-03-21T15:45:36Z")
Actually both ways work for identifying log file path. Can I send you the log directly (it’s too large to copy paste). Thanks. ---
**Author:** ![tpoterba](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/tpoterba/32/61_2.png) [@tpoterba](https://discuss.hail.is/u/tpoterba)\ **Post date:** [March 21, 2019, 3:45pm UTC](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904/20 "2019-03-21T15:45:55Z")
Yes, please send to [hail-team@broadinstitute.org](mailto:hail-team@broadinstitute.org) [Next page](https://discuss.hail.is/t/hail-0-2-help-for-warning-message/904.md?page=2)