# Stuck at the last steps of a stage
**URL:** https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154
**Category:** Hail Query & hailctl
**Created:** [October 31, 2019, 12:55pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154 "2019-10-31T12:55:38Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@knguyen142](https://discuss.hail.is/u/knguyen142)
#### Post date: [October 31, 2019, 12:55pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/1 "2019-10-31T12:55:38Z")
Hi,
We upgraded to the latest (0.2.26) because the gnomad v3 table is only compatible with 0.2.24 and above. We’re running into issues where it gets stuck at one of the stages `Stage 3: ....(9373 + 5) / 9378]`. This has been reproduced, and on 0.2.24 it stops at Stage 2 but right at the end again.
I looked at the logs and couldn’t find any Error or anything except lost workers, but I might not know what to look for.
Any help would be very appreciated.
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [October 31, 2019, 1:34pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/2 "2019-10-31T13:34:17Z")
if you look at the Spark UI, does it show 5 tasks remaining in progress?
Teh thing to do here is probably turn on Spark speculation.
---
### Author:  [@knguyen142](https://discuss.hail.is/u/knguyen142)
#### Post date: [October 31, 2019, 1:37pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/3 "2019-10-31T13:37:17Z")
I looked through the spark ui a bit but couldn’t really decipher it. Is turning on spark speculation just setting `spark.speculation` to true? I’m not sure if the task is just running slowly, we leave it on for 30 minutes even and invariably they just get stuck.
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [October 31, 2019, 1:42pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/4 "2019-10-31T13:42:24Z")
The stage / task view in the UI might help indicate which tasks are the problem, or if this is a UI bug and something else is to blame.
And yep, `spark.speculation=true` is the right thing
---
### Author:  [@knguyen142](https://discuss.hail.is/u/knguyen142)
#### Post date: [October 31, 2019, 2:04pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/5 "2019-10-31T14:04:34Z")
I’ll check that out soon, but here’s something interesting. I’m running the same thing but on 0.2.22 on gnomad v2 table and it seems to be progressing to stage 7 (not sure how the stages differ across versions). So it doesn’t look to be the VCF, is there anything else we can look into?
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [October 31, 2019, 3:52pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/6 "2019-10-31T15:52:13Z")
hmm, weird. What’s the pipeline?
---
### Author:  [@knguyen142](https://discuss.hail.is/u/knguyen142)
#### Post date: [October 31, 2019, 4:52pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/7 "2019-10-31T16:52:23Z")
It’s our seqr pipeline that joins the vcf variants with a bunch of annotations from a reference data set.
It looks like setting speculation to true allows it to make progress.
`If set to "true", performs speculative execution of tasks. This means if one or more tasks are running slowly in a stage, they will be re-launched.`
Does the slow task reach a bug or something and needs to be restarted? Why does that fix it?
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [October 31, 2019, 4:53pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/8 "2019-10-31T16:53:45Z")
This is something we’ve never been able to characterize. Performance should be reproducible, but clearly there are cases where it’s not.
Maybe a Spark executor gets in bad state thrashing memory, maybe something else…
---
### Author:  [@knguyen142](https://discuss.hail.is/u/knguyen142)
#### Post date: [October 31, 2019, 4:54pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/9 "2019-10-31T16:54:19Z")
Should that setting be put in the general hailctl or does it affect performance?
---
### Author:  [@tpoterba](https://discuss.hail.is/u/tpoterba)
#### Post date: [October 31, 2019, 4:55pm UTC](https://discuss.hail.is/t/stuck-at-the-last-steps-of-a-stage/1154/10 "2019-10-31T16:55:16Z")
I think we’re planning on putting that as a hailctl default, yeah.