# Help for finding rare variants for 100 patients **URL:** \ **Category:** Hail Query & hailctl\ **Created:** [February 21, 2023, 11:25pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130 "2023-02-21T23:25:03Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [February 21, 2023, 11:25pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/1 "2023-02-21T23:25:03Z")
Hello team, Can you please help me for an efficient way to find common rare variants among 100 patients? I have 100 gVCF files (each file represents a patient). Currently, my workflow is below: 1. Import the gVCF file into a MatrixTable: mt1 = hl.import\_vcf(‘xxx.g.vcf.gz’, force\_bgz=True, reference\_genome=“GRCh38”) 2. Filtering some gene locations defined by intervals fmt = hl.filter\_intervals(mt1,intervals) 3. Add the annotation from gnomad to get the allele frequency data db = hl.experimental.DB(region=‘us’, cloud=‘aws’) ann\_fmt = db.annotate\_rows\_db(fmt, ‘gnomad\_genome\_sites’) 4. Perform further filtering by the allele frequency rare\_mt = ann\_fmt.filter\_rows(ann\_fmt.gnomad\_genome\_sites.freq[0].AF \< 0.001) 5. extract the locus and alleles key from rare\_mt 6. repeat 1-5 for every patient and find the intersection of the rare variants. Thanks for helping me on the Steps 3 and 4 above in the other topic that I posted a couple of weeks ago. I was able to do the above steps 1-5 for one patient, but it took a lot of time (about 30-40 mins on the spark cluster with 8 nodes). Instead of writing a loop to go through the above steps 1-5 for each gVCF file (each patient), is there an efficient way to accomplish the tasks above? Your help is highly appreciated. ---
**Author:** ![jsmadsen](https://avatars.discourse-cdn.com/v4/letter/j/46a35a/32.png) [@jsmadsen](https://discuss.hail.is/u/jsmadsen)\ **Post date:** [February 22, 2023, 8:16am UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/2 "2023-02-22T08:16:46Z")
I use `import_gvcfs` ([Hail | Import / Export](https://hail.is/docs/0.2/methods/impex.html#hail.methods.import_gvcfs)) which can take multiple gVCF files and a partitions argument to filter to specific regions while loading the data. ```python mts = hl.import_gvcfs( vcf_files, partitions=regions, reference_genome="GRCh38", array_elements_required=False, ) # union multiple matrixtables before returning them if len(mts) == 1: return mts[0] else: return hl.MatrixTable.union_rows(*mts) ``` I think checkpointing after this makes subsequent operations faster but someone can correct me on this. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [February 22, 2023, 9:59pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/3 "2023-02-22T21:59:23Z")
Thanks very much for the advice! I tried the import\_gvcfs function: mts = hl.import\_gvcfs( paths, partitions=intervals, reference\_genome=“GRCh38”, array\_elements\_required=False, ) I got the error below: FatalError: HailException: range bounds must be inclusive I checked the intervals: [Interval(start=Locus(contig=chr1, position=18182239, reference\_genome=GRCh38), end=Locus(contig=chr1, position=21299915, reference\_genome=GRCh38), includes\_start=True, includes\_end=False), Interval(start=Locus(contig=chr1, position=25196014, reference\_genome=GRCh38), end=Locus(contig=chr1, position=28282103, reference\_genome=GRCh38), includes\_start=True, includes\_end=False), …] I can see that for each interval the includes\_end=False. Is there some function or method available to change this to True? I created the intervals using the code below: intervals=[] for i in range(0,88): intervals.append(hl.parse\_locus\_interval(GenePosition.iloc[i,7],reference\_genome=‘GRCh38’)) where an example of GenePosition.iloc[1,7] is chr16:28162962-31170875. Thanks. ---
**Author:** ![jsmadsen](https://avatars.discourse-cdn.com/v4/letter/j/46a35a/32.png) [@jsmadsen](https://discuss.hail.is/u/jsmadsen)\ **Post date:** [February 23, 2023, 7:47am UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/4 "2023-02-23T07:47:47Z")
> The start locus must precede the end locus. The default bounds of the interval are left-inclusive and right-exclusive. To change this, add one of `[` or `(` at the beginning of the string for left-inclusive or left-exclusive respectively. Likewise, add one of `]` or `)` at the end of the string for right-inclusive or right-exclusive respectively. [https://hail.is/docs/0.2/functions/genetics.html#hail.expr.functions.parse\_locus\_interval](https://hail.is/docs/0.2/functions/genetics.html#hail.expr.functions.parse_locus_interval) ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [February 24, 2023, 2:50am UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/5 "2023-02-24T02:50:24Z")
Thanks for the tip of fixing the intervals. I got my intervals fixed and re-ran the import\_gvcfs function. I got another error this time as below: FatalError Traceback (most recent call last) in 3 partitions=intervals, 4 reference\_genome=“GRCh38”, ----\> 5 array\_elements\_required=False, 6 ) in import\_gvcfs(path, partitions, force, force\_bgz, call\_fields, entry\_float\_type, reference\_genome, contig\_recoding, array\_elements\_required, skip\_invalid\_loci, filter, find\_replace, \_external\_sample\_ids, \_external\_header) /opt/conda/lib/python3.6/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) 575 def wrapper(_original\_func, \*args, \*\*kwargs): 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) → 577 return _original\_func(\*args, \*\*kwargs_) 578 579 return wrapper /opt/conda/lib/python3.6/site-packages/hail/methods/impex.py in import\_gvcfs(path, partitions, force, force\_bgz, call\_fields, entry\_float\_type, reference\_genome, contig\_recoding, array\_elements\_required, skip\_invalid\_loci, filter, find\_replace, \_external\_sample\_ids, \_external\_header) 2447 find\_replace[1] if find\_replace is not None else None, 2448 \_external\_sample\_ids, → 2449 \_external\_header) 2450 vector\_ref = json.loads(vector\_ref\_s) 2451 jir\_vref = ir.JIRVectorReference(vector\_ref[‘vector\_ir\_id’], /cluster/spark/python/lib/py4j-0.10.7-src.zip/py4j/java\_gateway.py in **call** (self, \*args) 1255 answer = self.gateway\_client.send\_command(command) 1256 return\_value = get\_return\_value( → 1257 answer, self.gateway\_client, self.target\_id, self.name) 1258 1259 for temp\_arg in temp\_args: /opt/conda/lib/python3.6/site-packages/hail/backend/py4j\_backend.py in deco(\*args, \*\*kwargs) 29 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’ 30 ‘Hail version: %s\n’ —\> 31 ‘Error summary: %s’ % (deepest, full, hail. **version** , deepest), error\_id) from None 32 except pyspark.sql.utils.CapturedException as e: 33 raise FatalError(‘%s\n\nJava stack trace:\n%s\n’ FatalError: IllegalArgumentException: requirement failed Java stack trace: java.lang.IllegalArgumentException: requirement failed at scala.Predef$.require(Predef.scala:212) at is.hail.rvd.RVDPartitioner.(RVDPartitioner.scala:52) at is.hail.rvd.RVDPartitioner.(RVDPartitioner.scala:40) at is.hail.io.vcf.VCFsReader.(LoadVCF.scala:1888) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:536) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1$$anonfun$apply$11.apply(SparkBackend.scala:535) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1$$anonfun$apply$1.apply(ExecuteContext.scala:47) at is.hail.utils.package$.using(package.scala:638) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:47) at is.hail.backend.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:46) at is.hail.utils.package$.using(package.scala:638) at is.hail.annotations.RegionPool$.scoped(RegionPool.scala:17) at is.hail.backend.ExecuteContext$.scoped(ExecuteContext.scala:46) at is.hail.backend.spark.SparkBackend.withExecuteContext(SparkBackend.scala:275) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:535) at is.hail.backend.spark.SparkBackend$$anonfun$pyImportVCFs$1.apply(SparkBackend.scala:534) at is.hail.utils.ExecutionTimer$.time(ExecutionTimer.scala:52) at is.hail.utils.ExecutionTimer$.logTime(ExecutionTimer.scala:59) at is.hail.backend.spark.SparkBackend.pyImportVCFs(SparkBackend.scala:534) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) Hail version: 0.2.78-b17627756568 Error summary: IllegalArgumentException: requirement failed I don’t know what this error means and how to fix it. Your help is highly appreciated! ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [February 28, 2023, 8:15pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/6 "2023-02-28T20:15:35Z")
@shengwei66 Can you update to the latest version of Hail? That version of Hail is over a year old. Unfortunately the team has limited resources and cannot support old versions of Hail. The latest version is 0.2.109. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [March 1, 2023, 2:38am UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/7 "2023-03-01T02:38:01Z")
Thanks Dan for your reply. I am running the analysis on the DNAnexus platform, and I am not sure if the Hail version can be updated to the latest version. Maybe I will need to look for other approach of importing the gvcf files. How about the vds.combiner? Is this function available in Hail 0.2.78? Thanks. ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [March 1, 2023, 8:59pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/8 "2023-03-01T20:59:04Z")
The VDS Combiner uses the same subsystem. I’m quite confident you’re running into a known bug from late 2021 / early 2022 which was fixed on February 22, 2022 in [PR 11355](https://github.com/hail-is/hail/pull/11355). There’s no way to work around this, versions of Hail before 0.2.85 just had a bug in `hl.import_gvcfs`. I think the most valuable thing you can do is contact DNANexus. They need to hear directly from users the importance of regularly updating Hail. Hail is an evolving piece of software with new features, faster implementations, and bug fixes being released every week. FWIW, two of DNANexus’ competitors, Terra and the All of Us RWB, both regularly update Hail. Those platforms should generally be no more than a month out of date. ---
**Author:** ![shengwei66](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/shengwei66/32/897_2.png) [@shengwei66](https://discuss.hail.is/u/shengwei66)\ **Post date:** [March 1, 2023, 9:37pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/9 "2023-03-01T21:37:10Z")
Thanks Dan for your advice. I will contact DNANexus for requesting the update of Hail. ---
**Author:** ![Juan](https://avatars.discourse-cdn.com/v4/letter/j/a698b9/32.png) [@Juan](https://discuss.hail.is/u/Juan)\ **Post date:** [March 18, 2023, 11:32pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/10 "2023-03-18T23:32:17Z")
Hi, I’m getting a very similar error. I most say I just restarted using hail after a while and I had to upgrade hail. This is the current setup: Name: hail Version: 0.2.112 Name: pyspark Version: 3.3.2 Using hailctl to launch an google cloud instance, then after connecting to the notebook I get this error immediately after trying hail.ini() I’m guessing I need to upgrade something else to make it compatible again?? help is appreciated, I’m stuck! > **Trace** > > * * * > > Py4JJavaError Traceback (most recent call last) > in \() > ----\> 1 hl.init() > > in init(sc, app\_name, master, local, log, quiet, append, min\_block\_size, branching\_factor, tmp\_dir, default\_reference, idempotent, global\_seed, spark\_conf, skip\_logging\_configuration, local\_tmpdir, \_optimizer\_iterations, backend, driver\_cores, driver\_memory, worker\_cores, worker\_memory, gcs\_requester\_pays\_configuration, regions) > > /opt/conda/miniconda3/lib/python3.10/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) > 575 def wrapper(_original\_func, \*args, \*\*kwargs): > 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) > → 577 return _original\_func(\*args, \*\*kwargs_) > 578 > 579 return wrapper > > /opt/conda/miniconda3/lib/python3.10/site-packages/hail/context.py in init(sc, app\_name, master, local, log, quiet, append, min\_block\_size, branching\_factor, tmp\_dir, default\_reference, idempotent, global\_seed, spark\_conf, skip\_logging\_configuration, local\_tmpdir, \_optimizer\_iterations, backend, driver\_cores, driver\_memory, worker\_cores, worker\_memory, gcs\_requester\_pays\_configuration, regions) > 346 )) > 347 if backend == ‘spark’: > → 348 return init\_spark( > 349 sc=sc, > 350 app\_name=app\_name, > > in init\_spark(sc, app\_name, master, local, log, quiet, append, min\_block\_size, branching\_factor, tmp\_dir, default\_reference, idempotent, global\_seed, spark\_conf, skip\_logging\_configuration, local\_tmpdir, \_optimizer\_iterations, gcs\_requester\_pays\_configuration) > > /opt/conda/miniconda3/lib/python3.10/site-packages/hail/typecheck/check.py in wrapper(\_\_original\_func, \*args, \*\*kwargs) > 575 def wrapper(_original\_func, \*args, \*\*kwargs): > 576 args, kwargs_ = check\_all(\_\_original\_func, args, kwargs, checkers, is\_method=is\_method) > → 577 return _original\_func(\*args, \*\*kwargs_) > 578 > 579 return wrapper > > /opt/conda/miniconda3/lib/python3.10/site-packages/hail/context.py in init\_spark(sc, app\_name, master, local, log, quiet, append, min\_block\_size, branching\_factor, tmp\_dir, default\_reference, idempotent, global\_seed, spark\_conf, skip\_logging\_configuration, local\_tmpdir, \_optimizer\_iterations, gcs\_requester\_pays\_configuration) > 425 app\_name = app\_name or ‘Hail’ > 426 gcs\_requester\_pays\_project, gcs\_requester\_pays\_buckets = convert\_gcs\_requester\_pays\_configuration\_to\_hadoop\_conf\_style(gcs\_requester\_pays\_configuration) > → 427 backend = SparkBackend( > 428 idempotent, sc, spark\_conf, app\_name, master, local, log, > 429 quiet, append, min\_block\_size, branching\_factor, tmpdir, local\_tmpdir, > > /opt/conda/miniconda3/lib/python3.10/site-packages/hail/backend/spark\_backend.py in **init** (self, idempotent, sc, spark\_conf, app\_name, master, local, log, quiet, append, min\_block\_size, branching\_factor, tmpdir, local\_tmpdir, skip\_logging\_configuration, optimizer\_iterations, gcs\_requester\_pays\_project, gcs\_requester\_pays\_buckets) > 182 self.\_jbackend, log, True, append, branching\_factor, skip\_logging\_configuration, optimizer\_iterations) > 183 else: > → 184 self.\_jbackend = hail\_package.backend.spark.SparkBackend.apply( > 185 jsc, app\_name, master, local, True, min\_block\_size, tmpdir, local\_tmpdir, > 186 gcs\_requester\_pays\_project, gcs\_requester\_pays\_buckets) > > /usr/lib/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java\_gateway.py in **call** (self, \*args) > 1319 > 1320 answer = self.gateway\_client.send\_command(command) > → 1321 return\_value = get\_return\_value( > 1322 answer, self.gateway\_client, self.target\_id, self.name) > 1323 > > /usr/lib/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py in get\_return\_value(answer, gateway\_client, target\_id, name) > 324 value = OUTPUT\_CONVERTER[type](answer[2:], gateway\_client) > 325 if answer[1] == REFERENCE\_TYPE: > → 326 raise Py4JJavaError( > 327 “An error occurred while calling {0}{1}{2}.\n”. > 328 format(target\_id, “.”, name), value) > > Py4JJavaError: An error occurred while calling z:is.hail.backend.spark.SparkBackend.apply. > : java.lang.IllegalArgumentException: requirement failed > at scala.Predef$.require(Predef.scala:268) > at is.hail.backend.spark.SparkBackend$.apply(SparkBackend.scala:227) > at is.hail.backend.spark.SparkBackend.apply(SparkBackend.scala) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) > at py4j.Gateway.invoke(Gateway.java:282) > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) > at py4j.commands.CallCommand.execute(CallCommand.java:79) > at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) > at py4j.ClientServerConnection.run(ClientServerConnection.java:106) > at java.base/java.lang.Thread.run(Thread.java:829) ---
**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\ **Post date:** [March 20, 2023, 11:50pm UTC](https://discuss.hail.is/t/help-for-finding-rare-variants-for-100-patients/3130/11 "2023-03-20T23:50:20Z")
I’m not sure what you’re doing exactly, but this error means spark is already running. You probably need to do `hl.init(sc=spark.sparkContext())` or something like that.