Error summary: VCFParseError: unexpected end of line

Hail Query & hailctl
1

Hi,
I am trying to run a GWAS with a vcf file. I used the same script that had worked in the past for joining chromosomes in plink, then create the vcf file, which I can share if needed.

When I try to import the vcf file, the following error happens.
Any help is greatly appreciated.
Hail version 0.2
Python 3.7

    FatalError                                Traceback (most recent call last)
<ipython-input-1-392114ebb2a4> in <module>
      7 from bokeh.plotting import figure, output_file
      8 
----> 9 ds = hl.import_vcf('/Users/un/genome_chr_all_extracted.vcf', reference_genome='GRCh37').write('/Users/un/genome/MEGA_Chip_genome.mt', overwrite=True)
     10 mt = hl.read_matrix_table('genome/MEGA_Chip_genome.mt', types={'GQ':'int'})
     11 

</Users/un/.pyenv/versions/3.7.3/lib/python3.7/site-packages/decorator.py:decorator-gen-1204> in write(self, output, overwrite, stage_locally, _codec_spec, _partitions)

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hail/typecheck/check.py in wrapper(__original_func, *args, **kwargs)
    583     def wrapper(__original_func, *args, **kwargs):
    584         args_, kwargs_ = check_all(__original_func, args, kwargs, checkers, is_method=is_method)
--> 585         return __original_func(*args_, **kwargs_)
    586 
    587     return wrapper

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hail/matrixtable.py in write(self, output, overwrite, stage_locally, _codec_spec, _partitions)
   2527 
   2528         writer = MatrixNativeWriter(output, overwrite, stage_locally, _codec_spec, _partitions, _partitions_type)
-> 2529         Env.backend().execute(MatrixWrite(self._mir, writer))
   2530 
   2531     class _Show:

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hail/backend/backend.py in execute(self, ir, timed)
    107 
    108     def execute(self, ir, timed=False):
--> 109         result = json.loads(Env.hc()._jhc.backend().executeJSON(self._to_java_ir(ir)))
    110         value = ir.typ._from_json(result['value'])
    111         timings = result['timings']

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/py4j/java_gateway.py in __call__(self, *args)
   1255         answer = self.gateway_client.send_command(command)
   1256         return_value = get_return_value(
-> 1257             answer, self.gateway_client, self.target_id, self.name)
   1258 
   1259         for temp_arg in temp_args:

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/hail/utils/java.py in deco(*args, **kwargs)
    223             raise FatalError('%s\n\nJava stack trace:\n%s\n'
    224                              'Hail version: %s\n'
--> 225                              'Error summary: %s' % (deepest, full, hail.__version__, deepest)) from None
    226         except pyspark.sql.utils.CapturedException as e:
    227             raise FatalError('%s\n\nJava stack trace:\n%s\n'

FatalError: VCFParseError: unexpected end of line

Java stack trace:
java.lang.RuntimeException: error while applying lowering 'InterpretNonCompilable'
	at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:26)
	at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:18)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:18)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$apply$1.apply(CompileAndEvaluate.scala:16)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:14)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:65)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:65)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:10)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:9)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:9)
	at is.hail.backend.Backend.execute(Backend.scala:65)
	at is.hail.backend.Backend.executeJSON(Backend.scala:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1364 in stage 2.0 failed 1 times, most recent failure: Lost task 1364.0 in stage 2.0 (TID 2730, localhost, executor driver): is.hail.utils.HailException: genome_chr_all_extracted.vcf:column 114424: unexpected end of line
...  0/0 0/0 0/0 0/0 0/0 0/0 0/1 0/0 ./.
                                        ^
offending line: 7	124979418	rs76221489	G	A	.	.	PR	GT	0/0	0/0	0/0	0/0	0/0	0/0...
see the Hail log for the full offending line
	at is.hail.utils.ErrorHandling$class.fatal(ErrorHandling.scala:20)
	at is.hail.utils.package$.fatal(package.scala:74)
	at is.hail.io.vcf.LoadVCF$$anonfun$parseLines$1$$anon$1.hasNext(LoadVCF.scala:1299)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:106)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:95)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$.writeSplitRegion(RichContextRDDRegionValue.scala:95)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:901)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:899)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: is.hail.io.vcf.VCFParseError: unexpected end of line
	at is.hail.io.vcf.VCFLine.parseError(LoadVCF.scala:57)
	at is.hail.io.vcf.VCFLine.nextField(LoadVCF.scala:294)
	at is.hail.io.vcf.LoadVCF$.parseLine(LoadVCF.scala:1384)
	at is.hail.io.vcf.MatrixVCFReader$$anonfun$17.apply(LoadVCF.scala:1597)
	at is.hail.io.vcf.MatrixVCFReader$$anonfun$17.apply(LoadVCF.scala:1597)
	at is.hail.io.vcf.LoadVCF$$anonfun$parseLines$1$$anon$1.hasNext(LoadVCF.scala:1277)
	... 52 more

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
	at is.hail.sparkextras.ContextRDD.collect(ContextRDD.scala:223)
	at is.hail.rvd.RVD.writeRowsSplit(RVD.scala:915)
	at is.hail.expr.ir.MatrixValue.write(MatrixValue.scala:214)
	at is.hail.expr.ir.MatrixNativeWriter.apply(MatrixWriter.scala:39)
	at is.hail.expr.ir.WrappedMatrixWriter.apply(MatrixWriter.scala:24)
	at is.hail.expr.ir.Interpret$.run(Interpret.scala:583)
	at is.hail.expr.ir.Interpret$.alreadyLowered(Interpret.scala:54)
	at is.hail.expr.ir.InterpretNonCompilable$.interpretAndCoerce$1(InterpretNonCompilable.scala:16)
	at is.hail.expr.ir.InterpretNonCompilable$.is$hail$expr$ir$InterpretNonCompilable$$rewrite$1(InterpretNonCompilable.scala:53)
	at is.hail.expr.ir.InterpretNonCompilable$.apply(InterpretNonCompilable.scala:58)
	at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.transform(LoweringPass.scala:50)
	at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)
	at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3$$anonfun$1.apply(LoweringPass.scala:15)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
	at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:15)
	at is.hail.expr.ir.lowering.LoweringPass$$anonfun$apply$3.apply(LoweringPass.scala:13)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
	at is.hail.expr.ir.lowering.LoweringPass$class.apply(LoweringPass.scala:13)
	at is.hail.expr.ir.lowering.InterpretNonCompilablePass$.apply(LoweringPass.scala:45)
	at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:20)
	at is.hail.expr.ir.lowering.LoweringPipeline$$anonfun$apply$1.apply(LoweringPipeline.scala:18)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at is.hail.expr.ir.lowering.LoweringPipeline.apply(LoweringPipeline.scala:18)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$apply$1.apply(CompileAndEvaluate.scala:16)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:69)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:14)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:65)
	at is.hail.backend.Backend$$anonfun$execute$1.apply(Backend.scala:65)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:10)
	at is.hail.expr.ir.ExecuteContext$$anonfun$scoped$1.apply(ExecuteContext.scala:9)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.annotations.Region$.scoped(Region.scala:18)
	at is.hail.expr.ir.ExecuteContext$.scoped(ExecuteContext.scala:9)
	at is.hail.backend.Backend.execute(Backend.scala:65)
	at is.hail.backend.Backend.executeJSON(Backend.scala:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

is.hail.utils.HailException: genome_chr_all_extracted.vcf:column 114424: unexpected end of line
...  0/0 0/0 0/0 0/0 0/0 0/0 0/1 0/0 ./.
                                        ^
offending line: 7	124979418	rs76221489	G	A	.	.	PR	GT	0/0	0/0	0/0	0/0	0/0	0/0...
see the Hail log for the full offending line
	at is.hail.utils.ErrorHandling$class.fatal(ErrorHandling.scala:20)
	at is.hail.utils.package$.fatal(package.scala:74)
	at is.hail.io.vcf.LoadVCF$$anonfun$parseLines$1$$anon$1.hasNext(LoadVCF.scala:1299)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:106)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:95)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$.writeSplitRegion(RichContextRDDRegionValue.scala:95)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:901)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:899)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

is.hail.io.vcf.VCFParseError: unexpected end of line
	at is.hail.io.vcf.VCFLine.parseError(LoadVCF.scala:57)
	at is.hail.io.vcf.VCFLine.nextField(LoadVCF.scala:294)
	at is.hail.io.vcf.LoadVCF$.parseLine(LoadVCF.scala:1384)
	at is.hail.io.vcf.MatrixVCFReader$$anonfun$17.apply(LoadVCF.scala:1597)
	at is.hail.io.vcf.MatrixVCFReader$$anonfun$17.apply(LoadVCF.scala:1597)
	at is.hail.io.vcf.LoadVCF$$anonfun$parseLines$1$$anon$1.hasNext(LoadVCF.scala:1277)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:106)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:102)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:101)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1$$anonfun$apply$2.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:99)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2$$anonfun$apply$1.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:97)
	at is.hail.io.RichContextRDDRegionValue$$anonfun$2.apply(RichContextRDDRegionValue.scala:95)
	at is.hail.utils.package$.using(package.scala:596)
	at is.hail.io.fs.HadoopFS.writeFile(HadoopFS.scala:406)
	at is.hail.io.RichContextRDDRegionValue$.writeSplitRegion(RichContextRDDRegionValue.scala:95)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:901)
	at is.hail.rvd.RVD$$anonfun$25.apply(RVD.scala:899)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$cmapPartitionsWithIndex$1$$anonfun$apply$16.apply(ContextRDD.scala:281)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at is.hail.sparkextras.ContextRDD$$anonfun$run$1$$anonfun$apply$8.apply(ContextRDD.scala:219)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:403)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:409)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)






Hail version: 0.2.33-5d8cae649505
Error summary: VCFParseError: unexpected end of line
2

So this error message suggests that we expected there to be more information in this line of the file. It says line 7 was the offending line. Is line 7 the first line of actual data, after all the meta information lines? (Meta information lines are the ones starting with ##)

3

Thanks for your comment and questions. Does not look like line 7 is the first line, from what I can tell. When I try opening the vcf file in terminal, it appears that the ## meta information lines are more than 7. How would you suggest trying to isolate this line 7 and looking through all the fields? I do not have much experience with vcf files. Thank you again, all inputs are appreciated.

`$ more genome_chr_all_extracted.vcf`

Will produce the following:

    ##fileformat=VCFv4.2

    ##fileDate=20200518

    ##source=PLINKv1.90

    ##contig=<ID=1,length=249222528>

    ##contig=<ID=2,length=243041412>

    ##contig=<ID=3,length=197848557>

    ##contig=<ID=4,length=190914290>

    ##contig=<ID=5,length=180698414>

    ##contig=<ID=6,length=170907735>

    ##contig=<ID=7,length=159124174>

    ##contig=<ID=8,length=146296405>

    ##contig=<ID=9,length=141101940>

    ##contig=<ID=10,length=135440227>

    ##contig=<ID=11,length=134945121>

    ##contig=<ID=12,length=133831320>

    ##contig=<ID=13,length=115103151>

    ##contig=<ID=14,length=107287664>

    ##contig=<ID=15,length=102397318>

    ##contig=<ID=16,length=90170096>

    ##contig=<ID=17,length=81103683>

    ##contig=<ID=18,length=78015181>

    ##contig=<ID=19,length=59097934>

    ##contig=<ID=20,length=62960293>

    ##contig=<ID=21,length=48099611>

    ##contig=<ID=22,length=51238250>

    ##INFO=<ID=PR,Number=0,Type=Flag,Description="Provisional reference allele, may not be based on real reference genome">

    ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">

    #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT
4

I’m sorry, I misread the error message.

it says:

offending line: 7	124979418	rs76221489	G	A	.	.	PR	GT	0/0	0/0	0/0	0/0	0/0	0/0...

as in, the offending line is contig 7, locus 124979418. Nothing about line 7.

Anyway, it’s possible you’re just missing a sample on that particular line. Maybe check out that line in the VCF file and see if it has fewer samples than you’d expect.

1 Like
5

also recommend VCF-validator: https://vcftools.github.io/perl_module.html#vcf-validator

1 Like