# FileNotFoundException when reading VDS

**URL:** <https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228>\
**Category:** Help \[0.1\]\
**Created:** [June 5, 2017, 8:23pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228 "2017-06-05T20:23:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joel\_Thibault](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/joel_thibault/32/74_2.png) [@Joel\_Thibault](https://discuss.hail.is/u/Joel_Thibault)\
**Post date:** [June 5, 2017, 8:23pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228/1 "2017-06-05T20:23:34Z")

</div>

Continuing my adventures from [earlier](http://discuss.hail.is/t/running-hail-with-a-remote-spark/227/10) I am now successfully submitting Spark jobs from my Docker container to DataProc. Now I’m attempting to run Hail commands in Pyspark. Not sure if the problem I seeing is a result of my unusual setup.

I’m trying to load a local (ok, Docker-mounted) VDS file that I have copied from a previous successful Hail run, but I’m getting a File Not Found error.

```
>>> from hail import *
>>> hc = HailContext(sc)
hail: info: SparkUI: http://10.128.0.6:4040
>>> count = hc.read('file:/home/joel/work/1kg.vds').count()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<decorator-gen-153>", line 2, in read
  File "/hail/python/hail/java.py", line 104, in handle_py4j
    raise FatalError(msg)
hail.java.FatalError: FileNotFoundException: File file:/home/joel/work/1kg.vds/rdd.parquet/part-r-00000-4b3d3bac-2697-423b-8283-07af77b46a72.snappy.parquet does not exist

```

However I do see the file `/home/joel/work/1kg.vds/rdd.parquet/part-r-00000-4b3d3bac-2697-423b-8283-07af77b46a72.snappy.parquet`

Do you know why it might not see this file? Thanks.

Also, do I have the syntax right for local files? I also tried `hc.read('/home/joel/work/1kg.vds').count()` but I got a different error `hail.java.FatalError: arguments refer to no files` instead

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [June 5, 2017, 9:22pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228/2 "2017-06-05T21:22:41Z")

</div>

It looks like you may have an old version of Hail. Newer versions include a full Java stack trace. If you can use a newer version, that’d help debugging in the future. In general, the latest version of the Hail jar and the python files are available at:

```nohighlight
dking@wmb16-359 # gsutil ls "gs://hail-common/hail-hail-is-master-all-spark2.0.2-$(gsutil cat gs://hail-common/latest-hash.txt).jar"
gs://hail-common/hail-hail-is-master-all-spark2.0.2-53e9d33.jar
dking@wmb16-359 # gsutil ls "gs://hail-common/pyhail-hail-is-master-$(gsutil cat gs://hail-common/latest-hash.txt).zip" 
gs://hail-common/pyhail-hail-is-master-53e9d33.zip

```

You correctly deduced that the `file:` prefix is required, the default file system is whatever Hadoop file system is connected to the cluster.

My best guess is that the user executing the Spark job does not have the same permissions that your shell session has. @tpoterba is out of town this week, but might have a better idea of what is wrong when he returns. It’s also possible this is a now fixed bug in the version of Hail you’re using.

---

<div class="post-metadata">

**Author:** ![danking](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/danking/32/43_2.png) [@danking](https://discuss.hail.is/u/danking)\
**Post date:** [June 6, 2017, 12:39pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228/3 "2017-06-06T12:39:39Z")

</div>

@Joel_Thibault I’ve given some more thought to this issue and I’m somewhat surprised that you’re reading from the local file system. I guess this error may come from an executor on a worker node which doesn’t share a local file system with the master node where, I assume, you’ve placed `1kg.vds`. Usually, when storing a parquet file like a VDS one stores it on a network mounted file system like HDFS so that the cluster’s executors can ingest the file in parallel. I’m not sure if Dataproc has a traditional HDFS. Google pushes Google Storage pretty aggressively. Is it possible for you to store `1kg.vds` in Google Storage instead?

---

<div class="post-metadata">

**Author:** ![Joel\_Thibault](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/joel_thibault/32/74_2.png) [@Joel\_Thibault](https://discuss.hail.is/u/Joel_Thibault)\
**Post date:** [June 6, 2017, 3:28pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228/4 "2017-06-06T15:28:34Z")

</div>

OK - our workflow will be using GCS anyway, so I’ll stop trying to make local files work where it’s not appropriate. I’ll upgrade Hail too so I won’t waste your time making bug reports for old versions!

Thanks.

---

<div class="post-metadata">

**Author:** ![Joel\_Thibault](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/joel_thibault/32/74_2.png) [@Joel\_Thibault](https://discuss.hail.is/u/Joel_Thibault)\
**Post date:** [June 7, 2017, 4:23pm UTC](https://discuss.hail.is/t/filenotfoundexception-when-reading-vds/228/5 "2017-06-07T16:23:20Z")

</div>

Working much better now with `gs://` files. Thanks.