# Pass additional .py files with custom functions to hailctl dataproc start

**URL:** <https://discuss.hail.is/t/pass-additional-py-files-with-custom-functions-to-hailctl-dataproc-start/3982>\
**Category:** Hail Query & hailctl\
**Created:** [October 30, 2024, 8:10pm UTC](https://discuss.hail.is/t/pass-additional-py-files-with-custom-functions-to-hailctl-dataproc-start/3982 "2024-10-30T20:10:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![miargentieri](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/miargentieri/32/1123_2.png) [@miargentieri](https://discuss.hail.is/u/miargentieri)\
**Post date:** [October 30, 2024, 8:10pm UTC](https://discuss.hail.is/t/pass-additional-py-files-with-custom-functions-to-hailctl-dataproc-start/3982/1 "2024-10-30T20:10:52Z")

</div>

I am creating a hail dataproc cluster using the following code:

```python
hailctl dataproc start aargenti \
    --region us-central1 \
    --zone us-central1-b \
    --init gs://path/code/install_packages.sh \
    --master-machine-type n1-highmem-8 \
    --master-boot-disk-size 100 \
    --num-workers 4 \
    --worker-machine-type n1-highmem-8 \
    --worker-boot-disk-size 500 \
    --num-secondary-workers 50 \
    --secondary-worker-boot-disk-size 40 \
    --max-idle=10m

```

Once the cluster is running, I submit a .py file to run on the cluster (e.g., `hailctl dataproc submit aargenti /path/python_script.py `). However, I would like to be able to load additional .py files into the python environment on the dataproc cluster so that I can import additional custom functions into my main python\_script.py file (by adding something like `from custom_functions.py import *` to the top of my python\_script.py file). Could you please provide some guidance on how I can achieve this? I tried listing the custom\_functions.py file using the --py-files option when running `hailctl dataproc start `, but this doesn’t seem to work. custom\_functions.py is also saved on GCP.

Thanks.

---

<div class="post-metadata">

**Author:** ![chrisvittal](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/chrisvittal/32/327_2.png) [@chrisvittal](https://discuss.hail.is/u/chrisvittal)\
**Post date:** [October 30, 2024, 8:25pm UTC](https://discuss.hail.is/t/pass-additional-py-files-with-custom-functions-to-hailctl-dataproc-start/3982/2 "2024-10-30T20:25:41Z")

</div>

The `--pyfiles` argument is passed to `hailctl dataproc submit`, not to `start`.

So the submit command would look like:

```python
hailctl dataproc submit --pyfiles /path/custom_functions.py cluster /path/python_script.py

```

---

<div class="post-metadata">

**Author:** ![miargentieri](https://yyz2.discourse-cdn.com/flex036/user_avatar/discuss.hail.is/miargentieri/32/1123_2.png) [@miargentieri](https://discuss.hail.is/u/miargentieri)\
**Post date:** [October 30, 2024, 8:43pm UTC](https://discuss.hail.is/t/pass-additional-py-files-with-custom-functions-to-hailctl-dataproc-start/3982/3 "2024-10-30T20:43:46Z")

</div>

Thanks for such a quickly reply - that worked perfectly!