You can set the memory using an environment variable:
PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell" ipython
This will start an ipython
notebook with 8 GB of memory. If you want ipython
to always start with 8 GB of memory, you can add this to your .bashrc
(or the equivalent file for your shell):
export PYSPARK_SUBMIT_ARGS="--driver-memory 8g --executor-memory 8g pyspark-shell"