Pyspark shell on Spark is very troublesome and can only be entered in the command line. In addition to SPARK SUBMIT, you can also use Jupyter Notebook.

Run ipython

First configure environment variables
Add the following two lines to/etc/Profile:

export PYSPARK_SUBMIT_ARGS="--master spark://192.168.175.128:7070 pyspark-shell"

export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/build:$PYTHONPATH

If you report PY4J error, you can add another line:

export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH

Open Notebook

$ jupyter notebook

Initialized Pyspark

import os
execfile(os.path.join(os.environ["SPARK_HOME"], 'python/pyspark/shell.py'))

SC variables should be obtained

In [2]: sc
Out[2]: <pyspark.context.SparkContext at 0x10a982b10>

Another way

Set environment variables

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

and then run directly

pyspark

Notebook automatically opens, SC variables can be obtained automatically