Why can domestic open source library libhv be included by AWESOME-C and AWESOME-CPP

2023-01-23   ES  

Pyspark shell on Spark is very troublesome and can only be entered in the command line. In addition to SPARK SUBMIT, you can also use Jupyter Notebook.

Run ipython

First configure environment variables
Add the following two lines to/etc/Profile:

export PYSPARK_SUBMIT_ARGS="--master spark://192.168.175.128:7070 pyspark-shell"

export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/build:$PYTHONPATH

If you report PY4J error, you can add another line:

export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH

Open Notebook

$ jupyter notebook

Initialized Pyspark

import os
execfile(os.path.join(os.environ["SPARK_HOME"], 'python/pyspark/shell.py'))

SC variables should be obtained

In [2]: sc
Out[2]: <pyspark.context.SparkContext at 0x10a982b10>

Another way

Set environment variables

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

and then run directly

pyspark

Notebook automatically opens, SC variables can be obtained automatically

source

Related Posts

IDEA configures ECMAScript6 syntax to ECMAScript5 to improve JS coding efficiency

spark batch data to enter hbase

jackson manual

[.NET code farmer] C# generic programming

Why can domestic open source library libhv be included by AWESOME-C and AWESOME-CPP

Random Posts

numerical calculation · Episode 10: Mixed-integer secondary plan (CVXPY version)

linux process operation wakeme

Kotlin – Open, Final, Abstract modifier sinocrazy

python: Then write Parzen Window Classifier

pycharm background replacement