Why Can't I Find the Specified Python Environment After Adding the Python Package?
I cannot find the specified Python environment after adding the Python 3 package.
Set spark.yarn.appMasterEnv.PYSPARK_PYTHON to python3 in the conf file to specify the Python 3 environment for the compute cluster.
New clusters use the Python 3 environment by default.
Parent topic: Spark Job Development