Print out SparkSession Config Options

A forum for general discussion of the Python programming language.

Print out SparkSession Config Options

Postby mmike5 » Fri Sep 09, 2016 6:13 pm

Hi,

When I start pyspark, a SparkSession is automatically generated and available as 'spark'. I would like to print/view the details of the spark session but am having a lot of difficulty accessing these parameters.

Pyspark auto creates a SparkSession. This can be created manually using the following code:


Code: Select all
from pyspark.sql import SparkSession

spark 
= SparkSession\
    .builder\
    .appName("PythonSQL")\
    .config("spark.some.config.option", "some-value")\
    .getOrCreate() 


I would like to view/print the appname and config options. The reason I would like to see these is as a result of another issue that I am experiencing which this may shed light on.
Thank for any insight, the help is very much appreciated, thanks Michael
Last edited by micseydel on Fri Sep 09, 2016 7:43 pm, edited 1 time in total.
Reason: Code tags. Initial post lock.
mmike5
 
Posts: 3
Joined: Fri Sep 09, 2016 6:06 pm

Return to General Discussions

Who is online

Users browsing this forum: Bing [Bot] and 4 guests