Skip to main content

Spark

Use this to configure a SparkContext for a your task. Task's marked with this will automatically execute natively onto K8s as a distributed execution of spark

Attributes​

  • spark_conf: Optional[Dict[str, str]] = None

    • Spark configuration dictionary.

  • hadoop_conf: Optional[Dict[str, str]] = None

    • Hadoop configuration dictionary.

  • executor_path: Optional[str] = None

    • Path to the Python binary for PySpark execution.

  • applications_path: Optional[str] = None

    • Path to the main application file.

  • driver_pod: Optional[PodTemplate] = None

    • Pod template for the driver pod.

  • executor_pod: Optional[PodTemplate] = None

    • Pod template for the executor pods.