Package cern.nxcals.api.utils

Class SparkUtils

  • public final class SparkUtils
extends java.lang.Object

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static org.apache.spark.SparkConf createSparkConf​(cern.nxcals.api.config.SparkProperties config)
      Creates Spark configuration
      static org.apache.spark.sql.SparkSession createSparkSession​(cern.nxcals.api.config.SparkProperties properties)
      Creates Spark session with the given properties
      static org.apache.spark.sql.SparkSession createSparkSession​(org.apache.spark.SparkConf config)
      Creates Spark session with the given config
      static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull cern.nxcals.api.config.SparkProperties sparkProperties)
      Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
      static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull org.apache.spark.SparkConf sparkConf)
      Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
      static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull org.apache.spark.SparkConf sparkConf, @NonNull java.util.List<cern.nxcals.common.config.SparkSessionModifier> sessionModifiers)
      Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
      static void modifySparkSession​(org.apache.spark.sql.SparkSession session, java.util.Collection<cern.nxcals.common.config.SparkSessionModifier> modifiers)
      Modifies Spark session according to the collection of modifiers

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • createSparkConf

        public static org.apache.spark.SparkConf createSparkConf​(cern.nxcals.api.config.SparkProperties config)
        Creates Spark configuration
        Parameters:
        config - Spark properties config
        Returns:
        Configuration for a Spark application

      • createSparkSession

        public static org.apache.spark.sql.SparkSession createSparkSession​(org.apache.spark.SparkConf config)
        Creates Spark session with the given config
        Parameters:
        config - config of the session
        Returns:
        spark session

      • modifySparkSession

        public static void modifySparkSession​(org.apache.spark.sql.SparkSession session,
                                      java.util.Collection<cern.nxcals.common.config.SparkSessionModifier> modifiers)
        Modifies Spark session according to the collection of modifiers
        Parameters:
        session - session to modify
        modifiers - to apply to the session

      • createSparkSession

        public static org.apache.spark.sql.SparkSession createSparkSession​(cern.nxcals.api.config.SparkProperties properties)
        Creates Spark session with the given properties
        Parameters:
        properties - properties of the session
        Returns:
        spark session

      • createSparkSessionSupplier

        public static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull
                                                                                                        @NonNull org.apache.spark.SparkConf sparkConf,
                                                                                                        @NonNull
                                                                                                        @NonNull java.util.List<cern.nxcals.common.config.SparkSessionModifier> sessionModifiers)
        Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
        Parameters:
        sparkConf - config of the session
        sessionModifiers - to apply to the session

      • createSparkSessionSupplier

        public static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull
                                                                                                        @NonNull org.apache.spark.SparkConf sparkConf)
        Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
        Parameters:
        sparkConf - config of the session

      • createSparkSessionSupplier

        public static java.util.function.Supplier<org.apache.spark.sql.SparkSession> createSparkSessionSupplier​(@NonNull
                                                                                                        @NonNull cern.nxcals.api.config.SparkProperties sparkProperties)
        Creates Spark session supplier that is able to recreate and return a new instance of the SparkSession in case the current one was closed.
        Parameters:
        sparkProperties - Spark properties config