1 Reply Latest reply on Nov 15, 2018 4:31 AM by David Fernandez

    Spark 2

    Hans Feldmann

      I'm running 9.0.18 with the Hadoop plug-in and would like to know how to enable Spark 2. I've tried change the profile, but it's not working. This is the warning message I'm seeing.

      Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set

      Spark1 will be picked by default

        • 1. Re: Spark 2
          David Fernandez

          I thinnk the official Spark v2 certification will come in the next release of the Control-M plugin for Hadoop.

           

          In the meantime, I remember a company that had both Spark v1 and v2 in the same system (looks like in your case) and wanted to start using Spark v2 from Control-M, so what they did was just to create a symlink for the Spark submit command.

           

          With both versions installed, you invoke “spark-submit” for Spark v1 and “spark2-submit” for Spark v2. You can check the info here: https://www.cloudera.com/documentation/spark2/latest/topics/spark_running_apps.html (this is from Cloudera but I suppose it should apply to other Hadoop distros).

           

          To make Control-M work with Spark v2, they just created a symlink so that "spark-submit" points to “spark2-submit”.

           

          They were using Spark v1 only for some old tests (and not from Control-M), so they also created a new symlink so that a new alias e.g. "spark-submit-v1" will point to the original "spark-submit" executable.

          1 of 1 people found this helpful