1 of 1 people found this helpful
I thinnk the official Spark v2 certification will come in the next release of the Control-M plugin for Hadoop.
In the meantime, I remember a company that had both Spark v1 and v2 in the same system (looks like in your case) and wanted to start using Spark v2 from Control-M, so what they did was just to create a symlink for the Spark submit command.
With both versions installed, you invoke “spark-submit” for Spark v1 and “spark2-submit” for Spark v2. You can check the info here: https://www.cloudera.com/documentation/spark2/latest/topics/spark_running_apps.html (this is from Cloudera but I suppose it should apply to other Hadoop distros).
To make Control-M work with Spark v2, they just created a symlink so that "spark-submit" points to “spark2-submit”.
They were using Spark v1 only for some old tests (and not from Control-M), so they also created a new symlink so that a new alias e.g. "spark-submit-v1" will point to the original "spark-submit" executable.