Originally posted by: Steve Haertel

Creating Spark instance groups can be quick or slow, depending on how many options and parameters are customized or changed. In IBM Spectrum Conductor with Spark 2.2.1, you can now create and manage Spark instance group templates to speed up registration time and lower the number of configuration mistakes.

Spark instance group templates can be accessed by using the Templates button on the Spark Instance Groups page, under the Instance Group List tab. They can also be used from inside the Spark instance group creation wizard by using the Templates button in the top-right corner.

When you click on the Templates button, the template management dialog pops up. In this menu, you can create, edit, or remove templates, as well as show or hide templates that have been designated as drafts, and view their notes. Whether or not you choose to templatize a consumer when creating a template, you must select a consumer to own the template independently. This consumer owner controls who has access to that template, and is not related to the consumers on any Spark instance groups created from the template. Like service packages, templates that are created under consumers are available to any sub-consumers that are lower in the consumer tree. In the example below, a consumer administrator for LOB1-Production is able to access templates that are created under LOB1 or the root consumer, as well as any child consumers under LOB1-Production, if they existed.

Templates can be created manually, or they can be imported from other templates or existing Spark instance groups. The template creation wizard looks like the Spark instance group creation wizard, except that it has another tab for template metadata. Among settings in the template metadata tab is the ability to mark templates as drafts. This does not affect a template’s function. It only acts as a marker for you to know when templates are complete or incomplete. You might also want to specify notes that provide details with the template to others that might use it.

You can templatize as many or as few parameters as you wish, with the exception of the name, which must be unique.

It is also possible to templatize parameters with values that are not set until runtime. An example of this is the deployment directory.

You can use special variables in any field that you can enter a string during Spark instance group configuration. The three variables that you can use are:

1. $sig_name – The name of the Spark instance group.
2. $sig_consumer – The top consumer name (not full path) of the Spark instance group.
3. $sig_execuser – The execution user for the Spark instance group.

For example, if you create a template where the deployment directory is /tmp/$sig_name, then when you use that template, the value for the deployment directory changes as you type in the name. Below, you can see that the Spark instance group name is automatically filled in to the end of the Spark deployment directory.

You can even optionally chain variable names. For example, if your top consumer is $sig_name-Consumer, and you use /localdir/$sig_consumer as your spark.local.dir parameter, then when you use the template and set the name to “mySIG”, the spark.local.dir automatically generates the value localdir/mySIG-Consumer.

If you do not want to use these variables, you can click the pencil icon next to a field with a variable in it to override the value.

For information about the Spark instance group template RESTful APIs, access the RESTful API documentation in your cluster:

http://<hostname>:8280/cloud/apis/explorer/#/Templates

https://<hostname>:8643/cloud/apis/explorer/#/Templates

If you want to try IBM Spectrum Conductor with Spark version 2.2.1, you can download an evaluation version here. If you have any questions about our product or about Spark instance group templates, post them in our forum or let us know using our Slack channel!