Originally posted by: Sheryl_M

IBM Spectrum Conductor with Spark v2.1.0 is now available.

IBM Spectrum Conductor with Spark (known previously as IBM Platform Conductor for Spark) enables organizations to deploy Apache Spark efficiently and effectively, supporting multiple deployments of Spark, increasing performance and scale, and maximizing resource usage. Now available as part of the IBM Spectrum Computing portfolio, IBM Spectrum Conductor with Spark is the next-generation of software-defined computing infrastructure, better positioned to achieve faster results by efficiently analyzing, accessing, and protecting data.

This release of IBM Spectrum Conductor with Spark provides installation options to better accommodate your requirements. You can choose to install IBM Spectrum Conductor with Spark as standalone software packages or as bare metal packages:

• If your hosts are already deployed with a supported operating system, you can use the standalone software package to install IBM Spectrum Conductor with Spark on your existing hosts. With this option, you can use IBM Spectrum Conductor with Spark to efficiently manage and monitor multiple Spark deployments.
• If you want to manage your infrastructure as part of an integrated environment for infrastructure, storage, and application management, you can use the bare metal package to install IBM Spectrum Conductor with Spark. With this option, you can set up your infrastructure, provision bare metal nodes, and automatically install IBM Spectrum Conductor with Spark and IBM Spectrum Scale (a high-performance solution for storage management) on all nodes.

For more information on IBM Spectrum Conductor, refer to the IBM Knowledge Center.

So here's what you get with IBM Spectrum Conductor with Spark v2.1.0:

Scheduling of Spark batch jobs

You can now schedule Spark batch applications to run on a Spark instance group. You can schedule a batch application to run just once at a specified time or to run periodically at a specified time or interval. You can also write cron expressions to set complex schedules; for example, to submit a batch application every Monday and Tuesday at 7 AM.

Once you define a schedule, you can manage its entire lifecycle: you can view its history, modify the schedule, pause and resume batch application submission, or remove the schedule altogether.

Docker support for Spark instance groups, notebooks

You can now choose to enable Docker support for your Spark instance groups (including any associated notebooks). Docker is used to build, ship, and run distributed applications. By Dockerizing Spark instance groups, you encapsulate and execute its services inside separate Docker containers.

While we don’t ship a Docker image with IBM Spectrum Conductor with Spark in this release, it is quite easy for you to do it by yourself. Watch for a blog soon on how to build an IBM Spectrum Conductor with Spark Docker image.

Secure iPython

If you want to use the IPython-Jupyter notebook with IBM Spectrum Conductor with Spark, you can integrate a secure version of the sample, wherein the notebook user's credentials are required to log in to the IPython notebook GUI. For more information, see the Conductor-IPython-Jupyter sample on IBM Bluemix® DevOps Services.

Permissions for Spark instance groups

Consumer administrator permissions: With this release, a consumer administrator can also create Spark instance groups. The consumer administrator can create Spark instance groups under their branch of the consumer tree.

User impersonation permissions:  When creating a Spark instance group, you can now optionally request resources using the credentials of an EGO user. With this change, services associated with the Spark instance group, by default, use the credentials of the EGO user logged in to the cluster management console. Use this option to isolate users with higher levels of security than those required to request resources for a Spark instance group.

If you are a consumer administrator or have the ‘Services Assign Impersonation (Self)’ permission, the system automatically uses your credentials. If you are a cluster administrator or have the ‘Services Assign Impersonation (Any User)’ permission, you can specify any EGO user for impersonation.

For a complete list of what’s available in this release, see the Release Notes in the IBM Knowledge Center.

Get IBM Spectrum Conductor with Spark today!

To try out IBM Spectrum Conductor with Spark, download an evaluation version from our Service Management Connect page. You can take advantage of IBM Spectrum Conductor with Spark 2.1.0 now, or look for a functional and quality update soon with our upcoming Fix Pack 2.1.0.1. Watch this space for updates!