The new releases IBM Spectrum LSF Process Manager 10.1 and IBM Spectrum LSF Application Center 10.1 combine the strength of the two products, providing a seamless experience of using applications within flows, and building applications based on flows.
Over the years, IBM Spectrum LSF Application Center users have created many submission templates to integrate applications. These templates enable users to interact with intuitive, self-documenting interfaces to submit applications to LSF. Many of these applications are required to run one after another with interdependencies to form a workflow. It makes perfect sense to directly use the existing submission template interface when building such workflows in Process Manager. This can now be done with ease in Process Manager and Application Center 10.1.
Applications within flows
IBM Spectrum LSF Application Center 10.1 provides web-based Flow Editor to create and edit flow definitions, with similar interface to Process Manager Flow Editor Java client.
From Application Center, you can choose to create a new flow definition or edit an existing flow definition. In Figure 1, the user chooses to edit an existing flow definition to create a new version.
Figure 1. Create new version for a flow definition in Application Center.
After confirmation, the selected flow definition is opened in Flow Editor within a web browser, as shown in Figure 2. The flow contains just one step currently.
Figure 2. Web-based Flow Editor
In the web-based Flow Editor, you can directly add an application to the flow by choosing the icon 
on the tool bar. All the published submission templates from Application Center are available to be added to the flow (Figure 3).
Figure 3. Choose an available submission form to add to a flow
After the submission template is selected, its submission form interface is displayed for user to fill out all the necessary fields to run this application within the flow. Figure 4 shows a sample submission form for a Spark application. The submission template will submit an LSF job that runs Spark application in client mode using “spark-submit” command.
Figure 4. A sample submission form for a Spark application in a flow
After the submission template job is defined, you can connect the two steps in the flow (Figure 5).
Figure 5. Sample flow with two applications
Different applications can be mixed with regular LSF jobs and other work items within a flow. Figure 6 shows the flow after a regular LSF job is added.
Figure 6. Sample flow with two applications and one regular LSF job
After the new version is committed, you can submit/trigger the flow to run. To run the submission form jobs within the flow, Process Manager Server calls Application Center server REST API to invoke the corresponding submission script. The behavior of running a submission form within a flow is exactly the same as running it standalone. The progress of the flow can be monitored in Application Center. Figure 7 shows the flow chart where the first submission template job has started to run.
Figure 7. Monitoring a running flow in Application Center
Flows within applications
When creating a flow definition, a flow designer can specify input variables that represent the required parameters for the flow definition. An example is shown in Figure 8. When end users run the flow, they need provide values for these parameters by specifying variable name-value pairs. However, the meaning of the variables are not always easy for end users to understand.
Figure 8. Example of flow input variables
Application Center supports creating a submission template based on a flow definition. Through the submission template, the designer can provide user-friendly interface to end users. End user just need fill out the form then click Submit button to run the flow, without worrying about meaning of the input variables, just like the submission forms for applications.
In Application Center 10.1, flow-based submission template can be generated automatically. After the designer selects a flow definition and chooses to create a flow-based submission template, both submission form and submission script are automatically generated based on the input variables of the flow definition. In submission form, each input variable has a corresponding text field with field ID as “ID_<VariableName>”. The submission script calls jsubmit(jtrigger) command to submit the flow with corresponding variable values. The auto-generated submission form can work out-of-box right away. An example is shown in Figure 9.
Figure 9. Example of auto-generated flow-based submission form and submission script
The template designer can then further modify or customize the auto-generated submission form and submission script. For example, use better labels for the fields, or change a text field to a drop down list; or add business logic to the submission script. Figure 10 shows the modified submission template that is ready for end user to submit a flow.
Figure 10. Updated flow-based submission form ready for use
When a flow definition is exposed through the user-friendly submission form, the details of the flow definition become transparent to end users, and the flow definition acts the same way as an application, with the only difference that it will run as a flow instead of a job. In summary, a flow is now encapsulated within an application.
A flow-based submission form can be included in a main flow, just like a normal application. While a normal application runs as an LSF job, a flow-based submission form within a flow runs as a dynamic subflow in Process Manager. In other words, at runtime Process Manager Server will expand the submission form into a subflow, and set variable values for it based on the input variable values obtained from the submission form.
Conclusion
The new releases Process Manager 10.1 and Application Center 10.1 provide enhanced experience of using applications within flows, and generating submission templates based on flows. They provide template and flow designers great tools to reuse existing submission templates when designing flows, and in turn provide end users friendly and familiar interfaces to run applications and flows.
Tags: High-Performance Computing, HPC, IBM Spectrum Computing, IBM Spectrum LSF
#HPC#LSF#SpectrumComputingGroup