Modern life-sciences research relies on complex computational pipelines. Genomics, precision medicine, molecular modeling, and drug discovery can involve many processing stages with dependencies, different resource needs, and large numbers of samples.

Running one bioinformatics command is easy. Running a reliable pipeline across hundreds or thousands of samples is much harder.

IBM Spectrum LSF Process Manager addresses this by providing workflow orchestration on top of IBM Spectrum LSF.  Live demo 

The IBM Spectrum LSF Suite 10.2 demonstration shows a practical genomics example using BWA and SAMtools. A bioinformatician builds and tests an alignment workflow, connects jobs with success dependencies, parameterizes the flow, publishes a validated version, and makes it available to scientists for repeatable execution.

Scientists -> LSF Process Manager Workflow orchestration -> IBM Spectrum LSF Resource scheduling -> HPC Infrastructure ->  Scientific Results

From Commands to Scientific Workflows

A simplified sequencing pipeline might look like:

FASTQ -> BWA Alignment -> SAM  ->  SAMtools SAM -> BAM -> Sort BAM  -> Variant Calling -> Scientific Results

Without workflow management, users may rely on multiple scripts and custom logic to determine whether each stage completed successfully.

Process Manager turns these steps into an explicit flow, making dependencies and execution state easier to understand and manage.

Separating Bioinformaticians from Scientists

The demonstration highlights two user roles.

The bioinformatician acts as the Flow Designer. This user understands BWA, SAMtools, reference genomes, command-line parameters, HPC resources, and workflow dependencies.

The scientist or biologist acts as the Flow Submitter. This user mainly needs to provide sample information and run an approved workflow.

This reduces the need for every scientist to understand low-level LSF commands or bioinformatics application syntax.

Building the Alignment Pipeline

The workflow begins with a job named:

CreatingSam

This stage uses BWA to align sequencing reads against a reference genome and produce a SAM file.

The second stage is:

ConvertSAMtoBAM

which uses SAMtools to convert the SAM file to BAM format.

Part 3 adds another stage:

Sort

which sorts the BAM file using SAMtools.

The resulting flow is:

CreatingSam -> Succeeds  ->  ConvertSAMtoBAM -> Succeeds -> Sort

The Succeeds dependency is important. A downstream stage runs only when its prerequisite completes successfully.

This prevents meaningless processing when an upstream scientific step fails.

Parameterizing the Workflow

Instead of hard-coding values, the flow uses variables such as:

BWA_BIN
SAMTOOLS_BIN
THREADS
SAMPLENAME
REFERENCE
INPUT_DIR
OUTPUT_DIR

Conceptually, a BWA job can use:

${BWA_BIN} mem \
    -t ${THREADS} \
    ${REFERENCE} \
    ${INPUT_DIR}/${SAMPLENAME}_1.fastq \
    ${INPUT_DIR}/${SAMPLENAME}_2.fastq

This allows the same workflow to process many different samples without changing its logic.

For example:

SAMPLENAME = Patient001

and later:

SAMPLENAME = Patient002

can use the same published flow.

Testing and Selective Rerun

Part 3 demonstrates an important workflow-development feature: rerunning from selected points.

Suppose the first two stages are already working:

A bioinformatician can test the Sort stage without rerunning expensive alignment processing every time.

This is especially valuable in genomics, where upstream stages can take significant CPU time.

The development cycle becomes:

Design -> Run -> Inspect -> Modify -> Rerun selected stage -> Validate

This makes pipeline development faster and avoids unnecessary recomputation.

Draft, Version, and Publish

The demonstration also shows a workflow lifecycle based on drafts and published definitions.

A team can follow a process such as:

Draft -> Modify -> Test -> Validate -> Publish

This is useful in life-sciences environments because scientific pipelines evolve over time.

For example:

Alignment v1.0
Alignment v1.1
Alignment v1.2

A new version can be developed and tested without immediately affecting scientists who rely on the currently published version.

Publishing therefore creates a useful operational boundary between a workflow that is still under development and one intended for wider use.

One Definition, Many Executions

After publication, the same workflow definition can be submitted many times.

Each submission becomes its own flow instance with its own execution state and working data.

This allows one validated workflow to support large sequencing projects while keeping each sample execution separate.

Monitoring Scientific Processing

Process Manager also connects workflow state with job data, generated files, history, and flow definitions.

Instead of seeing only job IDs such as:

Job 182736 DONE
Job 182737 EXIT

the user can see the scientific context:

This makes troubleshooting more meaningful because the user can immediately identify where the scientific pipeline stopped.

Flow-specific working directories also help isolate logs, intermediate files, and results for different workflow instances.

Process Manager and LSF Have Different Roles

The distinction between workflow orchestration and workload scheduling is central.

Process Manager determines:

Which stage is ready?
Did its dependency succeed?
Which part should be rerun?
Which flow version is published?
What is the flow state?

IBM Spectrum LSF determines:

Which host runs the job?
How much CPU and memory are available?
Which queue applies?
When should the job start?

Together:

Process Manager understands the scientific workflow, while LSF understands the computing infrastructure.

Scaling Across Many Samples

Once the pipeline is validated, the same pattern can scale to thousands of samples:

Process Manager exposes independent work, while LSF determines how much can run based on cluster capacity and scheduling policy.

Different workflow stages can also request different resources. Alignment may need many CPUs and memory, while later AI analysis may require GPUs.

Conclusion

The IBM Spectrum LSF Process Manager genomics example demonstrates much more than a sequence of HPC jobs.

It shows how BWA and SAMtools commands can be transformed into a parameterized, dependency-aware, reusable scientific workflow.

Process Manager adds:

Parameters
Dependencies
Testing
Selective rerun
Versioning
Publishing
Submission
Monitoring

IBM Spectrum LSF then schedules the resulting workloads across available HPC resources.

The result is a clear separation of responsibilities:

Bioinformaticians design and evolve the computational pipeline.

LSF Process Manager orchestrates the workflow.

IBM Spectrum LSF schedules the compute resources.

Scientists submit their data and consume the results.

For genomics and other data-intensive life-sciences workloads, this approach turns collections of command-line tools into scalable, reusable, and manageable scientific pipelines.