This blog entry describes the factors that need to be considered in order to estimate the necessary efforts to migrate an existing environment of IBM’s products Lombardi Teamworks, WebSphere Lombardi Edition (WLE), WebSphere Process Server (WPS) or Business Process Manager (BPM) to a new version of Business Automation Workflow (BAW).
Unfortunately, there is no mathematical formula and no ‘magic’ spreadsheet that would be able to just calculate the efforts based on some basic information about the migration’s source environment (even though that would make selling BAW and BAW migrations much easier, I know).
But there are some lessons learned from completed BAW migrations that make it easier to come up with a reliable high-level estimate. These lessons learned are summarized in this article.
Please note that it is always strongly recommended for a BAW migration project to involve experts from IBM Services.
IBM Services will run or guide the migration as a project with clearly defined phases. The initial phase (Discovery) aims at coming up with a reliable estimate and plan for the necessary migration activities.
Which factors influence BAW migration efforts?
As already mentioned, it is very difficult to create an accurate estimate for BAW migration efforts. That is because the sizing depends on so many very different parameters, like the following:
• Chosen migration method (runtime migration vs. artifact migration)
• Infrastructure layout and topology
• System performance (servers and network)
• Number and complexity of deployed applications
• Number and complexity of implemented customizations
• Amount of available supporting tooling, e.g. routines to test applications
• Availability of necessary skills for the migration (impact on elapsed time)
• Lombardi/WPS/BPM/BAW skills of the professionals running the migration
• Amount and quality of the available migration documentation
Let’s look at these parameters in more detail…
Chosen migration method
First of all, you need to make a very conscious decision regarding the migration method that will be applied during a migration.
Starting with BPM V8.5, there are basically only two version-to-version migration methods left:
• Migrating business data and applications
• Artifact migration
Before that version of BPM, there were other names for migration methods like:
• Runtime migration
• Application migration (in two flavors)
• Artifact migration
Please note that an artifact migration has to be done almost always on top of a runtime migration or “Migrating business data and applications”! The runtime migration ensures that the existing business data (running and completed process and task instances etc.) is preserved and attempts to keep the existing applications in the target environment as-is. It does NOT include the migration of applications to the code base of the target version of the product or guarantee that all applications will run without errors in the new environment.
In addition, migrations can either be done locally on the source systems, or they can be performed using additional / new hardware (remote migration).
For all of these migration methods you have to consider things like retaining running instances, keeping existing customizations, migrating existing integrations with backend systems, etc.
Also, you will probably need a different set of experts to perform different migration methods! For “Migrating business data and applications” or “Runtime migration” you need infrastructure and integration experts, while you need application development and integration experts for “Application migration” or “Artifact migration”. Skilled application developers may also be required in case an incompatibility of an existing application with the new BAW product causes issues during a runtime migration.
As you can imagine, all of these aspects can add immensely to the overall effort. So the seemingly simple question “How am I going to do the migration?” actually has a huge impact on the migration efforts necessary to implement it.
Infrastructure layout and topology
Another key factor for the estimation of migration efforts is the topology of the existing installation and the intended topology of the target environment.
It’s quite obvious that the migration of one single-cluster BPM environment is significant less effort that the migration of four three- or four-cluster BPM environments (Development, Test, Staging, Production), the Production environment maybe even implemented with high availability and disaster recovery (HA/DR) capabilities and spread over more than one location. Starting with version 8.5 of BPM, it is also possible to change the topology during migration, e.g. changing an environment from a single-cluster setup to a three-cluster setup. Topology changes add extra effort to BAW migrations (architecture and configuration work).
System performance
The performance of the source and target systems used in the migration, as well as the performance of the network connecting them, is another important consideration. One of the most time-consuming activities during migration is the migration of data in the database, so the performance of the database server plays a key role. I have seen migrations where this step alone took more than 9 hours to complete. There were others where it even took several days!
System performance has also to be considered as migration effort when tuning the target environment of the BAW migration. Each version of BPM/BAW uses a different version of Java and WebSphere Application Server. As a result, each version needs to be tuned and optimized for scalability and performance in a different way.
Number and complexity of deployed applications
When migrating old applications to a new version of BAW, both, their number and their complexity, need to be considered when sizing the migration effort. In any case you will need application development experts to actively support the migration, either the original developers of the applications or at least people skilled enough to change them down to the latest detail if necessary. Only experts with that kind of skill will be able to resolve potential issues with migrated applications (e.g. APIs may have changed and thus the application code needs to be adjusted) or process and task instances causing problems during migration (e.g. dealing with orphaned tasks, failed events and the like).
Some applications may just continue to run in the new environment as they did in the old one, but you cannot always count on that. So, sizing in at least some effort for assessing every migrated application is a good practice.
Number and complexity of implemented customizations
In cases where significant customizations are implemented in the migration source environment, e.g. by customizing the out-of-the-box Process Portal, it is necessary to plan for some effort to migrate these changes over to the target environment. This applies also to cases where an external UI is implemented for the users, leveraging the documented product APIs. These custom UIs need to be thoroughly tested with the target version of the BAW migration in order to make sure that they still work.
Amount of available tooling
When you need to migrate a large number of applications you can either validate them manually in the target environment, performing test runs on each one of them, or you can execute a suite of automated regression tests which is significantly less effort. In the ideal case, those regression tests would have been developed together with the applications. In the next-best case, you would use the opportunity while migrating and develop such tests for future updates (as part of the migration testing). In the worst case, you never have any automated tests and always have to manually test the proper function of the applications.
Anything that you can prepare beforehand that helps to validate the migrated environment (another example would be automated tests for the required backend connectivity like the proper access to Web services) is immensely helpful and reduces migration effort.
Availability of necessary skills for the migration
I have already mentioned the importance of skilled application developers for the migration. Another key skill that is required is the database administrator (DBA). There have been migrations that took longer than necessary because key skills of the customer were not available when they were needed. That’s one of the reasons why a good project manager looking after such things is one of the foundations for a successful migration project.
Amount and quality of available migration documentation
As with any other IT project, proper preparation and documentation are key factors for success. You can save a lot of migration effort when you have all necessary information available before starting each migration activity. Lists with all IP addresses, ports, server names, etc. as well as lists with all application and module names, etc. should be there in every migration project.
Best practices for sizing BAW migrations
Even though the estimating game is not an easy one, there are some heuristics and best practices that can be applied in order to get a good understanding of the size and complexity of a given migration project.
Installing the new environment(s)
The installation of one clustered BAW environment (base installation only, no complex backend integration, no tuning) typically takes something between 3 and 5 days, depending on the complexity of the environment (number of nodes, availability of required resources). Again, this is one environment – Development, Test, Staging OR Production. The net effort gets bigger with every environment added. Some efforts can be parallelized (resulting in reduced elapsed time), but the net effort remains the same.
This effort does not include configuration for high availability and disaster recovery (HA and DR) or performance tuning of the target environments.
Additional configuration and tuning of the environment(s)
As mentioned already, depending on the BAW product version of the target environments, different settings need to be applied in order to optimize the system performance. This effort goes on top of the basic installation effort.
Please note that tasks to performance-tune the target environment need to be performed before (Operating System and networking setup, etc.), during (e.g. some settings in the configuration of the target deployment environment) and after the BAW migration (e.g. monitoring and fine-tuning of JVM settings).
Configuring a target environment for high availability and/or disaster recovery also introduces very significant extra effort. As a rule-of-thumb, the effort estimate for the environment setup should be doubled compared to a non-HA/DR environment. The actual effort depends on the level of high availability and resiliency to be achieved.
“Migrating business data and applications” (configuration, instances and data)
The migration of the BAW runtime is a key element in the overall migration procedure. Following the installation of the product binaries and other preparation steps, this is where the existing configuration, applications and data get moved. Most issues occur in this phase of the migration, sometimes requiring rolling back the migration, resolving the issue and repeating several migration steps. If properly planned, rollbacks of migrations should only occur when developing a migration routine (using a Test or Staging environment). It should definitely be avoided for migrations of Production environments which typically happen during rare, well-defined maintenance windows. In developing the runtime migration routine for a given Production environment, the goal is to minimize the required efforts and elapsed time, so that the migration and the subsequent regression / end-to-end test fit into the maintenance window / migration weekend.
Migrating the databases (part of “Migrating business data and applications”)
To get an estimate for the expected duration of the database migration, it is a good practice to use the time required to fully copy the database (over the same network that is used during the migration) and add some buffer to be on the safe side (e.g. 10-20%). This is for runtime migrations; for artifact migrations the database effort is part of the installation (creation of new databases).
Starting with BPM V8.5, clones of the existing databases can be used for the migration. This adds much more flexibility to the migration routine and greatly simplifies migration rollbacks. Cloning the databases is equivalent to taking a database backup (which is mandatory for migrations anyway), so no additional effort is introduced by this capability.
Migrating existing applications (“Artifact migration”)
In good cases, a runtime migration might successfully move all deployed applications and their running instances to the new environment. In many cases though, applications need to be adjusted in order for them to run with the new BAW code. It is very hard to estimate how likely it is that an application needs to be modified. Very significant application development skills and knowledge about the changes introduced between the source and target version of the migrated product are required to make this assessment. Therefore, the infrastructure AND application development teams of the customer need to be involved in the planning of a migration as early as possible! At some point, each application needs to be migrated in order to fully leverage the capabilities of the new BAW version and also in order to make it more future-proof (in line with the BAW product strategy).
The following heuristics for the effort to analyze and, if necessary, adjust existing applications have worked well for me (and others) in the past:
• Small/simple application: 1 day (including documentation)
• Medium sized application: 3 days (including documentation)
• Large/complex application: 5 days (including documentation)
• Very large/complex application: 2 weeks or more (including documentation)
Please note that these numbers are net effort, not elapsed time! In addition, the tasks cannot be arbitrarily executed in parallel.
Development environments vs. Test, Staging and Production environments
The migration of Development environments (Process Centers) is typically rather different from the migration of Test, Staging and Production environments (Process Servers). Process Centers are almost always migrated using “Artifact migration” whereas Process Servers (especially for Production environments) tend to be migrated using “Migrating business data and applications” more often.
In order to get an estimate for the migration efforts for a Production environment, take the time for all migration activities in the first migrated Process Server environment and determine how similar the environments are. The migration of the Test or Staging environments is almost always more effort and more complex than the Production one because:
• Typically, more applications / templates are deployed (but fewer instances exist).
• Test and Staging environments tend to be less powerful (but also less utilized) than Production environments.
In scenarios with lots of application development, this might be different.
It is a good practice to make the development where the migration routine is developed, tested and documented as similar to Production as possible!
• Extra effort is needed to document the necessary migration steps, serving also as instructions for follow-on migrations.
• Extra effort is needed to resolve issues encountered during migration (hopefully not to be repeated in the other environments).
• Extra effort is needed to educate the professionals running the migration (not necessary for the follow-on migrations unless the professionals change in between).
• Initial effort is to be invested for the migration planning. This can be re-used considerably for follow-on migrations.
Testing
Testing the migrated binaries, artifacts and processes is very often underestimated!
Still, it is a crucial part of the migration project and should be included in the estimate. This applies to all forms of BAW environments. Obviously, application testing should be done and completed in the Development and Test environments before rolling out the migrated applications to the Staging and Production environments. Infrastructure testing has to be done for all environments.
Configuration tuning should be tested in Development and Test before applying the validated tuning to Staging and Production.
Summary
I hope that the insights presented in this article make it easier to answer the rather difficult question: “How long will it take to get me the new BAW?”
Activity
|
Magnitude of Effort (net effort)
|
| Migration planning, project management and documentation |
Depends on the scale and complexity of the migration project; at least a few days, typically much more |
| Installation of the target environments |
Typically, at least 3-5 days per environment (including configuration); double the estimate for an HA/DR setup |
| Implementation of required integrations (LDAP, backend applications, etc.) |
Depends on the complexity of the environment; at least a few days |
| Performance tuning |
At least 1-3 days per environment with some ongoing effort after the migration (monitoring + fine-tuning) |
| Migration of customizations (e.g. Process Portal) |
Depends on the size and complexity of the implemented customizations; at least a few days for testing the Process Portal with the target version of BAW |
| “Migrating business data and applications” (configuration, instances and data) |
Typically, about 1-2 days, not including installation; exception: very large installations with huge amounts of data |
| Database migration (part of “Migrating business data and applications”) |
The elapsed time depends on the amount of existing data to be migrated; typically, a bit more than the time required to copy the database (add a 10-20% buffer for safety) |
| Migration of applications (“Artifact migration”) |
Depends on the size and complexity of the application and its compatibility with the new BAW;
S: 1 day per application
M: 3 days per application
L: 5 days per application
XL: 10 days or more per application |
| Testing |
Depends on the number and complexity of applications; at least a few days in non-Production environments, 0.5-1.0 days of regression testing in Production |
Remember: always run BAW migrations as projects, with a project manager, a project plan and proper tracking of the project’s progress. And involve IBM Services and/or a skilled IBM business partner.