Introduction and Background

One of the great things about the open-source community is its togetherness and methods to innovate and deliver new features and the projects keep moving. A lot of times once a system is under active development, a version of a community project is identified to utilize and from there carry forward with your development. If the release you are working with has all or most of the features you need, sometimes things stay the same, for a long time!

What happens though in these systems when something like everyone's favorite Christmas 2021 story of Log4J exploitation comes out and your version is no longer being worked on in the community - do you migrate forward however many versions (if possible), fix your version yourself and hope to build it or hope the community gets to fixing it so you can update in place? There is a fourth option that helps here, which is to get Enterprise support on your open-source project, and in the case of Drools, jBPM and Kogito, that support can be provided by IBM through their enterprise build of the projects in IBM Business Automation Manager Open Edition. The product itself is built on the same principles as it was when the project was under Red Hat's lead in being completely open-source. What this means is shifts from community to enterprise can be fairly smooth from an end-user's perspective. In this blog we will walk through this process.

For the purposes of this blog, I am going to time travel back to March 2020. We're in the midst of the beginning of the COVID pandemic and companies are starting to realize they need more automation in their processes that used to be able to have humans interacting in person, but due to office shutdowns and other challenges are not able to do so, hopefully not hitting too close to home here! So you're the developer in charge of an IT On-boarding system that you want to use an open-source process engine and identified jBPM and Drools are your go to solution because you liked the capability to use a process engine and rules engine in the same packaging. You get your project up and running and everyone is happy that it's working smoothly. So smoothly in fact that you kind of just leave it untouched outside of the occasional change here and there for the next year and a half.

Fast forward to December 2021, Red Hat puts out a notice about log4j and exploits that can be done within it - jBPM and Drools were unaffected by the exploit due to their usage of SLF4J over log4j see this KIE Community blog about the log4j exploit, however your boss states that we're wanting to go to an enterprise support software model for all of our open-source software to avoid missing Christmas dinner again. Why do they want to do this, because had this exploit affected one of the enterprise solutions, Red Hat and now IBM, after the transition to the IBM Automation portfolio, produce fixes on supported versions of the software and in the case of log4j, the fixes required were produced mostly within 48 hours of the exploit being identified and released for their customers. In the community editions, this is not always possible since the production of major/minor versions isn't necessarily owned by any one party. Within the enterprise model, fixes can be pushed back and applied to all of these versions when vulnerabilities are identified. This is the beauty of an enterprise backed solution - applying a patch update versus a potential much larger migration effort. Now that we understand the background of why this is required, let's get into the meat of what is actually required to move from the community to the enterprise editions. 

What should we do first?

The model that IBM is utilizing with enterprise support is that a version of the community projects are identified and those effectively get branched and hardened for enterprise release. The hardening process includes a further testing, dependency and security scans, build compatibility matrixes, etc. Starting in the mid-7.x stream, it became very easy to identify which version of the community your enterprise release would match up to - in this case February 2020, we're going to say we're using Community version 7.29 which came out a few months earlier for it's initial release, but could be other versions in that time frame the procedure will be similar. So we've identified our version and see that it's got a similar version with enterprise support. So the first path in migration is to move to the most similar version of our community to enterprise pairing. When we browse Red Hat's repository of versions, we notice that this version is going to be their 7.6 version, which correlates to the 7.30 build in the community. This was found by using this Red Hat support entry outlining the versions that are incorporated with each version to determine that version. Now, there are two main considerations to the migration: the runtime itself and the persistence layer that backs it. When I migrate my instances, I try to identify which path will be what I deem the smoothest to the most current version. In the case of the identified version, so how do I do this? Let's start exploring that. The biggest thing to be aware of, in this migration, there will be some system downtime, but if handled correctly, it will be very quick to return to operation. You will not be updating a running instance, so please keep this in mind - there will be some kind of downtime.

Identifying the quickest component to upgrade - check the persistence layer first!

When you download jBPM from the community site, there are several things that get delivered with the package, the one we first want to look at within the installer package is the database schemas/ddls that are delivered with the release. In this case since there are several years of updates that will need to be done, whenever there is a database change from a version to version update, the update scripts are included in the bamoe-version-add-ons/rhpam-version-add-ons file as part of the product downloads. You will get these from Red Hat at Red Hat Downloads for Process Automation Manager, Passport Advantage to download 8.0 and IBM Fix Central for all current and future patch releases of BAMOE. In this case because we are going several years back, we will be going back to the Red Hat releases and bridging that with the IBM release, hopefully your move won't have to be this many steps and you won't have to do both, but I figured it was better to be as detailed as possible versus assuming where anyone reading this may be. These same steps once you get to the first version of Enterprise with Red Hat apply to the move for the shift to the most current releases delivered by IBM Business Automation Manager Open Editions so a two-for-one blog! So use the steps that you need and ignore the rest!

The downloadable file for the database that we need to acquire for each release at the minor version (7.X) will need to be done for both the add-ons zip [bamoe-version-add-ons.zip/rhpam-version-add-ons.zip] and the update zip [bamoe-version-update.zip/rhpam-version-update.zip]. As a part of this exercise, since our community release is between versions that were delivered as a part of Enterpirse, we will compare the database schemas to the associated version in the enterprise release - which will be found in the add-ons downloaded file. In the case of our situation here, we're looking at it being found within rhpam-7.6.0-add-ons.zip and when this archive is expanded, within the rhpam-7.6.0-migration-tool/ddl-scripts/[YOURDB] you can analyze the DDLs that are here. I used PostgreSQL in my sample environment, but you can check the others if you used a different. This was intentional as for both PostgreSQL and Oracle databases have a slightly different Springboot setup of the database when it was first created, but the path remains the same. These DDLs are the exact same! This is our ideal situation, we don't have a database migration in our first step! This is why the enterprise model that Red Hat before and IBM now uses is very helpful, by having frequent releases based on releases, you have fewer risks associated with the move - and then your patch updates fall within a stream. In this situation, let's say there was a database move in the closest versions, if this was the case, you would need to follow some of the data migrations (which there will be some talks of this in subsequent sections when we bring our environment to the most current versions refer to section [MIGRATION]). All of the database migrations will be handled the same regardless of chosen runtime. The process to move from version to version can be handled in sequence. So in the case of our situation, we moved from Community 7.29 to Enterprise 7.30 - this database schema was the same. So since these database schemas match, and we're planning on going to 8.0.3 of IBM Business Automation Manager, how do we get there from a persistence layer? This is relatively straight forward. Along the way there were several enhancements/fixes to the database schema, which need to be applied in sequence. To do this, you will use your preferred method of adding SQL statements of the appropriate distribution. 

All of the database migrations will be handled the same regardless of chosen runtime. The process to move from version to version can be handled in sequence. So in the case of our situation, we moved from Community 7.29 to Enterprise 7.30 - this database schema was the same. 

So since these database schemas match, and we're planning on going to 8.0.3 of IBM Business Automation Manager, how do we get there from a persistence layer? This is relatively straight forward. Along the way there were several enhancements/fixes to the database schema, which need to be applied in sequence. To do this, you will use your preferred method of adding SQL statements of the appropriate distribution. 

How to update the runtime database

For migrating the database, in my opinion the easiest way is to do it in sequence as most of the jobs are *very* small (like 2-4 lines of SQL). They are all mapped by the version progression so it would be fairly quick. So for instance, if we're running a PostgreSQL database, what we would need to do is the following recipe.

1. Access either Fix Central - requires an active subscription or there is the opportunity to try out the new Developer's Preview Edition , which will be going live not long after this blog goes out (and be subsequently updated with the live link and the terms of usage).
2. From here, depending on what you need to get, you will download a few files (this blog entry is using 8.0.3, but replace with most current version). The most common ones are highlighted below, but download what makes the most sense for your environment. In this blog entry's focus we are targeting a KIE Server deployment, if your deployment is different, adjust as required.
   
   1.  bamoe-8.0.3-add-ons.zip contains things like the Database Migration Scripts (found at path /bamoe-8.0.3-add-ons/upgrade-scripts/*Database*)
   2. bamoe-8.0.3-business-central-eap7-deployable.zip is if you are running Business Central on your JBoss Enterprise Application Platform (EAP)
   3. bamoe-8.0.3-kie-server-ee8.zip if you are running KIE Server on JBoss EAP.
   4. bamoe-8.0.3-maven-repository.zip if you need to have a disconnected loading of the Maven repository in your environment
   5. bamoe-8.0.3-offline.zip - to help build the Maven repository in an offline fashion
   6. bamoe-8.0.3-update.zip - the updates that get applied with the overall product set. 
   7. ILMT_for_IBMAMOE_8.0.3.zip is how IBM measures the usage deployed in an environment of the product
3. Now that we have those files we can start to migrate our database. So since we're using a PostgreSQL database, let's grab those SQL statements. Remember we started from community 7.29 - and we figured this most correlated to 7.30 in Enterprise released as the best match since it was the same schema, so if we navigate to bamoe-8.0.3-migration-tool/upgrade-scripts/postgresql we will use this to build our list of database updates we need to apply. Since this upgrade is occurring during an outage, we're going to apply them in sequence. So the way that I am running it is to run them as follows:
   1.  Connect to your database instance in your method of preference
   2. Apply the SQL scripts, in the repository attached to this demonstration (which does not do the entire migration, but parts of it) I have included a folder that you can run see which SQL files I am running. *Of note*: the PostgreSQL database moved from an OID storage to Bytea as the preferred method around 7.12 timeframe. There is not a migration to move from OID to Bytea provided at this time. If you are (as in the case here) moving from a version without Bytea, use the OID job **rhpam-7.12-to-7.13-oid.sql** to migrate forward, if using a more current release with Bytea storage the job **rhpam-7.12-to-7.13-bytea.sql** to migrate forward. 

An example of how to run these updates would look similar to the below sample code ran from the upgrade directory of your database of choice:

psql -h $dbhost -U $jbpmuser < rhpam-7.7-to-7.8.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.8-to-7.9.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.9-to-7.10.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.10-to-7.11.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.11-to-7.12.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.12-to-7.13-oid.sql
psql -h $dbhost -U $jbpmuser < rhpam-7.13-to-7.13.1.sql

Once you have ran these SQL statements, your datasource is now up to date. If yo uare running a different database than PostgreSQL, replace the psql command with your appropriate database and tooling of choice, the files should largely be the same.

Migrating the runtime - the it depends environment!

Now the real reason we're migrating is not typically the database, but because we need some kind of patches that have been applied to our runtime environment. This is the much more crucial aspect of this. Now this is not going to be a fully comprehensive guide to migrating things like Wildfly to JBoss Enterprise Application Platform and then jBPM, but focusing on the other aspects of it. I'm not a platform expert, so I'm not going to pretend to be one! I will often setup an entirely new JBoss instance versus running the update tool for EAP and this is largely because I don't have enough domain expertise to understand what is different between the two. You can do that method as you prefer based on your JBoss skills in your organization. This Red Hat Article on Migrating EAP will help with the migration, so I won't cover the specifics here. The recommendation for moving forward would be to get to the latest of each environment that you can get to. In the case of this blog's release date, IBM Business Automation Manager 8.0.3 is the most current version, which is based off the community release of 7.67.  There are two main ways to utilize the KIE Server, Spring Boot and through the use of JBoss Enterprise Application Platform (EAP). This section will cover situations, the first is going to be the shorter version, Spring Boot and then followed with EAP.

Spring Boot Migrations

## Spring Boot migrations 
If you designed your jBPM application based on the use of deploying the KIE Server with the use of Spring Boot, your migration path is quick quick from this point on out. The projects themselves are based on Maven architectures, which simplifies the process considerably (by design!). So in the case of bringing our application from Community 7.29 to our enterprise supported version of IBM Business Automation Manager 8.0.3. Depending on how your application is constructed in the pom.xml, this might be a very deployment if you're using either a Maven Bill of Materials or the KIE Server Spring Boot Starter.

These changes will require a potential update to your Maven **settings.xml** to reflect connecting to the Red Hat repository which is found at https://maven.repository.redhat.com/ga/ and can be added to your settings.xml with the following update if you're able to update and connect to external repositories. If mirrors are required, that's a whole separate discussion, but an example of a mirror can be viewed at BAMOE University in the following lab as an example.

<profile>
	<id>red-hat-maven</id>
	<repositories>
		<repository>
			<!-- Red Hat Maven Repository-->
			<id>jboss-ga-repository</id>
			<url>https://maven.repository.redhat.com/ga/</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>
	</repositories>
	<pluginRepositories>
		<pluginRepository>
			<id>jboss-ga-plugin-repository</id>
			<url>https://maven.repository.redhat.com/ga/</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>
</profile>

f you are using a Bill of Materials build you would use the following sourced from the Red Hat Maven Repository:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.redhat.ba</groupId>
            <artifactId>ba-platform-bom</artifactId>
            <version>8.0.3.redhat-00002</version> <!-- BOM version for BAMOE 8.0.3 -->
            <scope>import</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

If you are using the Spring Boot Starter, you will update your pom.xml to build your project (e.g. if you started from start.jbpm.org) then you would need to update the version found in the provided pom.xml of any service that has jBPM or Drools dependencies. This is not guaranteed, but of the typical three projects produced  (model, kjar, service), the ones that by default have the libraries used are the kjar project and the service project. In these projects you would update the dependency version found in the **properties** tag for the *version.org.kie* to reflect the version required, at the time of blog writing this was 7.67.2.Final-redhat-00006 as referenced from this pom.xml snippet:

<properties>
	<version.org.kie>7.67.2.Final-redhat-00006</version.org.kie>
	<maven.compiler.target>1.8</maven.compiler.target>
	<maven.compiler.source>1.8</maven.compiler.source>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

After these changes are completed, your project will bring in the latest attached dependencies and if there are any CVEs that were addressed within those updates, they should not be coming up in your scanners. So you can run a `mvn clean install` and everything will start building locally or run it through your pipeline to deploy and test it out. Once this is deployed, you will have successfully moved your project from the community version of jBPM to the enterprise version with support provided by IBM for the project now with a number to call when you have issues or a means to open up support cases to provide means outside of the normal community paths.

Migrating on JBoss Enterprise Application Platform (EAP)

One of the most common ways of having KIE Server deployed is on top of JBoss Enterprise Application Platform (EAP). This guide will not go through migrating the Open-source Wildfly to the Red Hat supported version of JBoss EAP, but you can refer to these guides here from Red Hat Support and the JBoss Community on what would be required to migrate the underlying platform. Just as a side note, per the subscription terms and conditions for IBM Business Automation Manager Open Edition found here in Section 7 is an included subscription for Red Hat JBoss Enterprise Application Platform tied to the usage of the Open Editions software's usage so the underlying application platform would be supported in this case too.

So for the purposes of this entry, we're going to assume you're now on JBoss EAP 7.4, as this is the version you would want to be on for IBM Business Automation Manager 8.0.3. Revisiting our situation from the start of this article way back when we started, we are currently at 7.6 and moving towards 8.0.3. In your environment's deployment, you would deploy the necessary components you are utilizing (e.g. Business Central and KIE Server) to the EAP instance to have the 7.6 versions deployed in those environments. So you would undeploy the community version that you have (or in a clean environment deploy a new instance of 7.6 to it) if you want to test it as is. The situation where you would want to validate the like for like comparison, you would deploy 7.6 and follow the next few steps. Since your database is already migrated, hypothetically you could go direct to IBM Business Automation Manager 8.0.3 directly after moving the data source.

If you have decided to do the instance where you need to step through the versions to validate the updates, you will run the following sequence of events to get to the most current starting with the Red Hat releases and then shifting to the IBM releases (remember we started this project back in 2020, 3+ years have passed so a lot of steps are being shown here). In these instances you would re-run the *apply-updates* job on each component for each version. This would be done in series, because the patch updates are targeted to address the components and not delivered all at once in the infrastructure. Before running the updates, please review this support documentation on the apply-updates if there are customized files in your deployment you need to ensure do not get replaced. So in this sequence you would update through the usage of the files found by downloading all of the associated  rhpam-version-update.zip retrieved from Red Hat Downloads for Process Automation Manager:

1. Red Hat Process Automation Manager 7.6 to 7.7 [rhpam-7.7-update.zip]
2. Red Hat Process Automation Manager 7.7 to 7.8 [rhpam-7.8-update.zip]
3. Red Hat Process Automation Manager 7.8 to 7.9 [rhpam-7.9-update.zip]
4. Red Hat Process Automation Manager 7.9 to 7.10 [rhpam-7.10-update.zip]
5. Red Hat Process Automation Manager 7.10 to 7.11 [rhpam-7.11-update.zip]
6. Red Hat Process Automation Manager 7.11 to 7.12 [rhpam-7.12-update.zip]

All of these would be ran applying to the components installed, for example under each folder's version you would target your application server and run the following when deploying updates to the KIE Server.

./apply-updates.sh APP-SERVER-HOME/standalone/deployments/kie-server.war    rhpam-kie-server-ee8

This will get you to the most version before IBM's release of 8.0 that was in June 2022. From here you would have a few more patches to apply. These would be those associated with the IBM release and the year of patches since the initial version. So in this case you would apply the fixes doing:

1. Red Hat Process Automation Manager 7.12 -> IBM Business Automation Manager Open Edition 8.0 [IBAMOE-8.0-UPD.zip]
2. IBM Business Automation Manager Open Edition 8.0.0-> 8.0.1 [bamoe-8.0.1-update.zip]
3. IBM Business Automation Manager Open Edition 8.0.1->8.0.2 [bamoe-8.0.2-update.zip]
4. IBM Business Automation Manager Open Edition 8.0.2->8.0.3 [bamoe-8.0.3-update.zip]

With those steps completed, you have now applied over 3.5 years of updates in a few minutes. I know it looks like a lot, but the steps are capable of being automated and repeatable very easily.

Conclusion

There was a lot covered in this blog and really this is only the first steps in the migration - you have your underlying application that can be updated as well (in Spring this is more or less already handled but EAP deployments you should update the Maven Dependencies in the project too to capture these updates as well grabbing all transitive dependencies along the way!). This is largely the infrastructure and some of the considerations. Hopefully this blog entry helps you get to the most current release. As with all open-source software, please feel free to reach out and comment and we can help make this process more seamless! Let me know if you have questions!