Content Management and Capture

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hi Florian,

first, restore the object store DB isn't enough to rollback if your deployment includes files (entry templates, ...) which are stored on file system.
Depending on your infrastructure, as example, may be the VMWare snapshot may help to rollback the environment.
If you're in an OCP environment, you may apply the deployment in a parallel CPE POD and when the deployment is valid, make it available.

Best regards

------------------------------
Yannick Martin
------------------------------

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hello Yannick,

Yes, you are totally true concerning the file system dependency, even if we can imagine to use a database store for administrative assets (entry templates for example).
I like the idea of a parallel CPE POD but unfortunately, when you add content assets (classes, property templates, choice lists, etc.) or process assets (queues, in-baskets, etc.), all this stuff is stored in one unique database and you can't restore it unless you are ready to lose users actions during the period.
Or you mean a CPE POD with a copy of original database but how can you reconcile datas between databases (the original and the copy) ?
It is the same for VMWare snapshot and the consistency between changes made on a VM and database...


------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Original Message:
Sent: Thu December 09, 2021 03:30 AM
From: Yannick Martin
Subject: FileNet Deployment Manager and CI / CD

Hi Florian,

first, restore the object store DB isn't enough to rollback if your deployment includes files (entry templates, ...) which are stored on file system.
Depending on your infrastructure, as example, may be the VMWare snapshot may help to rollback the environment.
If you're in an OCP environment, you may apply the deployment in a parallel CPE POD and when the deployment is valid, make it available.

Best regards

------------------------------
Yannick Martin

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

We tend not to use FDM as it really isn't suitable for CI/CD. We tend to use in-house tools (often customized to the client) to accomplish the task. All the APIs you need are there in Jace.jar.

We usually use some outside version control (either configuration management or just a git repo) for the configurations of classes and properties, etc. The big challenge is idempotency not rollback (if your tool can manage idempotency it can always roll forward a prior version of the configuration and achieve the prior state). Making sure to use the batch apis (i.e. to commit transactionally) is one key, but also understanding the order of operations and interrelatedness of various configuration objects is critical.

We've made this work in both traditional and container environments using Ansible, Chef and custom Spring Batch jobs, it's just a question of the needs and environment. I'm hoping IBM is working on some more OOTB options (maybe that work like the CI/CD options available with ADP?).

------------------------------
Eric Walk
Director

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hello Eric

What are you writing is very interesting, clearly FDM is lacking features to make continous deployment, so we hope like you IBM is working on it.

If I understood correctly so you created specific tools based on CE API. Idempotency can be very difficult to achieve with CE assets deployment.
For exemple : I have a document class with 3 properties (version 1) and I want to deploy a 4th property template on this document class (version 2). So I can use API to deploy version 2 but typically I have some difficulty to see how this operation can be idempotent.
Can I ask you Eric for elaborating a little bit more your statement for this example please ? It could be very useful for my understanding.
Thank you very much.

Regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Original Message:
Sent: Thu December 09, 2021 08:32 AM
From: Eric Walk
Subject: FileNet Deployment Manager and CI / CD

We tend not to use FDM as it really isn't suitable for CI/CD. We tend to use in-house tools (often customized to the client) to accomplish the task. All the APIs you need are there in Jace.jar.

We usually use some outside version control (either configuration management or just a git repo) for the configurations of classes and properties, etc. The big challenge is idempotency not rollback (if your tool can manage idempotency it can always roll forward a prior version of the configuration and achieve the prior state). Making sure to use the batch apis (i.e. to commit transactionally) is one key, but also understanding the order of operations and interrelatedness of various configuration objects is critical.

We've made this work in both traditional and container environments using Ansible, Chef and custom Spring Batch jobs, it's just a question of the needs and environment. I'm hoping IBM is working on some more OOTB options (maybe that work like the CI/CD options available with ADP?).

------------------------------
Eric Walk
Director

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

I don't 100% understand how it's accomplished, to be honest, I know it wasn't simple. My high-level understanding is that we built tooling to interrogate the current state of the system and determine the delta to the ideal state from the authoritative configuration source. Then we can apply appropriate changes to reconcile the system to the ideal state. This sometimes means adding new configurations (i.e. adding a new property template) but other times it means removing something. It does get tricky because there are times that something should be removed but cannot be and that's where this all breaks down and we must concede it won't be perfect...

The operation itself is not idempotent, but we build a wrapper around it to make it so. In this way we can run the entire pipeline over and over and empower it to decide what tasks it needs to perform, if any, to reconcile the current state to the desired state. It's like how the Operator works in an OpenShift deployment. It's not perfect...

Best,
Eric

------------------------------
Eric Walk
Director

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------

Original Message:
Sent: Fri December 10, 2021 08:57 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello Eric

What are you writing is very interesting, clearly FDM is lacking features to make continous deployment, so we hope like you IBM is working on it.

If I understood correctly so you created specific tools based on CE API. Idempotency can be very difficult to achieve with CE assets deployment.
For exemple : I have a document class with 3 properties (version 1) and I want to deploy a 4th property template on this document class (version 2). So I can use API to deploy version 2 but typically I have some difficulty to see how this operation can be idempotent.
Can I ask you Eric for elaborating a little bit more your statement for this example please ? It could be very useful for my understanding.
Thank you very much.

Regards,
Florian Kiebel

------------------------------
Florian KIEBEL
Practice Leader
Amexio

Original Message:
Sent: Thu December 09, 2021 08:32 AM
From: Eric Walk
Subject: FileNet Deployment Manager and CI / CD

We tend not to use FDM as it really isn't suitable for CI/CD. We tend to use in-house tools (often customized to the client) to accomplish the task. All the APIs you need are there in Jace.jar.

We usually use some outside version control (either configuration management or just a git repo) for the configurations of classes and properties, etc. The big challenge is idempotency not rollback (if your tool can manage idempotency it can always roll forward a prior version of the configuration and achieve the prior state). Making sure to use the batch apis (i.e. to commit transactionally) is one key, but also understanding the order of operations and interrelatedness of various configuration objects is critical.

We've made this work in both traditional and container environments using Ansible, Chef and custom Spring Batch jobs, it's just a question of the needs and environment. I'm hoping IBM is working on some more OOTB options (maybe that work like the CI/CD options available with ADP?).

------------------------------
Eric Walk
Director

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

Hi Florian

 I think this is an interesting topic and I would see various aspects:
1) using a simple mechanism to apply structural changes to an Objectstore. As some of the replies include I have used a set of text files for each change which is then interpreted by some java classes which are then calling the methods from the jace.jar. In the code I always check for the prerequisites and then apply the change. That is pretty straight forward but the effort to maintain such framework is time consuming.
2) After you have applied the changes and you decide to return to a previous version - now the problem gets ugly. As long as this is a dev system you can always start from a baseline and this might be fine, but for a productive system where you have a few million records that might be quiet tough. Think about the fact you would like to remove a property template from a class but you have already ingested a few thousand documents in an objectstore which contains millions of documents.
Even if you manage to update the meta data to null and then trying to remove it from the class you still have the column in the underlaying docversion table. You reached here the point where the simple concept of a CI/CD pipeline might not work anymore. 

------------------------------
Dominik Baer
------------------------------

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------

The three options "do it manually", "write some code" or "use FDM" all work but first you have to understand what you have. Our product, Case Librarian (https://www.caselibrarian.com) , does that.

 It extracts configuration and statistical details from CPE, and other components, and helps you understand what you have. It generates documentation quickly, easily and consistently, including comparative information of your current configuration and FDM exports, for example. It generates graphical, textual and analysable information that allows you to know what you have, what you had, and what you may end up with. 

Whether you use any of the options, you still need to know you've made the right changes and understand how a fairly complex system is built, across multiple environments, with probably multiple other software dependencies.
Navigating ACCE requires you to remember things, only works one thing at a time, doesn't provide comparative or historical information, doesn't provide ranges, limits or impacts of chances. It's all up to the skill and expertise of your people. Case Librarian is our twenty years of expertise made easy. 

------------------------------
David Alfredson
------------------------------

Original Message:
Sent: Thu March 16, 2023 04:14 PM
From: Dominik Baer
Subject: FileNet Deployment Manager and CI / CD

Hi Florian

 I think this is an interesting topic and I would see various aspects:
1) using a simple mechanism to apply structural changes to an Objectstore. As some of the replies include I have used a set of text files for each change which is then interpreted by some java classes which are then calling the methods from the jace.jar. In the code I always check for the prerequisites and then apply the change. That is pretty straight forward but the effort to maintain such framework is time consuming.
2) After you have applied the changes and you decide to return to a previous version - now the problem gets ugly. As long as this is a dev system you can always start from a baseline and this might be fine, but for a productive system where you have a few million records that might be quiet tough. Think about the fact you would like to remove a property template from a class but you have already ingested a few thousand documents in an objectstore which contains millions of documents.
Even if you manage to update the meta data to null and then trying to remove it from the class you still have the column in the underlaying docversion table. You reached here the point where the simple concept of a CI/CD pipeline might not work anymore. 

------------------------------
Dominik Baer

Original Message:
Sent: Wed December 08, 2021 05:44 AM
From: Florian KIEBEL
Subject: FileNet Deployment Manager and CI / CD

Hello all,

In a "classic" world, when you want to deploy a new asset (a new document class for example) in Content Platform Engine, you have to use FileNet Deployment Manager in order to export this asset from your development environment and then import it in your production environment. Then import will translate your asset in changes to database tables structure and/or lines in database tables associated to the target object store.
Note that import needs to interrupt service generally in order to avoid perturbation induced by user actions.
When the import failed (generally complex one), you can have remnants of what was imported successfully. Then if you want to rollback completely, you have to restore database with all problems it can imply, the most important of them is unavailability of the service...

In a "DevOps" world where Continuous Integration and Continuous Deployment should take minutes or seconds and where rollback is also taken minutes or seconds, I would like to know how we can proceed with FileNet Deployment Manager.
Are there some tricks, tips in order to improve FDM for preventing service interruption and allowing quick rollback ?

If you have any propositions around this topic, I will be very happy you share it.

Thanks a lot

------------------------------
Florian KIEBEL
Practice Leader
Amexio
------------------------------