Hi,

What are our options for sharing data between different case activities that are implemented as BPMN processes?

Case properties are currently limited to the level of complexity of the data to be represented. We will therefore not use case properties to represent complex data, but only simple data types that will be propagated for access at case level, such as for search indexing, lists and case detail.

This leaves us with the following options for storing case information to be shared between activities:

1. Shared Business Object where the key is stored in the case properties and sent to the case that needs access.
2. Same as above but an external storage (ex Redis) of the data and a key that is handled at the case level.
3. Store a blob of data, represented as a string at the case level that can be distributed by the case to each activity who will deserialize/serialize it to/from the case instance.

We will have concurrency and data life cycling problems regardless of the solution above, as long as we cannot be offered complex data structures at the case level.

How do you think about this?

------------------------------
Jonas
------------------------------

More often than not we end up with external storage (usually just a simple database, nothing as fancy as Redis, if you want no sql for this application I might look at CouchDB instead). External storage is going to be the most flexible and can adapt to your requirement (instead of you needing to adapt to the limitations of the in-built options). It is also fairly common to use a long string, blob property or content element on the FileNet side to store JSON or XML formatted data. One thought is that if you store XML as a content element (i.e. case attachment) you can use xpath queries if you have full text search (Content Search Services) deployed. The other thought is to use FileNet's generic object modeling capabilities to create a collection of linked objects that represent your complex data.

All of that said, the external datastore is definitely the simplest and most common answer.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------

Thanks Eric.

Interesting because I imagined an external database as the least attractive option, since it requires explicit lifecycle management of that data. 

I guess this question is only interesting depending on how you design your solutions. If you have a case on top of a long-lived BPM process, data management is not an issue. You can keep the data in your main process and pass it by invoking sub-processes. But if you invoke the sub-processes via the case, you are faced with this problem.

It would be interesting to take part in any improvements of BAW, if anything is planned. I would prefer that the same type system used in BPM could also be implemented at the case level, without having to invent something that will later be subject to change.

Only time will tell.

------------------------------
Jonas
------------------------------

Original Message:
Sent: Tue August 24, 2021 08:22 AM
From: Eric Walk
Subject: Share data between case activities in BAW

More often than not we end up with external storage (usually just a simple database, nothing as fancy as Redis, if you want no sql for this application I might look at CouchDB instead). External storage is going to be the most flexible and can adapt to your requirement (instead of you needing to adapt to the limitations of the in-built options). It is also fairly common to use a long string, blob property or content element on the FileNet side to store JSON or XML formatted data. One thought is that if you store XML as a content element (i.e. case attachment) you can use xpath queries if you have full text search (Content Search Services) deployed. The other thought is to use FileNet's generic object modeling capabilities to create a collection of linked objects that represent your complex data.

All of that said, the external datastore is definitely the simplest and most common answer.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------

I'm definitely interested to see where some of this goes in the future. The integration of the Case constructs with the BPM constructs has been getting better and better over time, but it's still not a perfect fit b/c Case needs to be compatible with the FileNet workflow engine too and it has a more restrictive type model. My historical knowledge is all on the FN side of the house, so I suppose I'm just used to it.

The thing to remember about Cases is that, logically, a Case is not a long-lived workflow, it's a folder in the Content Repository (a special kind of magic folder, but a folder nonetheless). So, the solution to things in Case world are more often found in the FileNet object model (which is why I mentioned that you could do it with custom objects or document objects). But, because with complex data structures it's often challenging to model them in FileNet because you end up needing a collection of linked objects, it is historically more common to do it with an external database.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------

Original Message:
Sent: Tue August 24, 2021 01:45 PM
From: Jonas Ekstrom
Subject: Share data between case activities in BAW

Thanks Eric.

Interesting because I imagined an external database as the least attractive option, since it requires explicit lifecycle management of that data. 

I guess this question is only interesting depending on how you design your solutions. If you have a case on top of a long-lived BPM process, data management is not an issue. You can keep the data in your main process and pass it by invoking sub-processes. But if you invoke the sub-processes via the case, you are faced with this problem.

It would be interesting to take part in any improvements of BAW, if anything is planned. I would prefer that the same type system used in BPM could also be implemented at the case level, without having to invent something that will later be subject to change.

Only time will tell.

------------------------------
Jonas
------------------------------


Original Message:
Sent: Tue August 24, 2021 08:22 AM
From: Eric Walk
Subject: Share data between case activities in BAW

More often than not we end up with external storage (usually just a simple database, nothing as fancy as Redis, if you want no sql for this application I might look at CouchDB instead). External storage is going to be the most flexible and can adapt to your requirement (instead of you needing to adapt to the limitations of the in-built options). It is also fairly common to use a long string, blob property or content element on the FileNet side to store JSON or XML formatted data. One thought is that if you store XML as a content element (i.e. case attachment) you can use xpath queries if you have full text search (Content Search Services) deployed. The other thought is to use FileNet's generic object modeling capabilities to create a collection of linked objects that represent your complex data.

All of that said, the external datastore is definitely the simplest and most common answer.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com

Eric,

I see clear benefits of storing data externally, especially with regard to information security. Passing sensitive information between processing units (BAW, Excel, Email) involves increased risk. It is better to provide access to data when needed through central storage and access control of the information. But it stipulates that your architecture has services that can offer your processes the right information at the right time. In our case the business processing of customer data that was originated from a web-based form. The full form data should not be passed on to BAW, but preserved in the application and exposed as a service. Just passing events with identifiers to lookup the data.

I not sure we will build external storage of data solely to deal with what I consider to be shortcomings in the product. Then I rather believe in a pattern in accordance with what Mattias Green proposes where we nevertheless use the capabilities of the product to store documents within the context of a case.

------------------------------
Jonas
------------------------------

Original Message:
Sent: Tue August 24, 2021 01:58 PM
From: Eric Walk
Subject: Share data between case activities in BAW

I'm definitely interested to see where some of this goes in the future. The integration of the Case constructs with the BPM constructs has been getting better and better over time, but it's still not a perfect fit b/c Case needs to be compatible with the FileNet workflow engine too and it has a more restrictive type model. My historical knowledge is all on the FN side of the house, so I suppose I'm just used to it.

The thing to remember about Cases is that, logically, a Case is not a long-lived workflow, it's a folder in the Content Repository (a special kind of magic folder, but a folder nonetheless). So, the solution to things in Case world are more often found in the FileNet object model (which is why I mentioned that you could do it with custom objects or document objects). But, because with complex data structures it's often challenging to model them in FileNet because you end up needing a collection of linked objects, it is historically more common to do it with an external database.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com
------------------------------


Original Message:
Sent: Tue August 24, 2021 01:45 PM
From: Jonas Ekstrom
Subject: Share data between case activities in BAW

Thanks Eric.

Interesting because I imagined an external database as the least attractive option, since it requires explicit lifecycle management of that data. 

I guess this question is only interesting depending on how you design your solutions. If you have a case on top of a long-lived BPM process, data management is not an issue. You can keep the data in your main process and pass it by invoking sub-processes. But if you invoke the sub-processes via the case, you are faced with this problem.

It would be interesting to take part in any improvements of BAW, if anything is planned. I would prefer that the same type system used in BPM could also be implemented at the case level, without having to invent something that will later be subject to change.

Only time will tell.

------------------------------
Jonas


Original Message:
Sent: Tue August 24, 2021 08:22 AM
From: Eric Walk
Subject: Share data between case activities in BAW

More often than not we end up with external storage (usually just a simple database, nothing as fancy as Redis, if you want no sql for this application I might look at CouchDB instead). External storage is going to be the most flexible and can adapt to your requirement (instead of you needing to adapt to the limitations of the in-built options). It is also fairly common to use a long string, blob property or content element on the FileNet side to store JSON or XML formatted data. One thought is that if you store XML as a content element (i.e. case attachment) you can use xpath queries if you have full text search (Content Search Services) deployed. The other thought is to use FileNet's generic object modeling capabilities to create a collection of linked objects that represent your complex data.

All of that said, the external datastore is definitely the simplest and most common answer.

------------------------------
Eric Walk
Senior Technical Architect

O: 617-453-9983 | NASDAQ: PRFT | Perficient.com

Hi all.

I think I already know the answer to this, but could you give us an example of what "complex" looks like in your example?

As you know, today we have the base (aka flat) properties associated with the case and the case Business Objects which allow one more level of a group of repeating property groups. Internally, the latter is persisted in the CPE using the abstract dependant object feature.

------------------------------
DAVE PERMAN
------------------------------

Hi Dave,

We have several examples, but the most current is the form data as input to a customer onboarding process. It includes, among other things, ownership structure which is a hierarchical data structure, but also other nested structures that go deep - deeper than 1 level. As I said, we want to make the data available for the activities that are initiated from the case - automatically, manually and discretionarily - implemented as both new (by val) and existing (by ref) BPMN processes.

A design alternative would of course be to handle the data externally from the process and consume it when needed. But in this case, it would have been desirable to store it within the transactional lifecycle scope of the case itself and preferably in a strongly typed object structure that becomes available at both the case level and for the underlying processes without explicit (de)serialization.

Any ideas how to proceed?

------------------------------
Jonas
------------------------------

Original Message:
Sent: Wed August 25, 2021 11:16 AM
From: DAVE PERMAN
Subject: Share data between case activities in BAW

Hi all.

I think I already know the answer to this, but could you give us an example of what "complex" looks like in your example?

As you know, today we have the base (aka flat) properties associated with the case and the case Business Objects which allow one more level of a group of repeating property groups. Internally, the latter is persisted in the CPE using the abstract dependant object feature.

------------------------------
DAVE PERMAN
------------------------------

Great discussion, guys! I think a hybrid of the above suggested alternatives would be a good idea.

1. Model (common) Business Object in Workflow Designer.
2. Implement generic Toolkit uitlility to CRUD and serialize/deserialize BO to/from JSON. Perpahs with metadata such as schema version, location etc. Persist in CPE (let client decide where).
3. Reuse Toolkit across solutions. Persist/share BOs on the desired scope (customer level, parent case, current case, current process/activity, etc).

The usual schema evolution concerns apply here as well, and can be added to existing solution development practices - ie, if you change BO YXZ, make sure to also address forward/backward compatibility and/or migration concerns.

Persistence mechanism above can of course be pluggable, but to me it makes the most sense to persist as close to the case as possible, at the desired scope (customer level, parent case, current case/process/activity). FileNet supports both optimistic and pessimistic locking, so I'd suggest using that when appropriate.

------------------------------
Mattias Green
------------------------------

Original Message:
Sent: Wed August 25, 2021 01:35 PM
From: Jonas Ekstrom
Subject: Share data between case activities in BAW

Hi Dave,

We have several examples, but the most current is the form data as input to a customer onboarding process. It includes, among other things, ownership structure which is a hierarchical data structure, but also other nested structures that go deep - deeper than 1 level. As I said, we want to make the data available for the activities that are initiated from the case - automatically, manually and discretionarily - implemented as both new (by val) and existing (by ref) BPMN processes.

A design alternative would of course be to handle the data externally from the process and consume it when needed. But in this case, it would have been desirable to store it within the transactional lifecycle scope of the case itself and preferably in a strongly typed object structure that becomes available at both the case level and for the underlying processes without explicit (de)serialization.

Any ideas how to proceed?

------------------------------
Jonas
------------------------------


Original Message:
Sent: Wed August 25, 2021 11:16 AM
From: DAVE PERMAN
Subject: Share data between case activities in BAW

Hi all.

I think I already know the answer to this, but could you give us an example of what "complex" looks like in your example?

As you know, today we have the base (aka flat) properties associated with the case and the case Business Objects which allow one more level of a group of repeating property groups. Internally, the latter is persisted in the CPE using the abstract dependant object feature.

------------------------------
DAVE PERMAN