View Only

Cloud Pak for AIOPS - File Observer Best Practices

By Gurpreet Kaur posted Fri April 26, 2024 07:20 PM

Like

Cloud Pak for AIOPS - File Observer Best Practices

In AIOPS deployment, oftentimes we use file observer to bring in topology information. In production environments this information changes as the connectivity or underlying infrastructure changes. So it is recommended to run file observations after the change in underlying topology or regularly on schedule. If the file observer file is not properly created, this can lead to misleading topology data like leftover edges from previous runs or missing properties of resources, etc. This blog will take you through the best practices to manage resource and edge lifecycle so you can create appropriate topology files and have up-to-date topology information available for correlation after every observation.

Best Practices to define file observer, resources & edges:

When defining the file observer job, the file name must be the same for subsequent observation runs.

Resource Lifecycle:

Each resource within the scope of a provider has its lifecycle managed by the job that modeled it.
Load jobs expect to take snapshots of the entire system.
- If a load job finds resources X, Y, and Z on one run and the next run finds resources A, B, and X, then Y and Z will be deleted.
insertReplace/PUT will remove any properties not listed in the new entry (exception is a placeholder with only uniqueId + _references). This option should be used always when you expect subsequent runs to replace existing resource properties and edges hence keeping it up-to-date.
insertUpdate/POST will add to existing properties. This option should only be used if you want to merge properties from different source files for a single resource.

Edge Lifecycle:

To maintain edge lifecycle:
- always model edges using _references nested within a resource.
- Always use PUT / insertReplace semantics.
- Try to model edges in a consistent direction.
Edge type and direction per resource provide the replacement scope.
- With insertReplace, all edges of the same edgeType and direction will be replaced.
- Direction is implied by _toUniqueId/_fromUniqueId or, to remove all edges, using _direction.
Do not use E: format to define edges as it results in issues with edge lifecycle. Define edges using "_references".
Edge lifecycle can be maintained independently of properties using a payload with only uniqueId & _references for example:
- V:{"_operation":"InsertReplace","uniqueId":"B","_references":[{"_edgeType":"connectedTo","_direction":"out"},{"_edgeType":"dependsOn","_direction":"out"}]}

Use case: Add, remove, and update resources/edges in File Observations.

Note: From NOI 1.6.9 / AIOps 4.1/ ASM1.1.18 "_operation"="InsertReplace" is the default and recommended. So you can omit "_operation":"InsertReplace" in the resource definition entry

Example 1: Define resources and edges in the file observer

File: edgelifecycle.txt (version 1)

# model A connectedTo B and C

V:{"_operation":"InsertReplace","uniqueId":"A","entityTypes":["host"],"name":"A","_references":[{"_edgeType":"connectedTo","_toUniqueId":"B"},{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_direction":"out"}]}

# model B

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B"}

# model C

V:{"_operation":"InsertReplace","uniqueId":"C","entityTypes":["host"],"name":"C"}

# Add connectedTo from B to C

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B","_references":[{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_direction":"out"}]}

Example 2: Change link type between A and C from connectedTo to dependsOn

File: edgelifecycle.txt (version 2)

# model A connectedTo B and C

V:{"_operation":"InsertReplace","uniqueId":"A","entityTypes":["host"],"name":"A","_references":[{"_edgeType":"connectedTo","_toUniqueId":"B"},{"_edgeType":"dependsOn","_toUniqueId":"C"}]}

# model B

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B"}

# model C

V:{"_operation":"InsertReplace","uniqueId":"C","entityTypes":["host"],"name":"C"}

# Add connectedTo from B to C

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B","_references":[{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_direction":"out"}]}

Example 3: Remove all connectedTo edges from A, leaving just dependsOn

File: edgelifecycle.txt (version 3)

V:{"_operation":"InsertReplace","uniqueId":"A","entityTypes":["host"],"name":"A",

"_references":[{"_edgeType":"connectedTo","_direction":"out"},{"_edgeType":"dependsOn","_toUniqueId":"C"}]}

# model B

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B"}

# model C

V:{"_operation":"InsertReplace","uniqueId":"C","entityTypes":["host"],"name":"C"}

# Add connectedTo from B to C

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B","_references":[{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_direction":"out"}]}

Example 4: Have 2 links between A and C and one from A to B

File: edgelifecycle.txt (version 4)

# model A connectedTo B and C

V:{"_operation":"InsertReplace","uniqueId":"A","entityTypes":["host"],"name":"A","_references":[{"_edgeType":"connectedTo","_toUniqueId":"B"},{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_toUniqueId":"C"}]}

# model B

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B"}

# model C

V:{"_operation":"InsertReplace","uniqueId":"C","entityTypes":["host"],"name":"C"}

# Add connectedTo from B to C

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B","_references":[{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_direction":"out"}]}

Example 5: Remove all outgoing edges from B, leaving incoming edges

File: edgelifecycle.txt (version 5)

# model A connectedTo B and C

V:{"_operation":"InsertReplace","uniqueId":"A","entityTypes":["host"],"name":"A","_references":[{"_edgeType":"connectedTo","_toUniqueId":"B"},{"_edgeType":"connectedTo","_toUniqueId":"C"},{"_edgeType":"dependsOn","_toUniqueId":"C"}]}

# model B

V:{"_operation":"InsertReplace","uniqueId":"B","entityTypes":["host"],"name":"B","_references":[{"_edgeType":"connectedTo","_direction":"out"},{"_edgeType":"dependsOn","_direction":"out"}]}

# model C

V:{"_operation":"InsertReplace","uniqueId":"C","entityTypes":["host"],"name":"C"}

Use case: Sometimes connectivity changes and you need to remove the middle hop in the topology. The following example shows how to do it.

Example 6: Define resources and edges in the file observer file

File: connection_update.txt (version 1)

V:{"_operation":"InsertReplace","uniqueId":"X","entityTypes":["host"],"name":"X","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Y"}]}

V:{"_operation":"InsertReplace","uniqueId":"Y","entityTypes":["host"],"name":"Y","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Z"}]}

V:{"_operation":"InsertReplace","uniqueId":"Z","entityTypes":["host"],"name":"Z"}

Example 7: Change edge for resource X to ConnectTo Z directly

File: connection_update.txt (version2)

V:{"_operation":"InsertReplace","uniqueId":"X","entityTypes":["host"],"name":"X","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Z"}]}

V:{"_operation":"InsertReplace","uniqueId":"Y","entityTypes":["host"],"name":"Y","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Z"}]}

V:{"_operation":"InsertReplace","uniqueId":"Z","entityTypes":["host"],"name":"Z"}

Example 8: Remove all out-direction connectedTo edges from resource Y without removing the resource itself.

connection_update.txt (version 3)

V:{"_operation":"InsertReplace","uniqueId":"X","entityTypes":["host"],"name":"X","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Z"}]}

V:{"_operation":"InsertReplace","uniqueId":"Y","entityTypes":["host"],"name":"Y","_references":[{"_edgeType":"connectedTo","_direction":"out"}]}

V:{"_operation":"InsertReplace","uniqueId":"Z","entityTypes":["host"],"name":"Z"}

Example 9: Remove Y resource from topology by removing V: entry for resource. It will delete the Y resource from the topology along with all its edges.

connection_update.txt (version 4)

V:{"_operation":"InsertReplace","uniqueId":"X","entityTypes":["host"],"name":"X","_references":[{"_edgeType":"connectedTo","_toUniqueId":"Z"}]}

V:{"_operation":"InsertReplace","uniqueId":"Z","entityTypes":["host"],"name":"Z"}

Use case: Merge topology information from different sources

In many customer scenarios, topology and resource details information for any resource can originate from different sources. For example, technology-related information for a device can come from one source, its location from another, and connectivity to other domains can come from a third source.

Example 10: Define resources and edges in file observer for wireline source.

wireline.txt

V:{"_operation":"InsertReplace","uniqueId":"WirelineDevice1","entityTypes":["switch"],"name":"WirelineDevice1","Manufacturer": "ALCATEL"}

V:{"_operation":"InsertReplace","uniqueId":"WirelineDevice2","entityTypes":["router"],"name":"WirelineDevice2","Manufacturer": "HUAWEI","_references":[{"_edgeType":"connectedTo","_toUniqueId":"WirelineDevice1"}]}

WirelineDevice1 – Resource Details

WirelineDevice2 - ResourceDetails

{

"Manufacturer": "ALCATEL",

"_createdAt": "2024-04-26T18:25:39.244Z",

"_id": "NChqXdeSRVCh4ZY2TBipMg",

"_modifiedAt": "2024-04-26T18:31:54.332Z",

"_observedAt": "2024-04-26T18:31:54.332Z",

"_startedAt": "2024-04-26T18:31:54.332Z",

"_timeSinceLastUpdate": 119988,

"accessScopeTokens": [

"telco"

],

"beginTime": 1714156314332,

"changeTime": 1714156314332,

"createTime": 1714155939244,

"entityTypes": [

"switch"

],

"lastBroadcastTime": "2024-04-26T18:31:54.332Z",

"name": "WirelineDevice1",

"observedTime": 1714156314332,

"uniqueId": "WirelineDevice1",

"vertexType": "resource"

}

{

"Manufacturer": "HUAWEI",

"_createdAt": "2024-04-26T18:25:39.249Z",

"_id": "n2hUPQ9PRxSyGAHQQ-NY8w",

"_modifiedAt": "2024-04-26T18:31:54.333Z",

"_observedAt": "2024-04-26T18:31:54.333Z",

"_startedAt": "2024-04-26T18:31:54.333Z",

"_timeSinceLastUpdate": 175549,

"accessScopeTokens": [

"telco"

],

"beginTime": 1714156314333,

"changeTime": 1714156314333,

"createTime": 1714155939249,

"entityTypes": [

"router"

],

"lastBroadcastTime": "2024-04-26T18:31:54.333Z",

"name": "WirelineDevice2",

"observedTime": 1714156314333,

"uniqueId": "WirelineDevice2",

"vertexType": "resource"

}

Example 11: The location for these devices is loaded using wireline_locations.

Important: In initial releases, InsertUpdate was provided to merge properties from two different files with a common provider. InsertUpdate is not recommended as it creates problems with topology data, obsolete tokens that will never be removed and accumulate over time.

Instead, use merge rules to merge these files. In Cloud Pak for AIOPS 4.5, you also have a choice to enrich resource properties using file enrichment rules:

https://www.ibm.com/docs/en/cloud-paks/cloud-pak-aiops/4.5.0?topic=rules-defining-file-enrichment

wireline_location.txt

V:{"_operation":"InsertReplace","uniqueId":"WirelineDevice1","entityTypes":["switch"],"Location": "VANCOUVER"}

V:{"_operation":"InsertReplace","uniqueId":"WirelineDevice2","entityTypes":["router"],"name":"WirelineDevice2","Location": "RICHMOND"}

Create Merge rule to create merge token ‘uniqueId’ for these two Observations from provider ‘wireline’ and ‘wireline_location’. This will create a composite resource with all properties from two sources.

Merge Rule: WirelineMergeLocationm

Merged composite resource with Location information

Example 12: Define connectivity for resources ingested from 2 different observers.

Wireless.txt

Merge Rule: WirelessMergeWireline

V:{"_operation":"InsertReplace","uniqueId":"WirelessDevice1","entityTypes":["switch"],"name":"WirelessDevice1","Manufacturer": "CISCO","_references":[{"_edgeType":"connectedTo","_toUniqueId":"WirelineDevice2"}]}

V:{"_operation":"InsertReplace","uniqueId":"WirelessDevice2","entityTypes":["router"],"name":"WirelessDevice2", "Manufacturer": "CISCO","_references":[{"_edgeType":"connectedTo","_toUniqueId":"WirelessDevice1"}]}

# Trivial resource would require a merge rule on the wireline job and this job scoped to uniqueId of router
V:{"uniqueId":"WirelineDevice2","entityTypes":["router"]}

Use case: Deleting resources.

Warning: With the OOTB config, deleting more than 100k resources in 10 days is likely to cause problems due to Cassandra tombstone thresholds.

Option 1: The data admin routine “Delete provider resources” deletes all resources for a specific provider. This is the recommended way to delete resources unless there are multiple file observer jobs using the same provider name. In such a case, it will delete all the resources from multiple observations that share a common provider.

Option 2: Loading an empty file has 2 use cases:

1. Clearing out data from a single job when multiple jobs use the same provider.

2. In a PoC scenario, where you are making frequent changes to the file and re-loading, it can be simpler to use.

#howto #topologymanager #Topology #ASM #observer #upgrade #install #setup #CloudPakforWatsonAIOps #aiops

1 comment

60 views

Permalink

Comments

Cesar Ida

Mon May 20, 2024 09:40 AM

Great content, very useful tips, thanks Gurpreet!

IBM IT Automation Community

Come for answers, stay for best practices. All we're missing is you.

AIOps