What and why?
Isn't 'the cloud', virtualisation and Internet a wonderful thing? As consumers, we use services and applications without caring about where their supporting devices are physically located. That is, until there's a problem which may be due to where they're located and then we do care. As service providers, you'll care about physical locations because knowing where your devices are is useful for planning, maintenance, incident management and impact assessment. It's increasingly important because of external risks such a wildfires and floods but more on that later...
In this AIOps Bitesize, we'll show how to use IBM Cloud Pak for AIOps Topology Manager's geospatial capabilities to build a simple model of buildings, sites and cables and how to view them on map. Topology Manager supports GeoJSON Point, LineString and Polygon geometries to define where your resources are located. It can render those on a map and has automated and adaptive location clustering in map views to help handle large numbers of locations. Last but not least, you can define geospatial filters and save them to help manage your infrastructure.
In a future AIOps Demystified article, we'll look in depth at management practices and programmatic techniques using geospatial data.
Let's Go
We'll use the File Observer to provide the device data for this article by loading a recording from the Network Manager Observer. The File Observer is a popular integration that loads files built by users to describe a topology and is well suited to custom integrations. The Network Manager Observer retrieves network topology data from Network Manager that includes devices and their cards and ports, and how the devices are connected to each other. Each Observer other than the File Observer can record what they see to a file that can be loaded using the File Observer, and that's what we'll do here by using a recording from the Network Manager Observer.
We'll also create some geospatial data using https://geojson.io and use a file enrichment rule to attach it to the data from Network Manager. Check out the product documentation at these links for this AIOps Bitesize:-
TOP TIP: File enrichment rules are more efficient at fulfilling basic data enrichment use-cases than using multiple Observer jobs and merging their data. This is because the enrichment is performed by the Observer job on data in motion before the data is sent to the Topology Service. This in in contrast to merging data from multiple Observer jobs that need meta data management and merge processing after the data has been consumed by the Topology Service. If your use-case doesn't need to use relationships from multiple sources and merging, consider using file enrichment rules.
We'll do a much more in depth article about rules but for now the data flow is shown by the following diagram. Rules are assigned to the Observer types, providers, resource types and instances they're scoped to. When the Observer job runs, it evaluates the rule against each resource processed and if there's a condition match, the resource is updated.
1. Go to https://geojson.io and define a GeoJSON Point, LineString and Polygon in locations of your choice using the shape tools. I've created locations near the IBM London office including a Point in Jubilee Gardens, a Polygon representing the office building and a LineString along a path representing a cable.
TOP TIP: GeoJSON only supports the WGS84 coordinate reference system.
TOP TIP: GeoJSON coordinates doesn't use latitude, longitude coordinate ordering! It uses longitude, latitude instead - check out https://macwright.com/lonlat for some background.
TOP TIP: Topology Manager currently supports Point, LineString and Polygon geometries. It does not support the Multi-versions of those geometry types.
TOP TIP: Note that I've added a 'device' property to each location for this demo. In reality, you could have many devices at a given location in which case this would need to be an array/list property.
2. Once you've defined your locations, save the GeoJSON to a file by choosing Save -> GeoJSON and then https://geojson.io will save it as a compact and minified JSON file called map.geojson. The contents of that file will look something like the following and we're interested in feature records highlighted.
TOP TIP: Although GeoJSON supports properties in features, Topology Manager ignores them - specify properties on the resource instead.
{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"device":"ITNMDOMAIN:pueblo-abr-cr36.na.test.lab"},"geometry":{"coordinates":[[[-0.11584397387403556,51.503263227530425],[-0.1157000899228251,51.503421998828856],[-0.11651434228508606,51.503599089239486],[-0.1166059047991439,51.503444389608376],[-0.11584397387403556,51.503263227530425]]],"type":"Polygon"},"id":0},{"type":"Feature","properties":{"device":"ITNMDOMAIN:aspen-asbr-cr36.na.test.lab"},"geometry":{"coordinates":[-0.11828346087096975,51.503973623013366],"type":"Point"},"id":1},{"type":"Feature","properties":{"device":"ITNMDOMAIN:colorado1-cs29.na.test.lab"},"geometry":{"coordinates":[[-0.11555620597036409,51.50341589225047],[-0.11656012354274026,51.50364590612941],[-0.11637045833259663,51.50405707848955],[-0.11694599413988271,51.5041792081787]],"type":"LineString"},"id":2}]}
3. We now need to process our data a little into something closer to the file format Topology Manager uses for enrichment rules. This must be a text file where each line is a JSON record that contains the data to enrich matching resources with and a lookup property called the 'Enrichment file key'. The file and enrichment process is demonstrated by the following diagram.
To add our location data, we need to add a geolocation property that contains the GeoJSON feature of interest to the relevant records. We'll use the amazing jq command to process the map.geojson file to the point where all we need to do is upload the file to the rule.
TOP TIP: Use the jq -M option to produce monochrome output because colour output can cause problems due to control characters.
TOP TIP: If your geolocation data has an array/list of devices at each location, you need to have an enrichment line for each array element (device).
TOP TIP: How many decimal places should my latitude and longitude coordinates be? I'd suggest 5 is a useful sweet spot as it's accurate to within 1.11m, 6 is accurate to within 11.1cm.
mattduggan@thundercat Geo % jq -cM '.features |= map(.geolocation = . | .uniqueId = .properties.device | del(.geometry, .type)) | del(.. | .id?, .properties?) | .features . []' < map.geojson > mygeospatial.txt; cat mygeospatial.txt
{"geolocation":{"type":"Feature","geometry":{"coordinates":[[[-0.11584397387403556,51.503263227530425],[-0.1157000899228251,51.503421998828856],[-0.11651434228508606,51.503599089239486],[-0.1166059047991439,51.503444389608376],[-0.11584397387403556,51.503263227530425]]],"type":"Polygon"}},"uniqueId":"ITNMDOMAIN:pueblo-abr-cr36.na.test.lab"}
{"geolocation":{"type":"Feature","geometry":{"coordinates":[-0.11828346087096975,51.503973623013366],"type":"Point"}},"uniqueId":"ITNMDOMAIN:aspen-asbr-cr36.na.test.lab"}
{"geolocation":{"type":"Feature","geometry":{"coordinates":[[-0.11555620597036409,51.50341589225047],[-0.11656012354274026,51.50364590612941],[-0.11637045833259663,51.50405707848955],[-0.11694599413988271,51.5041792081787]],"type":"LineString"}},"uniqueId":"ITNMDOMAIN:colorado1-cs29.na.test.lab"}
What have we done here? The jq command did the following:-
- moved the device property value to a new uniqueId property.
- produced compact monochrome output.
- 'unwrapped' the array of features so we instead have a JSON record per GeoJSON feature, per line.
- deleted the parent type, properties, id and old geometry properties. Topology Manager will ignore what it doesn't need but it's good to be concise with our data.
- wrote a new mygeospatial.txt file and printed it out.
4. Now we've built our enrichment file, we need to create a file enrichment rule. Our business goal is to enrich any device with location data if the enrichment file has a line for the device's uniqueId property value. Navigate to Resource management -> Topology configuration -> Rules and create a New file enrichment rule as follows and save it. You now need to re-run your Observer job for the rule to take effect because they act on data-in-motion at integration time, not post collection.
5. Now your Observer job has run, a good way of checking if the rule worked correctly is to either search for the resources of interest or create a filter that finds resources with geolocation data as shown here. We can see the map icon and if we look at more details, we can see the geolocation data included.
6. Lets look at them on a map!!! Hold on... the map icon was greyed out - why is that? We haven't configured a tile server yet and so navigate to Resource management -> Topology configuration -> Advanced topology settings, setup the Resource map and save the changes. Here I've used OpenStreetMap as the tile server - check out https://wiki.openstreetmap.org/wiki/Raster_tile_providers for a list of raster tile servers or you can setup your own which should be considered best-practice.
7. Now the tile server is setup, there's a number of ways you can look at the location data - from the now highlighted map icon in resource list which can show the whole map or the location of a resource from its Resource details Location tab. Try zooming out to see the adaptive clustering in action.
TOP TIP: As a bonus tip, you can define geospatial filters for specific regions using the filter dialogues Geospatial filter condition. When defining an area, you can see the WKT representation of the shape drawn in the highlighted area. You can also paste WKT or GeoJSON into that text field to define the shape and that's useful if you have existing data such as state boundaries to use. Try saving your filter so you can easily recall it later.
TOP TIP: Your filters can combine conditions so you can easily define a saved filter such as show me all Cisco devices near the IBM office that have major problems.
Final Thoughts
In this AIOps Bitesize, we used Topology Manager's geospatial capabilities to attach site, building and cable models to resources using a file enrichment rule. You saw how we use GeoJSON Point, LineString and Polygon geometry models as a standardised way of modelling geospatial elements and how online tools like https://geojson.io can help author, validate and test your data. You also saw how we can visualise and filter location data.
What would your scenario be?