There are large challenges resulting from limited visibility on the nature of the traffic running in and on enterprise networks. Often the applications behind this traffic have un-managed APIs that may be shadow APIs, and have the potential to be non-compliant to governance standards. Also as time passes APIs, while still active, can become unmaintained and forgotten leading to security gaps and risks to the enterprise (so called zombie APIs).
IBM® API Connect Enterprise as a Service has added API discovery capability that allows you to quickly discover and view APIs running in the enterprise. IBM® API Connect API discovery uses data source collectors placed onto gateways and ingresses to retrieve traffic logs and traces. These are forwarded to the discovery service, analyzed, and OpenAPI documents are generated and presented for review.
Collector Types
There are three collector types:
The ”DataPower API Gateway proxy” data source collector allows the collection of api traffic data passing through the Datapower API Gateway acting as a proxy for backend applications. To enable the collection you can either create a new proxy API definition or update an existing proxy API which you have in your provider organization in IBM® API Connect Enterprise as a Service. The proxy API can contain a target endpoint with any number of accessible applications behind it and once traffic to those goes through the proxy API the discovery service will collect and then be able to reconstruct OpenAPI documents for these applications.
Full details are described at https://www.ibm.com/docs/en/api-connect/saas?topic=ad-configuring-datapower-api-gateway-proxy-collector-api-discovery.
The “OpenTelemetry on Istio” data source collector can be placed on any Istio instrumented cluster. The collector is configured and deployed by using a Helm chart, where it is registered with the Istio MeshConfig by updating the extensionProviders configuration. After the collector is deployed, any Istio pods that have API traffic running through Envoy then send this data to IBM® API Connect discovery service.
Full details are described at https://github.com/ibm-apiconnect/api-discovery-otel-collector.
The “GitHub” collector uses GitHub actions to send API documents from GitHub to IBM® API Connect. An action is configured on a GitHub repository and can be targeted at a single OpenAPI file, multiple OpenAPI files, a folder, or folders where a change to the target scope will result in that OpenAPI document being sent to the discovery service.
Full details are described at https://github.com/ibm-apiconnect/apic-discovery-action
OpenAPI Generation
Collector traffic data contains information such as target URL path, operation (HTTP method), response code, response body, and request body. A URL path may also contain path and query parameters. Discovery service OpenAPI generation extracts this information to group operations, determines path parameters, request and response details to form the OpenAPI document. As more data is received, the generation process re-evaluates the OpenAPI and updates the document based on new detail presented in the data.
How to use
The enterprise owners of a cluster want to determine what applications are hosted on an Istio instrumented cluster. They deploy the OpenTelemetry on Istio collector to the cluster and traffic data is forwarded to IBM® API Connect discovery service.
For the purpose of this exercise, a sample store front application has been deployed on the cluster and as users interact with the store front, traffic data is being generated.
Collector View
When the collector sends the first traffic data to your provider organization, the collector gets automatically added to the Sources tab in the discovery part of the API Manager UI. Navigate to the API discovery service using the Discovery icon (binoculars).
All collectors are listed under “Sources”. Note that the OpenTelemetry on Istio collector is present for the cluster being examined due to the traffic passed by the collector.
The collector detail shows its status and the time of last receipt of traffic data from that source.
A collector can be paused or deleted.
-
Pausing a collector will stop the discovery service from adding or updating discovered APIs documents from that source.
-
Deleting a collector will remove the collector instance from API Connect and all associated discovered APIs will be removed also.
Discovered APIs View
Discovered APIs are those shown under “APIs”. From here an enterprise owner can view what applications are being hosted on the cluster and can make a determination on what action to take on the applications those APIs are exposing.
In this example, four OpenAPI documents have been generated based on the traffic data received by the collector. Each OpenAPI shows the collector source and the time of last received data. OpenAPI definitions are continuously re-evaluated as new data is received from a collector.
To view the API document, select the API title and the OpenAPI document will be presented. The OpenAPI document can be downloaded to a local filesystem and can be copied to the APIs tab in the Develop section of the API Manager UI (Copy to draft).
This example has a single operation (login) and request and response information is present. As there is only a single response code in the definition, we know that there have only been successful logins in the time the collector has been sending data. Let’s simulate a failed login attempt.
The collector sends these traffic logs and the user interface shows that data has been received as the time is updated.
Viewing the OpenAPI document, we can see there has been an update to the document. A response code of “401 Unauthorized” has been added into the responses definition of the document.
After further user interaction with the store front, we can see more OpenAPI documents are generated. Lets look at the “cart” OpenAPI document.
Three operations are present in the document and a path parameter has been identified as part of one of the operations.
The example here has an obvious identifier at the end of the path - which is a user identifier. In this example, we are using a single user. If a second user was interacting with the shopping cart, the collector would forward those logs to the discovery service and the OpenAPI generation process would have identified the “identifier” as another path parameter and the OpenAPI document would be updated appropriately.
From the OpenAPI list, it is possible to:
-
Download the OpenAPI document to local filesystem
-
View summary data of the document
-
Copy the document to the APIs tab in the Develop section of the API Manager UI
-
Hide the OpenAPI document from the user and not shown regardless of collector updates (can be un-hidden if needed)
-
Removed from the API list (however, it can be discovered again).
Enable API Discovery
Contact your IBM representative to enable API Discovery for your organization.
For more detail, use the IBM® API Connect API discovery capability documentation.