New in API Connect v10.0.6 is the ability for APIC Analytics to efficiently retain your data over much longer periods.
As you may know, the underlying OpenSearch datastore is really a search engine rather than a traditional database and so, search performance is prioritised over storage efficiency. This means that relatively large amounts of RAM are required for each additional index and shard, mounting up as the cluster soaks up more and more data.
Traditionally, such systems either offloaded older data externally for long term storage, or backed up snapshots to cold storage. Neither approach keeps your data online for use in APIC Analytics. Our new solution provides best of both worlds - fast searching for recent data and fully online long term summary data.
Call volume trends report
APIC 10.0.6 includes the first of what will be multiple reports in the UI allowing the API Provider to track the usage of their APIs over longer periods of time. It shows the usage for individual APIs, Products and Plans over years, it can also show which ones are responsible for the most call volume, as well as the least with Leaderboards of the top and bottom 10 of each. There is the ability to see the breakdown of usage by provider org, catalog and space as well as metrics on call volume and data received and sent. The metrics also show what the long term trend is over the specified time frame. These reports could be exported quarterly in order to fulfil corporate regulatory or compliance requirements.
Summary data index
Analytics is traditionally used to store data for two purposes, the first is for problem determination, in order to do that it stores the full data set available with as much information as possible in order to allow users to dig into why something happened to a specific API event. This tends to be relatively recent - e.g. looking into a 500 error that happened last week, it doesn’t tend to require investigation of a specific transaction months ago. The second main use case is for data reporting, especially over time. This use case doesn’t require as much detail, it tends to be more about aggregations of data - e.g. per API or per consumer organization over a longer timeframe. Up to now this has meant a compromise between wanting to keep data longterm in order to be able to do long term accounting on usage and the resource requirements of storing large quantities of hugely detailed information for longer than is necessary for problem determination.
Now APIC includes the ability to transform the stored data for long term storage, this allows the retention of trend and summary information over considerable periods of time - multiple quarters, even years - and this retention is separate from the retention of the full detailed API events themselves. For example, this could allow customers to reduce their full detailed API event retention down to 2 weeks for short term problem determination and yet still be able to do API trend reporting over the last 2 years in the same resource footprint as being used today. There are separate indicies and they have separate retention policies, allowing full control of how long data is kept for each of the primary Analytics use cases.
The above example is retaining full API event data for 30 days but keeping summary trend data for a year and a day.
Reporting API consumption figures for regulatory or governance reasons does not normally require the storage of detailed HTTP headers, payloads, etc. This allows the long term summary data to focus on the more important “who called what, when”. It still allows the tracking of individual API, Product and Plan usage as well as per application and consumer organization.
API Connect 10.0.6 is a CD (Continuous Delivery) release and so will only receive short term support with no fixpacks/fixes. It is not LTS (Long Term Support). It is not recommended to use CD releases in production. This capability will be available to try out in APIConnect on AWS SaaS very shortly. CD releases allow customers to try out new features and provide feedback to the development team to allow the shaping of new features before they are delivered in the next LTS release. As such please let us know any feedback and feature requests.
#analytics #APIConnect