Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

Apic analytics uses Opensearch as its database but I'm afraid you can't connect directly to it. It's an internal component and the only access is via our api.

Why do you ask?

------------------------------
Chris Dudley

Original Message:
Sent: Wed January 10, 2024 12:37 PM
From: Tiago Tiago De Almeida Silva
Subject: Event data (logs) saved on the APIC

Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

------------------------------
Tiago Tiago De Almeida Silva
------------------------------

Thanks a lot for the reply Chris! Actually I only needed to know the sort of the database because I thought I could connect it, in some way, directly to the db2.

I have another question. Do you know if all data events are saved integrally and in real time in the Opensearch? I read the documentation and I saw the Analytics tool can only store up to 10,000 event data. According to what I understood I would need to offload the event data to a third-party system in order to get them in this way, right? 

Thanks once more for the help Chris!

------------------------------
Tiago Tiago De Almeida Silva

Original Message:
Sent: Thu January 11, 2024 01:51 AM
From: Chris Dudley
Subject: Event data (logs) saved on the APIC

Apic analytics uses Opensearch as its database but I'm afraid you can't connect directly to it. It's an internal component and the only access is via our api.

Why do you ask?

------------------------------
Chris Dudley

Original Message:
Sent: Wed January 10, 2024 12:37 PM
From: Tiago Tiago De Almeida Silva
Subject: Event data (logs) saved on the APIC

Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

------------------------------
Tiago Tiago De Almeida Silva
------------------------------

All databases inside APIC are internal components and there is no direct connection to any of them I am afraid. But we have a very feature rick API so have a look at what it allows. https://apic-api.apiconnect.ibmcloud.com/v10/10.0.5.LATEST.html#/IBMAPIConnectAnalyticsAPI_200/overview

I think you have misunderstood the documentation. The main API we use to query opensearch can only return 10,000 events, thats simply how it works and is one of the limitations of opensearch/elasticsearch. But you just make the query more specific so it returns less data - e.g. by narrowing down the time frame, or selecting a specific API/product whatever.

It can definitely handle storing billions and billions of api events - thats not a problem. You just need to size the system to accommodate however many transactions per second you are expecting and what data retention you are expecting.

Analytics in 10.0.5 stores each api event record for however long your retention is set for. In 10.0.7+ there is also the addition of long term summary data which allows you to keep a summarised version of those events for months if not years without needing to hugely scale up the size of the resources needed. You can then strike a balance between only keeping full event data for however long you need for problem determination and then use long term summary data for long term data trends and usage.

Should you need to access and return more than 10,000 records at a time then we do have a mechanism for doing that too using the scroll API. You'll find there is a blog post on this site specifically about how to use that.
https://community.ibm.com/community/user/integration/blogs/pablo-lopez-rodriguez/2022/12/05/using-analytics-scroll-operation

Thanks a lot for the reply Chris! Actually I only needed to know the sort of the database because I thought I could connect it, in some way, directly to the db2.

I have another question. Do you know if all data events are saved integrally and in real time in the Opensearch? I read the documentation and I saw the Analytics tool can only store up to 10,000 event data. According to what I understood I would need to offload the event data to a third-party system in order to get them in this way, right? 

Thanks once more for the help Chris!

------------------------------
Tiago Tiago De Almeida Silva

Original Message:
Sent: Thu January 11, 2024 01:51 AM
From: Chris Dudley
Subject: Event data (logs) saved on the APIC

Apic analytics uses Opensearch as its database but I'm afraid you can't connect directly to it. It's an internal component and the only access is via our api.

Why do you ask?

------------------------------
Chris Dudley

Original Message:
Sent: Wed January 10, 2024 12:37 PM
From: Tiago Tiago De Almeida Silva
Subject: Event data (logs) saved on the APIC

Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

------------------------------
Tiago Tiago De Almeida Silva
------------------------------

Thanks for the explanation Chris! I got it now! It was the info that I was looking for! 

Regarding a possible integration with the IBM API Pak for Data, I think the best way of integrating both systems is to offload the event data to a third-party system and import the data to the IBM API Pak for Data, right?

Sorry to bother you with all these questions. I am new with the APIC and I am trying to get an overview of the Analytics tools the platform offers. I really appreciate your help!

All databases inside APIC are internal components and there is no direct connection to any of them I am afraid. But we have a very feature rick API so have a look at what it allows. https://apic-api.apiconnect.ibmcloud.com/v10/10.0.5.LATEST.html#/IBMAPIConnectAnalyticsAPI_200/overview

I think you have misunderstood the documentation. The main API we use to query opensearch can only return 10,000 events, thats simply how it works and is one of the limitations of opensearch/elasticsearch. But you just make the query more specific so it returns less data - e.g. by narrowing down the time frame, or selecting a specific API/product whatever.

It can definitely handle storing billions and billions of api events - thats not a problem. You just need to size the system to accommodate however many transactions per second you are expecting and what data retention you are expecting.

Analytics in 10.0.5 stores each api event record for however long your retention is set for. In 10.0.7+ there is also the addition of long term summary data which allows you to keep a summarised version of those events for months if not years without needing to hugely scale up the size of the resources needed. You can then strike a balance between only keeping full event data for however long you need for problem determination and then use long term summary data for long term data trends and usage.

Should you need to access and return more than 10,000 records at a time then we do have a mechanism for doing that too using the scroll API. You'll find there is a blog post on this site specifically about how to use that.
https://community.ibm.com/community/user/integration/blogs/pablo-lopez-rodriguez/2022/12/05/using-analytics-scroll-operation

Thanks a lot for the reply Chris! Actually I only needed to know the sort of the database because I thought I could connect it, in some way, directly to the db2.

I have another question. Do you know if all data events are saved integrally and in real time in the Opensearch? I read the documentation and I saw the Analytics tool can only store up to 10,000 event data. According to what I understood I would need to offload the event data to a third-party system in order to get them in this way, right? 

Thanks once more for the help Chris!

------------------------------
Tiago Tiago De Almeida Silva

Original Message:
Sent: Thu January 11, 2024 01:51 AM
From: Chris Dudley
Subject: Event data (logs) saved on the APIC

Apic analytics uses Opensearch as its database but I'm afraid you can't connect directly to it. It's an internal component and the only access is via our api.

Why do you ask?

------------------------------
Chris Dudley

Original Message:
Sent: Wed January 10, 2024 12:37 PM
From: Tiago Tiago De Almeida Silva
Subject: Event data (logs) saved on the APIC

Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

------------------------------
Tiago Tiago De Almeida Silva
------------------------------

Hiya,

Good question - I think CP4D has the ability to pull data from a rest API, but i suspect it might not be as optimal as being able to natively query a database, so yes offload to your own elasticsearch or opensearch instance and then import that into CP4D is likely to be a good option.

Can I ask why you want to import the data into CP4D? I'm just curious as I'm looking into the integration possibilities for our analytics data and how we can help make things easier - suggestions welcome! :-)

------------------------------
Chris Dudley

Original Message:
Sent: Thu January 11, 2024 08:57 AM
From: Tiago Silva
Subject: Event data (logs) saved on the APIC

Thanks for the explanation Chris! I got it now! It was the info that I was looking for! 

Regarding a possible integration with the IBM API Pak for Data, I think the best way of integrating both systems is to offload the event data to a third-party system and import the data to the IBM API Pak for Data, right?

Sorry to bother you with all these questions. I am new with the APIC and I am trying to get an overview of the Analytics tools the platform offers. I really appreciate your help!

------------------------------
Tiago Silva

Original Message:
Sent: Thu January 11, 2024 08:15 AM
From: Chris Dudley
Subject: Event data (logs) saved on the APIC

All databases inside APIC are internal components and there is no direct connection to any of them I am afraid. But we have a very feature rick API so have a look at what it allows. https://apic-api.apiconnect.ibmcloud.com/v10/10.0.5.LATEST.html#/IBMAPIConnectAnalyticsAPI_200/overview

I think you have misunderstood the documentation. The main API we use to query opensearch can only return 10,000 events, thats simply how it works and is one of the limitations of opensearch/elasticsearch. But you just make the query more specific so it returns less data - e.g. by narrowing down the time frame, or selecting a specific API/product whatever.

It can definitely handle storing billions and billions of api events - thats not a problem. You just need to size the system to accommodate however many transactions per second you are expecting and what data retention you are expecting.

Analytics in 10.0.5 stores each api event record for however long your retention is set for. In 10.0.7+ there is also the addition of long term summary data which allows you to keep a summarised version of those events for months if not years without needing to hugely scale up the size of the resources needed. You can then strike a balance between only keeping full event data for however long you need for problem determination and then use long term summary data for long term data trends and usage.

Should you need to access and return more than 10,000 records at a time then we do have a mechanism for doing that too using the scroll API. You'll find there is a blog post on this site specifically about how to use that.
https://community.ibm.com/community/user/integration/blogs/pablo-lopez-rodriguez/2022/12/05/using-analytics-scroll-operation

------------------------------
Chris Dudley

Original Message:
Sent: Thu January 11, 2024 07:17 AM
From: Tiago Silva
Subject: Event data (logs) saved on the APIC

Thanks a lot for the reply Chris! Actually I only needed to know the sort of the database because I thought I could connect it, in some way, directly to the db2.

I have another question. Do you know if all data events are saved integrally and in real time in the Opensearch? I read the documentation and I saw the Analytics tool can only store up to 10,000 event data. According to what I understood I would need to offload the event data to a third-party system in order to get them in this way, right? 

Thanks once more for the help Chris!

------------------------------
Tiago Tiago De Almeida Silva

Original Message:
Sent: Thu January 11, 2024 01:51 AM
From: Chris Dudley
Subject: Event data (logs) saved on the APIC

Apic analytics uses Opensearch as its database but I'm afraid you can't connect directly to it. It's an internal component and the only access is via our api.

Why do you ask?

------------------------------
Chris Dudley

Original Message:
Sent: Wed January 10, 2024 12:37 PM
From: Tiago Tiago De Almeida Silva
Subject: Event data (logs) saved on the APIC

Hi guys,

I have a question. Does anyone know what sort of database is used to save the event data (logs) of the APIs? I tried to find it out on the documentation (10.0.5x_lts) but I didn't succeed.

Thanks in advance.

------------------------------
Tiago Tiago De Almeida Silva
------------------------------