API Connect

API Connect

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Analytics Data Export and Import

  • 1.  Analytics Data Export and Import

    Posted Mon April 01, 2024 12:14 AM

    Hi Chris,

    Just would like to check if there is any way to export and import analytics data from Management subsystem or Developer Portal? If it is not doable through Analytics Dashboard in Management or Developer portal, can we do it via API?  

    Currently the analytics data only can retain for maximum three months in APIC 10.0.5. So we are thinking if can export the analytics from APIC. Later if required, we can import it into another low environment for visualization. 



    ------------------------------
    Guo Jun Qiao
    ------------------------------


  • 2.  RE: Analytics Data Export and Import

    Posted Tue April 02, 2024 06:38 AM

    Hiya

    There is no import to analytics.

    You can get the raw api events via the api (see the blog post from Pablo on this site about the scroll api). But there is no way to import them again elsewhere.

    I think you'll need to make do as best you can until 10.0.8 is released in a few months and then make use of the long term summary data to keep event history for longer.

    In the meantime you can offload to a remote server (eg splunk. Elk etc) if you need to.

    Cheers

    Chris



    ------------------------------
    Chris Dudley
    ------------------------------



  • 3.  RE: Analytics Data Export and Import

    Posted Wed April 03, 2024 02:54 PM

    Hi Chris,

    In OVA setup with APIC version 10.0.5.4 currently we store data locally in API Connect Analytics. If we offload to a to a third-party system, is that offloaded data viewable in the API Manager and Developer Portal UIs?



    ------------------------------
    Nirmalan Rajasingam
    ------------------------------



  • 4.  RE: Analytics Data Export and Import

    Posted Wed April 03, 2024 03:02 PM

    No api manager and developer portal uis will only show data from our internal storage.

    APIC can send data (offload) to remote systems but it can't access those systems to query them later - it's fire and forget.
    Our UIs entirely work off internal storage only.

    If you disable internal storage and use offload only you'll find the analytics portion of both uis disappears.



    ------------------------------
    Chris Dudley
    ------------------------------



  • 5.  RE: Analytics Data Export and Import

    Posted Thu April 04, 2024 12:07 AM

    Hi Chris,

    Thank you for your response. Is it possible to offload a to a third-party system without disable internal storage? If so, we can view analytic data in both third-party system and APIC UIs.



    ------------------------------
    Nirmalan Rajasingam
    ------------------------------



  • 6.  RE: Analytics Data Export and Import

    Posted Thu April 04, 2024 01:47 AM

    Yes you can use internal storage and offload, that's pretty common.



    ------------------------------
    Chris Dudley
    ------------------------------



  • 7.  RE: Analytics Data Export and Import

    Posted Wed April 03, 2024 09:11 PM

    Thank you Chris :)



    ------------------------------
    Guo Jun Qiao
    ------------------------------



  • 8.  RE: Analytics Data Export and Import

    Posted Wed April 03, 2024 09:21 PM

    Hi Chris,

    I read below blog post from Pablo. Is it possible for Pablo to share a exported sample data.  Just curious on the format of the data. 

    https://community.ibm.com/community/user/integration/blogs/pablo-lopez-rodriguez/2022/12/05/using-analytics-scroll-operation



    ------------------------------
    Guo Jun Qiao
    ------------------------------



  • 9.  RE: Analytics Data Export and Import

    Posted Thu April 04, 2024 01:44 AM

    No need, the data is the same you see in the UI when you click on an API event.

    Try it yourself :-)



    ------------------------------
    Chris Dudley
    ------------------------------



  • 10.  RE: Analytics Data Export and Import

    Posted Thu April 04, 2024 03:48 AM

    Hi Chris,

    Just a confirm the data looks like below right?

    {
      "@version": "1",
      "resource_id": "::POST:/nemsdsvc",
      "immediate_client_ip": "101.111.153.13",
      "log_policy": "payload",
      "opentracing_info": [],
      "api_id": "ef7d21d8-190d-48f3-a3a8-54b20ef48461",
      "query_string": "",
      "client_id": "",
      "@timestamp": "2024-04-04T07:32:18.655974166Z",
      "gateway_ip": "101.111.153.45",
      "datetime": "2024-04-04T07:32:18.451Z",
      "client_ip": "101.111.153.13",
      "api_version": "1.0.0",
      "developer_org_title": "",
      "developer_org_id": "",
      "org_name": "dev",
      "response_http_headers": [
        {
          "Expires": "-1"
        },
        {
          "Cache-control": "no-store"
        },
        {
          "Set-Cookie": "[\"KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/xxxdev/; HttpOnly\",\"KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/xxxdev/; HttpOnly\"]"
        },
        {
          "X-XSS-Protection": "1; mode=block"
        },
        {
          "Pragma": "no-cache"
        },
        {
          "X-Frame-Options": "ALLOWALL"
        },
        {
          "Referrer-Policy": "no-referrer"
        },
        {
          "Date": "Thu, 04 Apr 2024 07:32:18 GMT"
        },
        {
          "Strict-Transport-Security": "max-age=31536000;"
        },
        {
          "X-Content-Type-Options": "nosniff"
        },
        {
          "Content-Type": "application/xml"
        },
        {
          "X-Global-Transaction-ID": "3da159a0660e57820000fcc0"
        }
      ],
      "request_body": "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">\r\n   <soapenv:Header/>\r\n   <soapenv:Body/>\r\n</soapenv:Envelope>",
      "domain_name": "dev231-gateway",
      "latency_info": [
        {
          "started": 0,
          "task": "Start"
        },
       



    ------------------------------
    Guo Jun Qiao
    ------------------------------



  • 11.  RE: Analytics Data Export and Import

    Posted Thu April 04, 2024 03:53 AM

    Yes, exact fields will depend on what was sent by the gateway, what log level was used etc…



    ------------------------------
    Chris Dudley
    ------------------------------