DataPower

DataPower

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
  • 1.  API Connect Gateway Service "down-Pending" status anomaly

    Posted Tue August 01, 2023 03:46 AM
    Edited by dalsu Byun Tue August 01, 2023 07:55 PM

    API Connect Gateway Service [down - Pending] 

    A cluster of 3 DataPower Gateways is configured and in use.
    If all 3 gateways are restarted, 1 API Connect Gateway Service out of 3 is in a down - pending state and the service is not automatically available.
    Gateway Peerings are all OK.

    User-defined policies are in use, and it remains in the same state even after restarting the API Connect Gateway Service.
    Error message: "apic-gw-service (default): The effective gateway peering object is down"

    IBM DataPower Gateway Configuration -------------------

    Hypervisor: VMware Virtual Platform

    Version: IDG.10.5.0.2 build 345566 on Sep 22, 2022 10:28:24 AM

    ------------------------------
    dalsu Byun
    ------------------------------



  • 2.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Tue August 01, 2023 10:30 AM

    You're showing us why the gateway service is down, but why is the peering down?

    I've seen this quite a bit after restarts, and we've found the peering alternately down because it is disabled, or down for other reasons where disabling and enabling resolves it.



    ------------------------------
    Joseph Morgan
    ------------------------------



  • 3.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Tue August 01, 2023 07:55 PM

    This is Start-up log of error server.

    Looking at the log, it seems that apic-gw-service is executed first before gateway peering starts.
    This seems to be a bug.

    -------------------------------------------------------

    20230801T233003.484Z [0x04f30005][mgmt][error] apic-gw-service(default): tid(111): The effective gateway peering object is down
    20230801T233003.496Z [0x00350015][mgmt][notice] smtp-server-connection(default): tid(5708): Operational state down
    20230801T233003.497Z [0x00350014][mgmt][notice] smtp-server-connection(default): tid(5708): Operational state up
    20230801T233009.354Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_obj): tid(9084): Operation state transition to up pending
    20230801T233009.354Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_obj): tid(9084): Operation state transition to up pending
    20230801T233009.391Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_APIprobe): tid(9020): Operation state transition to up pending
    20230801T233009.391Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_APIprobe): tid(9020): Operation state transition to up pending
    20230801T233009.829Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_scripratelim): tid(9148): Operation state transition to up pending
    20230801T233009.829Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_scripratelim): tid(9148): Operation state transition to up pending
    20230801T233009.937Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_subscription): tid(9180): Operation state transition to up pending
    20230801T233009.937Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_subscription): tid(9180): Operation state transition to up pending
    20230801T233010.778Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_ratelimit): tid(9116): Operation state transition to up pending
    20230801T233010.778Z [0x003500bb][mgmt][notice] gateway-peering(gateway_peering_ratelimit): tid(9116): Operation state transition to up pending
    20230801T233010.817Z [0x003500bb][mgmt][notice] gateway-peering(gateway_Peering_oAuth): tid(9052): Operation state transition to up pending
    20230801T233010.817Z [0x003500bb][mgmt][notice] gateway-peering(gateway_Peering_oAuth): tid(9052): Operation state transition to up pending
    20230801T233051.227Z [0x00350014][mgmt][notice] gateway-peering(gateway_peering_obj): tid(9084): Operational state up
    20230801T233058.324Z [0x00350014][mgmt][notice] gateway-peering(gateway_peering_APIprobe): tid(9020): Operational state up
    20230801T233058.325Z [0x00350014][mgmt][notice] gateway-peering(gateway_peering_scripratelim): tid(9148): Operational state up
    20230801T233100.769Z [0x00350014][mgmt][notice] gateway-peering(gateway_peering_subscription): tid(9180): Operational state up
    20230801T233101.049Z [0x00350014][mgmt][notice] gateway-peering(gateway_peering_ratelimit): tid(9116): Operational state up
    20230801T233103.577Z [0x00350014][mgmt][notice] gateway-peering(gateway_Peering_oAuth): tid(9052): Operational state up
    20230801T233103.612Z [0x00350014][mgmt][notice] api-security-token-manager(default): tid(111): Operational state up

    -------------------------------------------------------

    idg[domain_test1]# show gateway-peering-status

     Address      Configuration name           Pending updates Replication offset Link status Primary
     ------------ ---------------------------- --------------- ------------------ ----------- -------
     192.168.35.3 gateway_Peering_oAuth        0               645114             ok          yes
     192.168.35.3 gateway_peering_APIprobe     0               755031             ok          yes
     192.168.35.3 gateway_peering_obj          0               1594486            ok          yes
     192.168.35.3 gateway_peering_ratelimit    0               818332             ok          yes
     192.168.35.3 gateway_peering_scripratelim 0               791228             ok          yes
     192.168.35.3 gateway_peering_subscription 0               3332927            ok          yes
     192.168.35.4 gateway_Peering_oAuth        0               644071             ok          no
     192.168.35.4 gateway_peering_APIprobe     0               751916             ok          no
     192.168.35.4 gateway_peering_obj          0               1591519            ok          no
     192.168.35.4 gateway_peering_ratelimit    0               814615             ok          no
     192.168.35.4 gateway_peering_scripratelim 0               786335             ok          no
     192.168.35.4 gateway_peering_subscription 0               3309237            ok          no
     192.168.35.5 gateway_Peering_oAuth        0               639766             ok          no
     192.168.35.5 gateway_peering_APIprobe     0               751916             ok          no
     192.168.35.5 gateway_peering_obj          0               1594192            ok          no
     192.168.35.5 gateway_peering_ratelimit    0               816393             ok          no
     192.168.35.5 gateway_peering_scripratelim 0               791228             ok          no
     192.168.35.5 gateway_peering_subscription 0               3332913            ok          no

    idg[domain_test1]# show apic-gw-service

    apic-gw-service: default [down]
    ------------------------
     admin-state enabled
     local-address dpg3.apic.sample.com
     local-port 3000
     ssl-client TLS_clientprofile1  [up]
     ssl-server TLS_Serverprofile  [up]
     api-gw-address dpg3.apic.sample.com
     api-gw-port 9443
     gateway-peering default-gateway-peering  [down]
     gateway-peering-manager default  [up]
     v5-compatibility-mode off
     user-defined-policies validate-usernametoken_1.1.0  [up]
     slm-mode autounicast
     ip-unicast



    ------------------------------
    dalsu Byun
    ------------------------------



  • 4.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Tue August 01, 2023 08:42 PM

    Have you tried simply disabling and then re-enabling the default-gateway-peering object?



    ------------------------------
    Joseph Morgan
    ------------------------------



  • 5.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Tue August 01, 2023 08:55 PM
    Edited by dalsu Byun Tue August 01, 2023 08:58 PM

    Yes, but it's in the same condition.
    So, I rebooted the problematic API Gateway, and the API Connect Gateway service started up.
    It seems like a problem to have to do this every time I restart.

    idg[domain_test1]# show apic-gw-service

    apic-gw-service: default [up] (modified)
    ------------------------
     admin-state enabled
     local-address dpg1.apic.sample.com
     local-port 3000
     ssl-client TLS_Client_Profile  [up]
     ssl-server TLS_Server_Profile  [up]
     api-gw-address dpg1.apic.sample.com
     api-gw-port 9443
     gateway-peering default-gateway-peering  [down]
     gateway-peering-manager default  [up]
     v5-compatibility-mode off
     user-defined-policies validate-usernametoken_1.1.0  [up]
     user-defined-policies udp-allowedip_1.0.3  [up]
     slm-mode autounicast
     ip-unicast


    ------------------------------
    dalsu Byun
    ------------------------------



  • 6.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Wed August 02, 2023 07:48 AM
    Edited by Roshan Rama Wed August 02, 2023 07:54 AM

    This behaviour is caused by bug and is listed under the known limitations list. 

    https://www.ibm.com/docs/en/api-connect/10.0.5.x_lts?topic=overview-known-limitations

    There is an interim fix for this which can be applied to v10.0.5.2-ifix1 or you could upgrade to v10.0.5.3 or v10.0.5.4 which has the permanent fix.

    The interim fix can be found on Fix Central: https://www.ibm.com/support/fixcentral/swg/selectFixes?parent=ibm%7EWebSphere&product=ibm/WebSphere/IBM+API+Connect&release=10.0.5.2&platform=All&function=fixId&fixids=ibm-apiconnect-v10.0.5.2-ifix.LI82903%3A67094276418854&includeSupersedes=0&source=fc

    This fix does have a README with instruction on how to apply it.

    This bug was closed under APAR LI82903 which was released in v10.0.5.3



    ------------------------------
    Rosh
    ------------------------------



  • 7.  RE: API Connect Gateway Service "down-Pending" status anomaly

    Posted Wed August 02, 2023 08:52 PM
    Edited by dalsu Byun Fri August 04, 2023 01:07 AM

    It seems to be a Configuration sequence, TimeZone problem.
    Currently it works fine.

    --------------------------------------------------------------

    IBM DataPower API Gateway : v10.5.0.2
    API Connect : 10.0.5.3

    ------------------------------
    dalsu Byun
    ------------------------------