Maximo

 View Only
  • 1.  Gracefully handle MAS Manage downtimes

    Posted 5 days ago

    Hi,

    we're looking for a way to gracefully handle MAS Manage downtimes.
    What I mean in its essence is the ability to redirect user requests, originally pointing to MAS Manage, to some static web page when Manage is down (e.g. during updatedb phase of the build process and before server bundle POD becomes ready).
    For simplicity let's assume that the control over Manage downtime detection is fully manual so that it's Manage deployer's responsibility to activate/deactivate the redirect.

    With classic Maximo we were doing this IHS config-based conditional rewrite rule which was intercepting user's requests using "file exists" test. So whenever we were activating "maintenance page" redirect then we were creating a file to trigger the rewrite and once done we simply were deleting the file, restoring normal operations.
    It was just one of the ways to achieve what we intended, sufficient for our needs. Of course there are other ways to do it by e.g. updating DNS records, throwing in reverse-proxy in between end users and Maximo, etc. We do consider these options either by unnecessarily increasing application maintenance complexity (reverse proxy - yet another component to install and maintain) or resources dependency (IT personel available privileged to update DNS records).

    With MAS Manage we noticed that Routes in mas-<inst>-manage namespace, handling requests to server bundles are managed by operator and changes to Route's essential settings (e.g. spec.to.name, spec.port.targetPort) are being overwritten with every MAS Manage operator reconciliation cycle.
    Therefore we cannot mimic the approach we used in classic deployment by simply updating MAS Manage route(s) to point to some custom service, e.g. running nginx, which serves static content.

    It could be that we're missing something or doing something wrong. 
    Do you have any suggestions how to achieve what we're aiming for?
    Alternatively I would love to hear how else you're dealing with MAS Manage downtimes so that users see something more valuable than raw "HTTP 500 Service unavailable"? Any tips will be highly appreciated!

    Thank you!



    ------------------------------
    Andrzej Więcław
    Maximo Technical Consultant
    AFRY
    Wrocław, Poland
    ------------------------------


  • 2.  RE: Gracefully handle MAS Manage downtimes

    IBM Champion
    Posted 23 hours ago

    Hi Andrzej:

    Normally you should not get "Error 500" responses when all Pods are down... As Manage ingress is "managed" by a standard OpenShift Route, when all Pods are down, you should get "Error 503 - Application is not available" with a somewhat descriptive message from OpenShift stating that:

    Application is not available

    The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

    Possible reasons you are seeing this page:

    • The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists.
    • The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path.
    • Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.

    The above page is generic/standard but can be customized as per section "Customizing HAProxy error code response pages" on the URL: https://docs.openshift.com/container-platform/4.12/networking/ingress-operator.html 

    Unfortunately, it seems to be an "all or nothing" situation, the customized page will be shown for all possible Routes (including non-existing ones) that have no available serving Pods and will not be specific to Manage.

    Hope the above helps somewhat.

    Regards,

    Julio Perera

    Maximo Technical Consultant

    Interloc Solutions, US.



    ------------------------------
    Julio Perera
    ------------------------------



  • 3.  RE: Gracefully handle MAS Manage downtimes

    Posted 5 hours ago

    Hi Julio,

    you're right, when PODs are down then we indeed get HTTP 503, rather than mentioned HTTP.
    I haven't explored yet possibilities around HAProxy customization but maybe this is a way...

    Thank you!



    ------------------------------
    Andrzej Więcław
    Maximo Technical Consultant
    AFRY
    Wrocław, Poland
    ------------------------------