IBM Security Verify

 View Only
Expand all | Collapse all

ISAM9 with Open Shift

  • 1.  ISAM9 with Open Shift

    Posted Thu March 19, 2020 03:03 AM
    Hi ,
    I am doing ISAM9 configuration on open shift V3, can you help me resolve below three queries/issue 

    1. Whatever configuration we are doing using LMI its getting deleted whenever we are having open shift container restart, is it expected behavior ?

    2. Is there any other way to do the configuration like instance creation, junction creation using YAML file instead of LMI configuration.

    3. we are facing access of LMI as well but that also interim, do you know what could be the issue.

    ------------------------------
    Mayur Wattamwar
    ------------------------------


  • 2.  RE: ISAM9 with Open Shift

    Posted Thu March 19, 2020 03:34 AM
    HI

    Have you done publish snapshot after doing the change . it looks you are loosing the snapshot after container restart that's why you are losing configuration information.

    You can use Rest API in combination with an automation
    https://github.com/IBM-Security/isam-ansible-roles


    ------------------------------
    Tushar
    Tushar
    ------------------------------



  • 3.  RE: ISAM9 with Open Shift

    Posted Thu March 19, 2020 04:20 AM
    Hello Mayur,

    1. I wonder if you have defined a persistent volume for /var/shared on the config container.  If you don't have this then you will lose any files written to disk when the pod is recreated.  That sounds like what is happening.

    2. It is not possible to configure ISAM via YAML files - the configuration of the ISAM appliance remains the same as always (manual configuration via LMI or automated configuration via REST).

    3. How are you accessing the LMI?  If you are using a route, I would expect this to be stable.  I have seen some issues when using port-forward but have not been able to work out how to improve in this case.

    Have you seen my ISAM on OpenShift resources?  I have a cookbook (https://ibm.biz/isamopenshiftcookbook) and a set of OpenShift templates (within https://ibm.biz/isamdocker) that you might find useful.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 4.  RE: ISAM9 with Open Shift

    Posted Wed March 25, 2020 08:59 AM
    Jon, Thank you for your reply,
    we have mounted a persistence and assigned it to /var/shared on the config container, however we are still experiencing the same issue where configurations (license, database and ldap) are getting lost whenever a container get restarted. I have also noticed that the snapshots are not getting taking frequently, we only had one snapshot taken back on Mar 19 and nothing else ever since. I have also took a manual snapshot on Mar 24th and tried to apply it today on Mar 25th but I have received the following error:
    "The system encountered an error while it was attempting to apply the changes, and has reverted all values to the previous state. Contact IBM customer Support"


    container configuration:

    spec:
    progressDeadlineSeconds: 600
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: isamconfig
    strategy:
    rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1
    type: RollingUpdate
    template:
    metadata:
    creationTimestamp: null
    labels:
    app: isamconfig
    spec:
    containers:
    - env:
    - name: SERVICE
    value: config
    - name: CONTAINER_TIMEZONE
    value: Europe/London
    - name: ADMIN_PWD
    valueFrom:
    secretKeyRef:
    key: adminpw
    name: samadmin
    image: 'registry/ibmcorp/isam:9.0.7.0'
    imagePullPolicy: IfNotPresent
    livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 500
    periodSeconds: 20
    successThreshold: 1
    tcpSocket:
    port: 9443
    timeoutSeconds: 1
    name: isamconfig
    ports:
    - containerPort: 9443
    protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/shared
    name: isamconfig
    - mountPath: /var/application.logs
    name: samconfig
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
    runAsNonRoot: true
    runAsUser: 6000
    serviceAccount: isam
    serviceAccountName: isam
    terminationGracePeriodSeconds: 30
    volumes:
    - name: isamconfig
    persistentVolumeClaim:
    claimName: isamconfig
    - name: samconfig
    persistentVolumeClaim:
    claimName: samconfig
    - emptyDir: {}
    name: isamconfig-logs

    ------------------------------
    Mayur Wattamwar
    ------------------------------



  • 5.  RE: ISAM9 with Open Shift

    Posted Wed March 25, 2020 09:29 AM
    Edited by Jon Harry Wed March 25, 2020 09:34 AM
    Mayur, Ghiffary,

    Over the weekend, an issue was identified in SAM 9.0.7.0 and SAM 9.0.7.1 which prevents snapshots from being applied.
    You should update your image to either 9.0.7.0 IF2 or 9.0.7.1 IF4.  This should solve the issue of snapshots not being applied.

    Also note that ISAM images are no longer on Docker Store.  They are now on Docker Hub.  Image name you need is probably:  ibmcom/isam:9.0.7.0_IF2

    If you still have an issue after this, please let us know.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 6.  RE: ISAM9 with Open Shift

    Posted Thu April 02, 2020 06:42 AM
    Thank you Jon for this, we will update version. as of now we are doing manual snapshot to prevent our configuration. currently we are good so far but observe instance those are created using LMI are not able to see in running state but still able to connect from policy server and able create junction for this instance, is it expected behavior in ISAM9. 

    sh-4.2$ pdweb status
    webseald-default-isam9 yes no


    Also,in Junction back end we have added one server which is running but not able to see the status running , I am not able to do test connectivity due some restriction of command in our network within container.

    Server 1:
    ID: f55bd8d2-742f-11ea-9767-0a5816f90d3c
    Server State: unknown
    Operational State: Online




    ------------------------------
    Mayur Wattamwar
    ------------------------------



  • 7.  RE: ISAM9 with Open Shift

    Posted Thu April 02, 2020 07:11 AM
    Hello Mayur,

    When running ISAM in containers, the Policy Server in the configuration container isn't talking to the WebSEALs instances in the Reverse Proxy containers.  It is talking to special local WebSEAL processes running locally on the configuration container just for the purposes of updating the configuration (for junction creation etc.).

    These special Reverse Proxy processes are not running all the time. They are started only when they are needed (during junction create etc.) so it is expected to see them NOT running most of the time if you use the pdweb status command.

    If you connect to the Reverse Proxy container and issue the pdweb status command you should see that this is running.
    Be careful about making changes directly on the Reverse Proxy containers.... they will be lost on next reload/restart because configuration will be taken from snapshot again.

    Have a look at the "Access Manager on Docker" part of this course on Security Learning Academy.  It talks about the process of generating configuration in config container and then transferring out to the other containers for execution.

    https://www.securitylearningacademy.com/mod/hvp/view.php?id=11593

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 8.  RE: ISAM9 with Open Shift

    Posted Thu April 02, 2020 07:21 AM
    Hi Jon, I will go through this given document. but from your answer i realize there something different in my setup.
    I can see policy server and webseal instance both are on isamconfig container.

    sh-4.2$ uname -a
    Linux isamconfig-3087490447-89zp7 3.10.0-1062.12.1.el7.x86_64 #1 SMP Thu Dec 12 06:44:49 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
    sh-4.2$ pd_start status

    Security Access Manager servers

    Server Enabled Running Instance

    -------------------------------------------------------

    pdmgrd no yes
    pdacld no no
    pdmgrproxyd no no
    sh-4.2$ pdweb status
    webseald-default-isam9 yes no


    in runtime container i am not able see anything?
    sh-4.2$ uname -a
    Linux isamruntime-116847928-42mjw 3.10.0-1062.12.1.el7.x86_64 #1 SMP Thu Dec 12 06:44:49 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
    sh-4.2$ pdweb status
    sh-4.2$ pd_start status

    Security Access Manager servers

    Server Enabled Running Instance

    -------------------------------------------------------

    pdmgrd no no
    pdacld no no
    pdmgrproxyd no no


    is there anything we missed during our configuration? or will it be fine if we continue with existing setup.

    ------------------------------
    Mayur Wattamwar
    ------------------------------



  • 9.  RE: ISAM9 with Open Shift

    Posted Thu April 02, 2020 07:32 AM
    Hi Mayur,

    I hope the course I've pointed to will help explain.

    You won't see any ISAM "PD*" components in the runtime container... it is only running the AAC/Federation runtime (Java running under WebSphere Liberty).

    You will see the ISAM "PD*" components in the Reverse Proxy container.  Maybe you haven't created that container yet?  It is a separate OpenShift template to be deployed after configuration is created and published.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 10.  RE: ISAM9 with Open Shift

    Posted Sat April 04, 2020 10:23 AM
    Hi Jon,
    I have created reverse proxy container now and publish configuration but still not able to see the instance status as running,  is there anything i need to worry if its not running here.

     $ oc rsh isamwrprp1-1954645295-tw7xq
    sh-4.2$ pdweb status
    sh-4.2$
    sh-4.2$
    sh-4.2$ pd_start status

    Security Access Manager servers

    Server Enabled Running Instance

    -------------------------------------------------------

    pdmgrd no no
    pdacld no no
    pdmgrproxyd no no

    sorry for more question but how we can access webseal URL post junction creation. ?

    ------------------------------
    Mayur Wattamwar
    ------------------------------



  • 11.  RE: ISAM9 with Open Shift

    Posted Sat April 04, 2020 11:26 AM

    Hello Mayur,

    I would expect to see processes running here so I suspect something is wrong.

    what is output from

    oc logs -f  isamwrprp1-xxxxxxx ?

    Maybe the container is failing to retrieve config from the config container for some reason. 


    Jon. 



    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 12.  RE: ISAM9 with Open Shift

    Posted Sat April 04, 2020 12:46 PM
    Hi Jon,
    Thanks for your reply, I can see below logs.

    Error: WGAWA0662E An invalid response code was returned from the request to https://isamconfig:9443/shared_volume/fixpacks: 403
    2020-04-04T17:44:04+0100: ---- Retrying....
    Error: WGAWA0662E An invalid response code was returned from the request to https://isamconfig:9443/shared_volume/fixpacks: 403
    2020-04-04T17:44:15+0100: ---- Retrying....
    Error: WGAWA0662E An invalid response code was returned from the request to https://isamconfig:9443/shared_volume/fixpacks: 403
    2020-04-04T17:44:25+0100: ---- Retrying....
    Error: WGAWA0662E An invalid response code was returned from the request to https://isamconfig:9443/shared_volume/fixpacks: 403

    ------------------------------
    Mayur Wattamwar
    ------------------------------



  • 13.  RE: ISAM9 with Open Shift

    Posted Mon April 06, 2020 12:01 PM
    Mayur,

    These errors indicate that the Reverse Proxy container is failing to authenticate to the Configuration container.
    If you are using my OpenShift templates then the Reverse Proxy is attempting to authenticate using the values from the <app-name>-core secret for this purpose.  The references in the secret are:

    config-read-username and config-read-password.

    This secret is created when deploying the core application.  If you didn't specify values for then the username will be cfgsvc and the password will be dynamically generated.  You should be able to see the values by viewing the secret in the OpenShift console.

    Assuming you are using the cfgsvc user, this account already exists in the Configuration container but probably the password isn't set to match what is in that secret.  Use the LMI to set the password for this account to match.  You'll find it under Manage System Settings-->Account Management.

    This is all documented in section 9.2 of my OpenShift cookbook.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 14.  RE: ISAM9 with Open Shift

    Posted Fri April 10, 2020 01:52 PM

    Hi john,

                   We have upgraded the ISAMfrom 9.0.7.0  to  9.0.7.1 IF4, however we are still facing the same issue were the snapshot is not getting applied from the persistent volume. We have also lost all our configuration during the reboot to apply the fixpack.

     

    Thanks,

     

    Ghiffary Osman I Automation, Integration & Management Services | GTIS Middleware Services

    Tel: +1 (201)-499-2880

    Mob: +1 (240)- 821-5516

    Webex: https://barclays.webex.com/meet/gosman

    Email GOsman@barclaycardus.com

    Barclays, Barclays Technology

    400 Jefferson Park Whippany,NJ 07981, USA

    Respect  |  Integrity  |  Service  |  Excellence  |  Stewardship

    Creating opportunities to rise

     

    P Please consider the environment before printing this email

     


    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 15.  RE: ISAM9 with Open Shift

    Posted Fri April 10, 2020 02:17 PM
    Hello Ghiffary,

    When you start the 9.0.7.1 config container, it should detect the 9.0.7.0 configuration file and update it to 9.0.7.1. The other containers will then read this 9.0.7.1 file.

    What logs do you see in the updated configuration container?  Do you see the 9.0.7.1 snapshot being created in /var/shared/snapshots?

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 16.  RE: ISAM9 with Open Shift

    Posted Fri April 10, 2020 03:57 PM

     

    Hi Jon,

                   You are correct I do see the configuration file 9.0.7.0 published and the new 9.0.7.1 in the snapshot however it's the size of the newly created configuration file is empty and has nothing. Below is a screenshot of what I see in /var/shared/snapshots

     

     

     

    sh-4.2$ pwd

    /var/shared/snapshots

    sh-4.2$ ls -ltrh

    total 63M

    8.2M     Mar 24 19:53     isam_9.0.7.0_20200324-195309.263123_isamconfig-3087490447-89zp7.snapshot

    13M      Mar 31 14:25     isam_9.0.7.0_20200331-142518.789049_isamconfig-3087490447-89zp7.snapshot

    15M      Apr  1 17:23       isam_9.0.7.0_20200401-172340.500042_isamconfig-3087490447-89zp7.snapshot

    17M      Apr  6 17:39       isam_9.0.7.0_published.snapshot

    0             Apr 10 18:01      isam_9.0.7.1_published.snapshot

     

     

     

     

    As for logs I assume you are referring to the logs under /var/log

     

     

    /var/log

    sh-4.2$ ls -ltr

    total 5224

    193                       Oct  1  2019 grubby_prune_debug

    0                            Oct  1  2019 wtmp

    0                            Oct  1  2019 btmp

    0                            Feb  4 05:58 spooler

    0                            Feb  4 05:58 secure

    0                            Feb  4 05:58 maillog

    5544                     Feb  4 05:59 yum.log

    0                           Feb 13 01:51 waagent.log

    384064                               Feb 13 23:27 tallylog

    27857                  Apr 10 14:22 isslum.log

    0                            Apr 10 14:22 wga_notifications.log

    1752292              Apr 10 18:17 lastlog

    110129                               Apr 10 20:45 cron.log

    1225431              Apr 10 20:46 messages

     

    Thanks,

     

    Ghiffary Osman

     

     

    Restricted - Internal

     

    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 17.  RE: ISAM9 with Open Shift

    Posted Fri April 10, 2020 05:05 PM

    Hi john,

                   Inaddition to what I've sent earlier I do see the below logs when I restart the container.

    2020-04-10T21:01:57+0000: ---- Creating path: /var/application.logs/docker-f1784/system

     

    2020-04-10T21:01:57+0000: ---- Creating path: /var/application.logs/docker-f1784/db

    Docker detected

    [/var/shared/snapshots/isam_9.0.7.1_published.snapshot]

    End-of-central-directory signature not found. Either this file is not

    a zipfile, or it constitutes one disk of a multi-part archive. In the

    latter case the central directory and zipfile comment will be found on

    the last disk(s) of this archive.

    unzip: cannot find zipfile directory in one of /var/shared/snapshots/isam_9.0.7.1_published.snapshot or

    /var/shared/snapshots/isam_9.0.7.1_published.snapshot.zip, and cannot find /var/shared/snapshots/isam_9.0.7.1_published.snapshot.ZIP, period.

    [/var/shared/snapshots/isam_9.0.7.1_published.snapshot]

    End-of-central-directory signature not found. Either this file is not

    a zipfile, or it constitutes one disk of a multi-part archive. In the

    latter case the central directory and zipfile comment will be found on

    the last disk(s) of this archive.

    unzip: cannot find zipfile directory in one of /var/shared/snapshots/isam_9.0.7.1_published.snapshot or

    /var/shared/snapshots/isam_9.0.7.1_published.snapshot.zip, and cannot find /var/shared/snapshots/isam_9.0.7.1_published.snapshot.ZIP, period.

    2020-04-10T22:02:08+0100: ---- No configuration snapshot detected. Preparing the container

     

     

    Thanks,

     

    Ghiffary Osman I Automation, Integration & Management Services | GTIS Middleware Services

    Tel: +1 (201)-499-2880

    Mob: +1 (240)- 821-5516

    Webex: https://barclays.webex.com/meet/gosman

    Email GOsman@barclaycardus.com

    Barclays, Barclays Technology

    400 Jefferson Park Whippany,NJ 07981, USA

    Respect  |  Integrity  |  Service  |  Excellence  |  Stewardship

    Creating opportunities to rise

     

    P Please consider the environment before printing this email

     


    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 18.  RE: ISAM9 with Open Shift

    Posted Fri April 10, 2020 05:17 PM
    Ghiffary,

    the errors you are seeing now are related to the 0 length 9.0.7.1 snapshot file.  You will probably need to delete this file and then restart the config container to get it to reattempt the conversion of the 9.0.7.0 file.

    Id be interested to see the log file for that startup for any reason why the conversion is failing.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 19.  RE: ISAM9 with Open Shift

    Posted Mon April 13, 2020 09:46 AM

    Hi Jon;

                   I have removed the file and I have navigated to the startup logs but nothing happened, here is the output of /var/log/messages  

     

    Apr 10 22:41:20 mesa_syslogd[289]: Watchdog started

    Apr 10 22:41:20 mesa_syslogd[289]: Child process started (pid=291)

    Apr 10 22:41:20 mesa_syslogd[291]: Started

    Apr 10 22:41:20 mesa_syslogd[291]: Loading config

    Apr 10 22:41:20 mesa_syslogd[291]: Ready

    Apr 10 22:41:20 mesa_syslogd[291]: Warning: Unable to attach to eventsd; event reporting temporarily disabled

    Apr 10 22:41:22 mesa_control[290]: Found legacy published snapshot: isam_9.0.7.0_published.snapshot

    Apr 10 22:41:24 mesa_config[305]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:41:24 mesa_config[305]: Applying snapshot

    Apr 10 22:41:24 mesa_config[305]: Snapshot file: /var/shared/snapshots/isam_9.0.7.0_published.snapshot

    Apr 10 22:41:24 mesa_config[305]: Found version within /tmp/tmp.ido6cW/Settings: 9.0.7.0

    Apr 10 22:41:24 mesa_config[305]: Found model within /tmp/tmp.ido6cW/Settings: docker

    Apr 10 22:41:28 mesa_config[305]: Error: MesaSecureFileMgr::signatureIsValid(): acme error:failed:idup_se_singlebuffer_unprotect:GSS_S_FAILURE:GSS_S_INVALID_PKCS7_MESSAGE

    Apr 10 22:41:28 mesa_config[305]: Error: The snapshot file cannot be applied because it has been modified!

    Apr 10 22:41:28 mesa_config[305]: GLGSY0010E:: GLG_events:: ||

    Apr 10 22:41:31 mesa_config[325]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:41:31 mesa_config[325]: Error: DPWAP0072E   The key database, rt_profile_keys, does not exist.

    Apr 10 22:41:33 mesa_config[329]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:42:06 mesa_config[501]: Last message repeated 5 times

    Apr 10 22:42:06 mesa_config[501]: Error: DPWAP0072E   The key database, lmi_trust_store, does not exist.

    Apr 10 22:42:08 mesa_config[505]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:42:42 mesa_config[696]: Last message repeated 5 times

    Apr 10 22:42:42 mesa_config[696]: Error: DPWAP0072E   The key database, embedded_ldap_keys, does not exist.

    Apr 10 22:42:44 mesa_config[700]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:43:19 mesa_config[877]: Last message repeated 5 times

    Apr 10 22:43:23 mesa_control[892]: Bootstrapping from last known good policies

    Apr 10 22:43:23 mesa_control[892]: Adapting policy dir=/etc/policies/

    Apr 10 22:43:23 mesa_control[892]: Adapt policy succeeded

    Apr 10 22:43:23 mesa_control[892]: Translating policy dir=/etc/policies/ out=/etc/settings.tmp

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/addr_geo/addr_geo1_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/addr_host/addr_host2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/addr_mask/addr_mask2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/addr_range/addr_range2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/address_collection/address_collection2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/responses/email/email2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: Last message repeated 1 times

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/responses/logdb/logdb2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: Last message repeated 1 times

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/responses/rsyslog/rsyslog2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: Last message repeated 1 times

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/responses/snmp/snmp2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: Last message repeated 1 times

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/authdb/authdb4_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/authorization/authorization3_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/felb/felb6_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/hosts/hosts3_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/lmi/lmi2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/snmpd/snmpd2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/common/sysaccount/sysaccount2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/isam/activation/activation3_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/isam/cluster/cluster7_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/isam/dsc/dsc1_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/isam/locale/locale1_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/isam/rsyslog_forwarder/rsyslog_forwarder1_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/events/events2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mga/runtime_profile/runtime_profile13_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mga/runtime_profile/runtime_endpoints2_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/gw_net/gw_net1_2_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/local_admin_configuration/local_admin_configuration4_0_0.xml] validate passed

    Apr 10 22:43:23 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/system_alerts/system_alerts2_0_0.xml] validate passed

    Apr 10 22:43:24 mesa_translate[901]: XML Schema for [/etc/policies//cml/mesa/device_params/device_params2_0_0.xml] validate passed

    Apr 10 22:43:24 mesa_control[892]: Translate policy succeeded

    Apr 10 22:43:24 mesa_control[892]: Bootstrapping from settings

    Apr 10 22:43:26 mesa_config[908]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:43:26 mesa_config[908]: Bootstrapping

    Apr 10 22:43:26 mesa_config[908]: Enabling /etc/marker.custom.appliance_info.show by creating marker file.

    Apr 10 22:43:26 mesa_config[908]: Enabling /etc/marker.custom.cli.logs by creating marker file.

    Apr 10 22:43:26 mesa_config[908]: Error: Could not open the key database.

    Apr 10 22:43:26 mesa_config[908]: Hostname set to isamconfig-111900860-h9t9p

    Apr 10 22:43:26 mesa_config[908]: Setting kernel log level: 1

    Apr 10 22:43:26 mesa_config[908]: Setting maximum number of core files: 3

    Apr 10 22:43:26 mesa_eventsd: Dropping the queue size from 1000 to 10 due to system restrictions.

    Apr 10 22:43:26 mesa_eventsd_watchdog[919]: Watchdog started

    Apr 10 22:43:26 mesa_config[908]: Trying to start daemon: mesa_syslogd

    Apr 10 22:43:26 mesa_eventsd_watchdog[919]: Child process started (pid=920)

    Apr 10 22:43:26 mesa_eventsd[920]: Started

    Apr 10 22:43:26 mesa_eventsd[920]: Loading config

    Apr 10 22:43:26 mesa_eventsd[920]: Ready

    Apr 10 22:43:26 mesa_syslogd[289]: Watchdog exiting on SIGTERM (sent by 908), terminating child

    Apr 10 22:43:26 mesa_config[908]: Daemon already started but command line changed: mesa_syslogd

    Apr 10 22:43:26 mesa_config[908]: Trying to stop daemon: mesa_syslogd_watchdog

    Apr 10 22:43:26 mesa_syslogd[291]: Received SIGTERM, exiting

    Apr 10 22:43:26 mesa_syslogd[291]: Shutting down

    Apr 10 22:43:26 mesa_syslogd[938]: Watchdog started

    Apr 10 22:43:26 mesa_syslogd[938]: Child process started (pid=939)

    Apr 10 22:43:26 mesa_syslogd[939]: Started

    Apr 10 22:43:26 mesa_syslogd[939]: Loading config

    Apr 10 22:43:26 mesa_syslogd[939]: Ready

    Apr 10 22:43:26 mesa_syslogd[939]: Attached to eventsd; event reporting enabled

    Apr 10 22:43:26 mesa_config[908]: XML Schema for [/etc/policies//cml/common/authorization/authorization3_0_0.xml] validate passed

    Apr 10 22:43:26 mesa_config[908]: GLGSY9025I:: GLG_events:: |component=mesa_config|

    Apr 10 22:43:31 mesa_config[908]: GLGSY9029I:: GLG_events:: ||

    Apr 10 22:43:31 mesa_config[908]: FIPS Checksum Results: Success(1747), Modified(0), Missing(0), Errors(0)

    Apr 10 22:43:31 mesa_config[908]: Creating certificate database /opt/IBM/wlp/usr/servers/default/certs/lmi.db in FIPS mode

    Apr 10 22:43:31 mesa_config[908]: Creating default lmi certificate for isamconfig-111900860-h9t9p in FIPS mode

    Apr 10 22:43:41 mesa_config[1022]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:43:43 mesa_config[1028]: Last message repeated 1 times

    Apr 10 22:43:57 mesa_config[908]: Starting App Server

    Apr 10 22:43:57 su: (to www-data) root on none

    Apr 10 22:45:08 mesa_config[1390]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:45:10 mesa_config[908]: Component Liberty enabled FIPS mode.

    Apr 10 22:45:10 mesa_config[908]: Reading /proc/provhwinfo

    Apr 10 22:45:10 mesa_config[908]: Creating LUM host file for virtual/generic system: /etc/lum/host-info

    Apr 10 22:45:10 mesa_config[908]: host-info model: AMV 5100

    Apr 10 22:45:10 mesa_config[908]: host-info serial: 515c3e5a-6e36-443e-b72e-87a995d5231a

    Apr 10 22:45:10 mesa_config[908]: Executing: sed -i -e 's/^host_type.*/host_type        =S      v;/' /etc/lum/UpdateTypeNames.config

    Apr 10 22:45:10 mesa_config[908]: Prepopulating update history after firmware update

    Apr 10 22:45:10 mesa_config[908]: Writing /etc/lum/update-history.yaml

    Apr 10 22:45:10 mesa_config[908]: Starting iss-lum

    Apr 10 22:45:11 mesa_config[908]: Checking for automatic PAM update after firmware update

    Apr 10 22:45:11 iss-lum[1442]: Component LUM enabled FIPS mode.

    Apr 10 22:45:11 iss-lum[1442]: Notice: Component LUM enabled FIPS mode.

    Apr 10 22:45:11 iss-lum[1442]: Started thread '_asThread-instance' with ID 140034632029952

    Apr 10 22:45:11 iss-lum[1442]: Started thread 'LumTaskThread' with ID 140034623637248

    Apr 10 22:45:11 iss-lum[1442]: Started thread 'UpdateServicesThread' with ID 140034615244544

    Apr 10 22:45:13 mesa_config[1398]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:45:17 rsyslogd:  [origin software="rsyslogd" swVersion="8.24.0-41.el7_7.2" x-pid="1452" x-info="http://www.rsyslog.com"] start

    Apr 10 22:45:17 mesa_config[908]: XML Schema for [/etc/policies//cml/common/sysaccount/sysaccount2_0_0.xml] validate passed

    Apr 10 22:45:17 mesa_control[892]: Bootstrap from settings succeeded

    Apr 10 22:45:17 mesa_control[892]: GLGSY0015I:: GLG_events:: ||

    Apr 10 22:45:17 mesa_control[892]: Bootstrap succeeded

    Apr 10 22:45:18 rsyslogd: rsyslogd's userid changed to 6000

    Apr 10 22:45:23 mesa_config[1474]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:45:23 mesa_config[1474]: Error: WGAWA0279E   An incorrect user name or password has been supplied.

    Apr 10 22:45:23 mesa_config[1474]: GLGSY0021W:: GLG_events:: |user=cfgsvc|

    Apr 10 22:45:23 mesa_config[1474]: GLGSY0100W:: GLG_events:: |user=cfgsvc,host=22.249.13.1|

    Apr 10 22:45:24 mesa_config[1480]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:45:24 mesa_control[892]: Policy changes have been kept.

    Apr 10 22:45:36 mesa_config[1523]: Signature verification passed for file: /usr/sbin/mesa_config

    Apr 10 22:45:36 mesa_config[1523]: Error: WGAWA0279E   An incorrect user name or password has been supplied.

    Apr 10 22:45:36 mesa_config[1523]: GLGSY0021W:: GLG_events:: |user=cfgsvc|

    Apr 10 22:45:36 mesa_config[1523]: GLGSY0100W:: GLG_events:: |user=cfgsvc,host=22.249.13.1|

    Apr 10 22:45:48 mesa_config[1567]: Signature verification passed for file: /usr/sbin/mesa_config

     

     

     

    Also container POD log is still indicating that it's not able to detect a configuration snapshot

     

     

    2020-04-10T21:41:07+0000: Bootstrapping....

     

    2020-04-10T21:41:13+0000: ---- Log files for this container will be written to docker-508b0

    / /

    /

    2020-04-10T21:41:13+0000: ---- Creating path: /var/application.logs/docker-508b0/lmi

    2020-04-10T21:41:13+0000: ---- Creating path: /var/application.logs/docker-508b0/rsyslog_forwarder

    2020-04-10T21:41:14+0000: ---- Creating path: /var/application.logs/docker-508b0/isam_runtime/policy

    2020-04-10T21:41:14+0000: ---- Creating path: /var/application.logs/docker-508b0/isam_runtime/user_registry

    2020-04-10T21:41:14+0000: ---- Creating path: /var/application.logs/docker-508b0/wrp

    2020-04-10T21:41:14+0000: ---- Creating path: /var/application.logs/docker-508b0/system

    2020-04-10T21:41:14+0000: ---- Creating path: /var/application.logs/docker-508b0/db

    Docker detected

    2020-04-10T22:41:29+0100: ---- No configuration snapshot detected. Preparing the container

    2020-04-10T22:41:29+0100: now.

    Verifying checksums... Done

    2020-04-10T22:45:24+0100: --- Running.

    2020-04-10T22:45:24+0100: Log file: /var/application.logs.local/lmi/messages.log

    [4/10/20 22:45:04:896 BST] 0000002b com.ibm.ws.util I SESN0172I: The session manager is using the Java default SecureRandom implementation for session ID generation.

    [4/10/20 22:45:06:080 BST] 0000002d com.ibm.ws.session.WASSessionCore I SESN0176I: A new session context will be created for application key default_host/isam

    [4/10/20 22:45:06:083 BST] 0000002d com.ibm.ws.util I SESN0172I: The session manager is using the Java default SecureRandom implementation for session ID generation.

    [4/10/20 22:45:06:088 BST] 0000002b com.ibm.ws.app.manager.AppMessageHelper A CWWKZ0001I: Application xgs_50 started in 24.997 seconds.

    [4/10/20 22:45:10:181 BST] 00000041 com.ibm.ws.app.manager.AppMessageHelper A CWWKZ0022W: Application isam has not started in 30.098 seconds.

    [4/10/20 22:45:10:485 BST] 00000027 com.ibm.ws.kernel.feature.internal.FeatureManager A CWWKF0012I: The server installed the following features: [appSecurity-2.0, distributedMap-1.0, el-3.0, jaxrs-2.1, jaxrsClient-2.1, jndi-1.0, json-1.0, jsonp-1.1, jsp-2.3, servlet-4.0, ssl-1.0, usr:isam.cluster_filter-8.0.0, usr:isam.guixml_validators-8.0.0, usr:isam.offering_filter-8.0.0, usr:isam.security_filter-8.0.0, usr:isam.tenant_filter-9.0.0, usr:mesa.registry-8.0.0].

     

     

    Thanks,

     

    Ghiffary Osman I Automation, Integration & Management Services | GTIS Middleware Services

    Tel: +1 (201)-499-2880

    Mob: +1 (240)- 821-5516

    Webex: https://barclays.webex.com/meet/gosman

    Email GOsman@barclaycardus.com

    Barclays, Barclays Technology

    400 Jefferson Park Whippany,NJ 07981, USA

    Respect  |  Integrity  |  Service  |  Excellence  |  Stewardship

    Creating opportunities to rise

     

    P Please consider the environment before printing this email

     


    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 20.  RE: ISAM9 with Open Shift

    Posted Mon April 13, 2020 11:41 AM
    Hello Ghiffary,

    Thank you for the log; the important lines are:

    Apr 10 22:41:28 mesa_config[305]: Error: MesaSecureFileMgr::signatureIsValid(): acme error:failed:idup_se_singlebuffer_unprotect:GSS_S_FAILURE:GSS_S_INVALID_PKCS7_MESSAGE

    Apr 10 22:41:28 mesa_config[305]: Error: The snapshot file cannot be applied because it has been modified!

    These show that the 9.0.7.0 snapshot is still not being accepted even with the fixpack. 

    I checked with a development colleague and he has told me that, unfortunately, snapshots taken after the issue started are invalid - even if fix pack subsequently installed.

    It *might* be possible to workaround the issue by changing system time to sometime in Feb 2020 and then attempting the container start.

    Otherwise, depending on how much configuration you have done, the most practical way forward may be to restart your configuration with 9.0.7.1 FP4.

     If you want to dig further into this I would advise opening a support call.

    I'm sorry that you have been caught up in this issue.


    Jon. 



    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 21.  RE: ISAM9 with Open Shift

    Posted Mon April 13, 2020 12:22 PM

    Hi jon,

                   We are fine with reconfiguring the ISAM 9 from scratch, we were discussing this internally today, however I was wondering if we can remove all the previous snapshots that we had for ISAM 9.0.7.0 and then take a new snapshot with the new ISAM IF4 to see if we can upload it.  We have also opened a PMR to have the support team to take a deeper dive on this matter if it occurred again with the new snapshot.

     

    Thanks,

     

    Ghiffary Osman I Automation, Integration & Management Services | GTIS Middleware Services

    Tel: +1 (201)-499-2880

    Mob: +1 (240)- 821-5516

    Webex: https://barclays.webex.com/meet/gosman

    Email GOsman@barclaycardus.com

    Barclays, Barclays Technology

    400 Jefferson Park Whippany,NJ 07981, USA

    Respect  |  Integrity  |  Service  |  Excellence  |  Stewardship

    Creating opportunities to rise

     

    P Please consider the environment before printing this email

     


    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 22.  RE: ISAM9 with Open Shift

    Posted Tue April 14, 2020 09:12 AM
    Hi Ghiffary,

    If you remove all previous snapshots and restart the configuration container, it should come up with no configuration.  At that point you can perform your basic configuration and then take a new snapshot.  That snapshot will be a 9.0.7.1 snapshot file and, as long as you have 9.0.7.1 IF4 running, that snapshot should be fine and work to bring up other ISAM containers.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 23.  RE: ISAM9 with Open Shift

    Posted Wed April 15, 2020 03:39 PM

    Hi Jon,

                   Thank you for your support, we were able to get the snapshot picked up after deleting the pod and cleared out all files under the persistence volume; although I doubt that this action fixed the problem. However I have a quick question about the web proxy server; I have noticed that under ENV that there was a variable for INSTANCE_NAME, I was wondering if there is a way that we can configure the proxy server to host multiple instances instead of having only one?

     

     

     

     

    Thanks,

     

    Ghiffary Osman I Automation, Integration & Management Services | GTIS Middleware Services

    Tel: +1 (201)-499-2880

    Mob: +1 (240)- 821-5516

    Webex: https://barclays.webex.com/meet/gosman

    Email GOsman@barclaycardus.com

    Barclays, Barclays Technology

    400 Jefferson Park Whippany,NJ 07981, USA

    Respect  |  Integrity  |  Service  |  Excellence  |  Stewardship

    Creating opportunities to rise

     

    P Please consider the environment before printing this email

     


    Restricted - External

    Barclaycard

    www.barclaycardus.com

    This email and any files transmitted with it may contain confidential and/or proprietary information. It is intended solely for the use of the individual or entity who is the intended recipient. Unauthorized use of this information is prohibited. If you have received this in error, please contact the sender by replying to this message and delete this material from any system it may be on.






  • 24.  RE: ISAM9 with Open Shift

    Posted Wed March 25, 2020 09:27 AM

    Hi Jon,
    I work along with Mayur on this, we have already assigned a persistent volume for /var/shared but we are still facing the same issue where our licenses and all configurations related to database and ldap are getting lost due to container restart. I have also noticed that the snapshot only taken once 6 days ago with no new snapshots, I also took a manual snapshot and tried to upload it but it gave us an error saying "The system failed to apply the snapshot. The system encountered an error while it was attempting to apply the changes, and has reverted all values to the previous state. Contact IBM Customer Support":

    is there any way to configure the snapshot where we can control the frequency of taken them and rotation. 



    apiVersion: v1
    kind: Pod
    metadata:
    annotations:
    kubernetes.io/created-by: >
    {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"16697","name":"isamconfig-3087490447","uid":"d8244422-6d07-11ea-a962-005056ad97f3","apiVersion":"extensions","resourceVersion":"195994837"}}
    kubernetes.io/limit-ranger: >-
    LimitRanger plugin set: cpu, memory request for container isamconfig; cpu,
    memory limit for container isamconfig
    openshift.io/scc: isam-scc
    creationTimestamp: '2020-03-23T13:11:55Z'
    generateName: isamconfig-3087490447-
    labels:
    app: isamconfig
    pod-template-hash: '3087490447'
    name: isamconfig-3087490447-89zp7
    namespace: '16697'
    ownerReferences:
    - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: isamconfig-3087490447
    uid: d8244422-6d07-11ea-a962-005056ad97f3
    resourceVersion: '197861671'
    selfLink: /api/v1/namespaces/16697/pods/isamconfig-3087490447-89zp7
    uid: de5bfe54-6d07-11ea-a962-005056ad97f3
    spec:
    containers:
    - env:
    - name: SERVICE
    value: config
    - name: CONTAINER_TIMEZONE
    value: USA/Miami
    - name: ADMIN_PWD
    valueFrom:
    secretKeyRef:
    key: adminpw
    name: samadmin
    image: 'ibmcorp/isam:9.0.7.0'
    imagePullPolicy: IfNotPresent
    livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 500
    periodSeconds: 20
    successThreshold: 1
    tcpSocket:
    port: 9443
    timeoutSeconds: 1
    name: isamconfig
    ports:
    - containerPort: 9443
    protocol: TCP
    resources:
    limits:
    cpu: 500m
    memory: 2100Mi
    requests:
    cpu: 50m
    memory: 200Mi
    securityContext:
    capabilities:
    add:
    - CHOWN
    - DAC_OVERRIDE
    - FOWNER
    - KILL
    - NET_BIND_SERVICE
    - SETFCAP
    - SETGID
    - SETUID
    drop:
    - ALL
    privileged: false
    runAsNonRoot: true
    runAsUser: 6000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/shared
    name: isamconfig
    - mountPath: /var/application.logs
    name: samconfig
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: isam-token-lv9fd
    readOnly: true
    dnsPolicy: ClusterFirst
    imagePullSecrets:
    - name: isam-dockercfg-7vgfg
    nodeName: server.com
    nodeSelector:
    region: user
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
    runAsNonRoot: true
    runAsUser: 6000
    serviceAccount: isam
    serviceAccountName: isam
    terminationGracePeriodSeconds: 30
    volumes:
    - name: isamconfig
    persistentVolumeClaim:
    claimName: isamconfig
    - name: samconfig
    persistentVolumeClaim:
    claimName: samconfig
    - emptyDir: {}
    name: isamconfig-logs
    - name: isam-token-lv9fd
    secret:
    defaultMode: 420
    secretName: isam-token-lv9fd
    status:
    conditions:
    - lastProbeTime: null
    lastTransitionTime: '2020-03-23T13:11:55Z'
    status: 'True'
    type: Initialized
    - lastProbeTime: null
    lastTransitionTime: '2020-03-24T20:24:49Z'
    status: 'True'
    type: Ready
    - lastProbeTime: null
    lastTransitionTime: '2020-03-23T13:11:55Z'
    status: 'True'
    type: PodScheduled
    containerStatuses:
    - containerID: >-
    docker://6d80733936e52c3373113257f888c0724288fb5e9c268932971a2031753f5423
    image: registry/ibmcorp/isam:9.0.7.0'
    imageID: >-
    docker-pullable://registry/ibmcorp/isam@sha256:aa3eafb6bbcb0255cf5d929054940d17872bc1ad275d4b9346e6d47b18989d16
    lastState:
    terminated:
    containerID: >-
    docker://0efefa5cbec20206e492b8c92694ac5085072410a85bd92ba44a697646110f4e
    exitCode: 0
    finishedAt: '2020-03-24T20:23:27Z'
    reason: Completed
    startedAt: '2020-03-23T13:12:35Z'
    name: isamconfig
    ready: true
    restartCount: 1
    state:
    running:
    startedAt: '2020-03-24T20:23:37Z'
    hostIP: 179.2.125.24
    phase: Running
    podIP: 10.25.22.30
    qosClass: Burstable
    startTime: '2020-03-23T13:11:55Z'



    ------------------------------
    Ghiffary Osman
    ------------------------------