Maximo

 View Only
Expand all | Collapse all

UDS instance creation fails during MAS/Manage installation

  • 1.  UDS instance creation fails during MAS/Manage installation

    Posted Tue November 28, 2023 11:39 AM

    I am running OCP 4.12.42 in SNO and trying to install MAS 8.11.2, using the latest MAS-CLI (7.8.0)

    Execute the mas install command, input in all the parameters, I am installing Manage and DB2 as well, very vanilla setup, not customizations. 

    The pipeline install keeps failing at User Data Services, waiting for AnalyticsProxy to be ready.

    DB2-manage and mongodb will eventually fail as well, which I suspect because UDS couldn't install correctly.

    Has anyone been able to install MAS 8.11.2 with MAS-CLI 7.8.0?

    thanks,

    Bennet



    ------------------------------
    Bennet Tan
    ------------------------------


  • 2.  RE: UDS instance creation fails during MAS/Manage installation

    IBM Champion
    Posted Tue November 28, 2023 07:15 PM

    Can you see any errors on analitcsproxy installation?

    Go to 

    1. Installed Operators
    2. Common Services Project
    3. user-data-services-operator
    4. AnalyticsProxy details

    Try looking into the resources and events for what is happening.



    ------------------------------
    Maycon Belfort
    Consultant
    BPD Zenith
    Melbourne
    Australia
    ------------------------------



  • 3.  RE: UDS instance creation fails during MAS/Manage installation

    Posted Tue November 28, 2023 08:36 PM

    All installed operators under ibm-common-services are in Succeeded status.
    Looking at the IBM User Data Service operator (v2.0.12), analyticsproxy is stuck in installing state with an 'unknown playbook failure' error in the YAML.

    The instrumentationdb-instance1-xxxxx pod under ibm-common-services project is in CrashLoopBackOff status, below are the error logs:

    =================================================================================================
    creating directory /pgdata/pg13 ... ok
    creating directory /pgdata/pg13_wal ... ok
    creating subdirectories ... ok
    selecting dynamic shared memory implementation ... posix
    selecting default max_connections ... 20
    selecting default shared_buffers ... 400kB
    selecting default time zone ... UTC
    creating configuration files ... ok
    running bootstrap script ... 2023-11-29 00:58:03.242 UTC [94911] FATAL: data directory "/pgdata/pg13" has wrong ownership
    2023-11-29 00:58:03.242 UTC [94911] HINT: The server must be started by the user that owns the data directory.
    child process exited with exit code 1
    initdb: removing data directory "/pgdata/pg13"
    initdb: removing WAL directory "/pgdata/pg13_wal"
    pg_ctl: database system initialization failed
    2023-11-29 00:58:03,780 INFO: removing initialize key after failed attempt to bootstrap the cluster

    ================================================================================================

    The mas-mongo-ce pods under mongoce project failing to spin up, below are the logs:

    =================================================================================================
    {"t":{"$date":"2023-11-29T01:16:24.765+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1701220584:765218][1:0x7f1343592cc0], connection: __posix_open_file, 808: /data/WiredTiger.wt: handle-open: open: File exists"}}
    {"t":{"$date":"2023-11-29T01:16:24.770+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1007"}}
    {"t":{"$date":"2023-11-29T01:16:24.774+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1701220584:774463][1:0x7f1343592cc0], connection: __posix_open_file, 808: /data/WiredTiger.wt: handle-open: open: Operation not permitted"}}
    {"t":{"$date":"2023-11-29T01:16:24.777+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
    {"t":{"$date":"2023-11-29T01:16:24.777+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"1: Operation not permitted"}}
    {"t":{"$date":"2023-11-29T01:16:24.777+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":958}}
    {"t":{"$date":"2023-11-29T01:16:24.777+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

    ====================================================================================================

    Given that this installation is ran through the mas-cli which uses the ansible playbook, I am wondering if there is a wrong version implemented here. Hoping I can get the attention of @Alex Donatelli to confirm.



    ------------------------------
    Bennet Tan
    ------------------------------



  • 4.  RE: UDS instance creation fails during MAS/Manage installation

    IBM Champion
    Posted Tue November 28, 2023 09:22 PM

    You can also report this issue in the GitHub repository.

    https://github.com/ibm-mas/cli/issues



    ------------------------------
    Maycon Belfort
    Consultant
    BPD Zenith
    Melbourne
    Australia
    ------------------------------



  • 5.  RE: UDS instance creation fails during MAS/Manage installation

    Posted Wed November 29, 2023 08:51 AM

    I would create a Support Case. It may require investigation. My guess is that you are running into storage/permission issues. I would check to make sure you are using cluster-admin account, and your PVCs (in all the relevant Projects) are correctly bound. Regardless, create a Support Case with IBM.



    ------------------------------
    Arif Ali
    ------------------------------



  • 6.  RE: UDS instance creation fails during MAS/Manage installation

    IBM Champion
    Posted Wed November 29, 2023 02:07 AM

    Hi Bennet,

    i would recommend not using UDS anymore as IBM is sunsetting it next year. The alternative is either Slim UDS or DRO.



    ------------------------------
    Witold Wierzchowski
    ------------------------------



  • 7.  RE: UDS instance creation fails during MAS/Manage installation

    Posted Thu January 18, 2024 05:08 PM

    This issue was resolved by rebuilding the cluster and using the LVM operator as the storage. We were trying to use the NFS provisioner before (as used in a traditional cluster deployment) and that was the reason for this error. My conclusion is that an SNO deployment must utilize the LVM operator for storage.



    ------------------------------
    Bennet Tan
    ------------------------------



  • 8.  RE: UDS instance creation fails during MAS/Manage installation

    IBM Champion
    Posted Thu January 18, 2024 05:10 PM

    Bennet,

     

    It must depend on the NFS driver you are using as I was able to deploy with NFS storage.

     

    Jason






  • 9.  RE: UDS instance creation fails during MAS/Manage installation

    Posted Thu January 18, 2024 05:25 PM
    Edited by Bennet Tan Thu January 18, 2024 05:25 PM

    That's very interesting, I was using this NFS Provisioner. Curious if you were using the same. And what permission settings did you set on the underlying directory for the PVCs?



    ------------------------------
    Bennet Tan
    ------------------------------



  • 10.  RE: UDS instance creation fails during MAS/Manage installation

    IBM Champion
    Posted Thu January 18, 2024 05:40 PM

    That is the same driver / storage class that I am using.  Because I am using this as a development SNO instance the mounts are configured without security restrictions, so that may be the cause of your issue, but that is only speculation.

     

    Jason