Cloud Pak for Data

Cloud Pak for Data

Come for answers. Stay for best practices. All we’re missing is you.

 View Only

  • 1.  java.io.IOException: No space left on device

    Posted Tue April 07, 2020 09:50 PM
    Edited by System Admin Fri January 20, 2023 04:45 PM
      |   view attached
    Hello,

    Some pods got not to work well.
    ---------
    disco-watson-discovery-data-prep-agent-c47cf49bf-8mqk8 0/1 Init:0/1 0 22h
    disco-watson-discovery-dfs-induction-5fb744c787-gffsk 0/1 Init:0/5 0 1d
    disco-watson-discovery-elastic-0 0/1 CrashLoopBackOff 24 20h
    disco-watson-discovery-haywire-7cfcf5bc68-2lv4r 0/1 Init:0/1 0 22h
    disco-watson-discovery-project-data-prep-agent-c4d4ffc6d-bt5tk 0/1 Init:0/1 0 22h
    disco-watson-discovery-training-agents-7776695c7b-hhw8x 0/2 Init:0/1 0 22h
    disco-watson-discovery-training-data-crud-74cb89cf6c-vgkdz 0/1 Init:0/3 0 1d
    dv-0 2/3 Running 62 21h
    dv-caching-5f4f4fdd6c-hlrlw 0/1 Init:0/1 2 21h
    ---------
    As I checked the log of the container in disco-watson-discovery-elastic-0, I found the message below.
    "Caused by: java.io.IOException: No space left on device"


    I attached the results of df -Th on one of master/worker nodes.
    I just concerned about the USE percentage of / on worker nodes (sometimes points 84% at maximum), and I wonder I should add another worker node. *df -i seems to be not problem.

    And, in the first place, I do not know which space(/ or ohter one?) the message points.

    I appreciate if someone help me out.

    Thank you,
    Chris




    ------------------------------
    Chris
    ------------------------------
    #CloudPakforDataGroup

    Attachment(s)

    txt
    dfth_result.txt   34 KB 1 version


  • 2.  RE: java.io.IOException: No space left on device

    Posted Wed April 08, 2020 04:43 AM
    Hi,

    I think we need a support case for this, as this may require some data from you.

    What I would do:

    - check the PVC used by the 'disco-watson-discovery-elastic-0' pod (you can see it from oc describe po)
    - check the PVC parameters (what size/storage class) and the underlying PV (where it is based, etc)

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 3.  RE: java.io.IOException: No space left on device

    Posted Wed April 08, 2020 08:48 PM
    Hi,

    Thank you for your cooperation.
    I attached information you required.

    By the way, I found the support page maybe related to this issue.
    This report is so similar to mine that I will try it once.
    https://www.ibm.com/support/pages/node/5737749

    Thank you,
    Chris

    ------------------------------
    Chris
    ------------------------------

    Original Message


  • 4.  RE: java.io.IOException: No space left on device

    Posted Wed April 08, 2020 10:58 PM
      |   view attached
    Hi,

    I forgot to attach the file.

    ------------------------------
    Chris
    ------------------------------

    Attachment(s)

    txt
    info_elastic.txt   6 KB 1 version
    Original Message


  • 5.  RE: java.io.IOException: No space left on device

    Posted Thu April 09, 2020 06:40 AM
    Hi,


    Yes the link looks good, please let me know if it helps.

    Thanks

    ------------------------------
    TOMASZ HANUSIAK
    ------------------------------

    Original Message


  • 6.  RE: java.io.IOException: No space left on device

    Posted Thu April 09, 2020 08:27 PM
    Hi,

    The link solved my issue as well.

    Thank you,
    Chris

    ------------------------------
    Chris
    ------------------------------

    Original Message


Related Content