StreamSets

StreamSets

Connect with experts and peers to elevate technical expertise, solve problems and share insights.

 View Only

Do we have any way to know which job/pipeline taking more memory on SDC.

  • 1.  Do we have any way to know which job/pipeline taking more memory on SDC.

    Posted Thu December 14, 2023 10:22 AM

     

    Currently, JVM (openjdk/Oracle) only exposes overall CPU or memory metrics for a process.As of now, it's not possible at jvm level to pull the individual pipeline memory details.

    To know more about the the memory consumption we can analyse heap dumps ,thread dumps  and gc.log when the issue is happening.
     
    heap dumps:
    sudo -u sdc jcmd <pid>  GC.heap_dump <filename>.hprof
     
    Thread dumps:

    for i in {0..5} ; do echo $i ; jstack {pid}  > /tmp/sdc-$(date +%Y-%m-%d.%H:%M:%S).jstack ; sleep 5 ; done