Instana U

 View Only
Expand all | Collapse all

AKS Monitoring In Instana

  • 1.  AKS Monitoring In Instana

    Posted Fri August 02, 2024 05:26 PM

    Hi,

    Can someone help us with the list of metrics which we would get from Instana for AKS cluster.

    Does Instana give the logs @ pod level and will It discover all the services\APIs running In AKS post deploying the Instana agent on AKS cluster or do we need to perform any other configuration apart from agent Installation to get the logs, traces and all API call details?

    It would be great If you can help us with a sample SOP or sheet which we get If we onboard AKS cluster for monitoring In Instana.

    Thanks,

    Sravan



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------



  • 2.  RE: AKS Monitoring In Instana

    Posted Mon August 05, 2024 09:48 AM
    Hi, some questions to help you.

    We have onboarded AKS cluster for Instana monitoring but we are not able to see the log messages and traces when we click on Application and navigate to Logs page
    Q: Do you see the cluster in Instana (left hand menu, Platforms / Kubernetes) ?

    Can someone help us with the list of metrics which we would get from Instana for AKS cluster.

    Does Instana give the logs @ pod level and will It discover all the services\APIs running In AKS post deploying the Instana agent on AKS cluster or do we need to perform any other configuration apart from agent Installation to get the logs, traces and all API call details?
    Q: Almost 100% yes. What are the languages used to develop your microservices/API ?
    For nodejs, net.core, ruby & python you should install the Autotrace Webhook, https://www.ibm.com/docs/en/instana-observability/current?topic=kubernetes-instana-autotrace-webhook. Java, it's derivatives & PHP work out of the box

    A: For container logs, add the following directive to the agent's config map:
    com.instana.plugin.containerd:
      logs:
        enabled: true
        sendInterval: 60
        maxBufferSize: 16777216

    Regards,

    Luis






    Original Message


  • 3.  RE: AKS Monitoring In Instana

    Posted Tue August 06, 2024 01:40 AM
    Edited by Sravankumar Chandupatla Tue August 06, 2024 01:44 AM

    Hi,

    Yes, able to see the kubernetes cluster In the Instana UI,PFB.

    Below config should be added In the Instana.yaml file, right?

    com.instana.plugin.containerd:
      logs:
        enabled: true
        sendInterval: 60
        maxBufferSize: 16777216

    Auto trace Webhook, language used to develop the microservices Is .NET. Request your help on how to Install this auto trace webhook



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 4.  RE: AKS Monitoring In Instana

    Posted Tue August 06, 2024 07:37 AM

    Hello Sravankumar,

    1. Yes, containerd configuration goes in the Agent configuration yaml
    2. You can install the Autotrace webhook using Helm chart. 

    Thanks



    ------------------------------
    Mary Reni Rajaian
    ------------------------------

    Original Message


  • 5.  RE: AKS Monitoring In Instana

    Posted Tue August 06, 2024 07:58 AM

    HI,

    Thanks for the quick reply.

    So, I just need to add the key @download_key and install the below command on the cluster, right? OR do I need to add any other detail in the command.

    helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \ --repo https://agents.instana.io/helm instana-autotrace-webhook \ --set webhook.imagePullCredentials.password=<download_key>

    Thanks,

    SravanKumar



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 6.  RE: AKS Monitoring In Instana

    Posted Tue August 06, 2024 10:07 AM
    Yes, that's all.

    After the autotrace's deployment restart/recreate your own deployments for the autotrace to instrument the services automagically.

    Regards,

    Luis




    Original Message


  • 7.  RE: AKS Monitoring In Instana

    Posted Wed August 07, 2024 10:59 PM

    Sure Luis, will try the suggested configuration and keep you posted the updates.

    Thanks,

    Sravan Kumar



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 8.  RE: AKS Monitoring In Instana

    Posted Mon August 12, 2024 02:13 PM
    Edited by Sravankumar Chandupatla Tue August 13, 2024 02:14 AM
      |   view attached

    Hi,

    The attached auto trace-do we need to execute it as-is, or should we change the below environment variables as per our Instana environment?

    #set CoreCLR tracing environment variables

    ENV DOTNET_STARTUP_HOOKS

    ENV CORECLR_ENABLE_PROFILING

    ENV CORECLR_PROFILER

    ENV CORECLR_PROFILER_PATH

    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 9.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 01:14 AM

    Hi Team,

    Can I have an update on this?

    Thanks,

    Sravan Kumar



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 10.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 02:53 AM

    Hi,

    Is It possible to come on a screenshare call for few minutes, once you are available. So that you can help us to fix the Issue asap.

    As this has been pending for 2-3 weeks.

    Thanks,

    Sravan



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 11.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 03:21 AM

    Dear Mr. Chandupatla,

    from your previous post it is not clear there is an issue, at all.
    As I said, the context of the query is unknown and we do need more info to even give a priority to the question and decide when to schedule our help.


    Please do reach out to @brewery-agent and explain what is the issue.
    Team Agent or Team Profilers and Proxies will answer briefly there or if there is availability and the need we will schedule a videocall.


    Sincerely,



    ------------------------------
    Teresa Noviello
    Senior Software Engineer, Team Proxies, Profilers and eBPF, Instana - IBM
    Instana Engineering Team

    IBM Corporation
    c/o Instana
    71 S Wacker Drive, 7th floor
    Chicago, IL 60606
    ------------------------------

    Original Message


  • 12.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 02:53 AM

    Dear Mr.,
    the short answer to your question, if you need to set up all those variables for the autotrace correct behavior, is No.
    As you can se from the installation guide for the end customer, at https://www.ibm.com/docs/en/instana-observability/current?topic=kubernetes-instana-autotrace-webhook, they are not mentioned as mandatory.
    Another way to check their significance and default values is looking into the autotrace code.
    If you need more help, please expand a bit the context, such as:
    - Are you trying to build the Kubernetes locally?
    - On which platform is the autotrace installed?
    - Which guide are you following?
    - Is the autotrace malfunctioning?

    You can also reach the developers on Slack, @brewery-agent channel.

    Best,
    Teresa Noviello



    ------------------------------
    Teresa Noviello
    Senior Software Engineer, Team Proxies, Profilers and eBPF, Instana - IBM
    Instana Engineering Team

    IBM Corporation
    c/o Instana
    71 S Wacker Drive, 7th floor
    Chicago, IL 60606
    ------------------------------

    Original Message


  • 13.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 04:42 AM
    Edited by Sravankumar Chandupatla Tue August 13, 2024 04:46 AM

    Hi,

    I do not see the @brewery-agent channel In Slack with my login.

    As mentioned In last comment may I know which exact page we need to look In the following link :  https://www.ibm.com/docs/en/instana-observability/current?topic=kubernetes-instana-autotrace-webhook

    Thanks,

    Sravan



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 14.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 08:55 AM

    Dear Mr. Chandupatla,

    I am not sure, but I think you are in IBM consulting?
    This is probably the reason why you can't see the beforementioned channel.
    In order to organize and schedule the best support for you, I suggest you to open a CSP ticket.
    Please provide a detailed description of the problem in the ticket. No worries, if something is missing the customer team will circle back to you.
    If you have doubts, please ask your manager.

    Best,



    ------------------------------
    Teresa Noviello
    Senior Software Engineer, Team Proxies, Profilers and eBPF, Instana - IBM
    Instana Engineering Team

    IBM Corporation
    c/o Instana
    71 S Wacker Drive, 7th floor
    Chicago, IL 60606
    ------------------------------

    Original Message


  • 15.  RE: AKS Monitoring In Instana

    Posted Tue August 13, 2024 09:48 AM
    No, those variables will be managed by the webhook.

    I encourage you to deploy the webhook at your earliest opportunity. Remember to restart your deployments one by one, to allow the webook to inject the autotrace into each one. Look at the autotrace's pod to see it in action.

    Regards,

    Luis Sánchez @ Impulse IT





    Original Message


  • 16.  RE: AKS Monitoring In Instana

    Posted Wed August 14, 2024 02:34 AM
      |   view attached

    Hi,  We have deployed the Instana agent to the cluster using the command kubectl apply -f deployment.yaml. While we are able to see the resource-level metrics in Instana, the application-level logs and traces are not appearing on pods and containers. To troubleshoot the issue, as mentioned in the documentation and as suggested by you, we implemented the attached recommended Auto trace webhook configuration on one of the microservices, but we encountered errors."

    Exception from Pod log :

    Unhandled exception. System.ArgumentException: Startup hook assembly '/app/Instana.Tracing.Core.dll' failed to load. See inner exception for details.
     ---> System.IO.FileNotFoundException: Could not load file or assembly '/app/Instana.Tracing.Core.dll'. The system cannot find the file specified.
     
    File name: '/app/Instana.Tracing.Core.dll'
       at System.Runtime.Loader.AssemblyLoadContext.LoadFromPath(IntPtr ptrNativeAssemblyLoadContext, String ilPath, String niPath, ObjectHandleOnStack retAssembly)
       at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
       at System.StartupHookProvider.CallStartupHook(StartupHookNameOrPath startupHook)
       --- End of inner exception stack trace ---
       at System.StartupHookProvider.CallStartupHook(StartupHookNameOrPath startupHook)
       at System.StartupHookProvider.ProcessStartupHooks()
    Request your help to troubleshoot\fix the issue.
    Thanks,
    Sravan Kumar



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 17.  RE: AKS Monitoring In Instana

    Posted Wed August 14, 2024 09:29 AM
    Hi,

    As far as I understand from your message you deployed the agent, and then changed one of the deployments to include those ENV variables. It didn't work, however this is expected. Setting those variables also requires to include the Instana tracer DLL into your .NET project.

    The autotrace webhook does these steps automagically. In short the steps are:
    1. Need container logs ?
    Add the directive to the configmap of agent's deployment:
    com.instana.plugin.containerd:
      logs:
        enabled: true
        sendInterval: 60
        maxBufferSize: 16777216

    2. Deploy the agent:
    kubectl apply -f deployment.yaml

    3. Deploy the Autotrace Webhook:
    helm install --create-namespace --namespace instana-autotrace-webhook instana-autotrace-webhook \
      --repo https://agents.instana.io/helm instana-autotrace-webhook \
      --set webhook.imagePullCredentials.password=DOWNLOAD_KEY
    Replace DOWNLOAD_KEY with yours (Look for the key in the Linux One Liner installation):
    curl -o setup_agent.sh https://setup.instana.io/agent ... && ... sudo ./setup_agent.sh -a Oe95XXXXXXXXXXXXXXXXGQ ...

    4. Redeploy/restart your deployments. Don't include INSTANA's ENV variables, let the Autotrace Webhook do it's job

    5. Enjoy :-) !

    Regards,

    Luis





    Original Message


  • 18.  RE: AKS Monitoring In Instana

    Posted Fri August 16, 2024 06:50 AM

    Hi,

    Is it possible for you to schedule a call to help us with the issue resolution?

    Thanks,

    Sravan



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message


  • 19.  RE: AKS Monitoring In Instana

    Posted Sat August 17, 2024 02:44 PM
    We could speak, However, before doing the call please share the output of the following script:

    #!/bin/sh
    ###############################################################################
    #
    # This script is used to collect data for
    # the Instana Host Agent on Kubernetes / Openshift
    #
    # ./instana-k8s-mustgather.sh
    #
    current_time=$(date "+%Y.%m.%d-%H.%M.%S")
    export MGDIR=instana-mustgather-$current_time
    mkdir -p $MGDIR
    if command -v oc > /dev/null
    then
      CMD=oc
      LIST_NS="instana-agent instana-autotrace-webhook openshift-controller-manager"
    else
      CMD=kubectl
      LIST_NS="instana-agent instana-autotrace-webhook"
    fi

    $CMD get nodes > $MGDIR/node-list.txt
    $CMD describe nodes > $MGDIR/node-describe.txt
    $CMD get namespaces > $MGDIR/namespaces.txt
    $CMD describe cm instana-agent -n instana-agent > $MGDIR/configMap.txt

    if [ $CMD == "oc" ]
    then
      $CMD get clusteroperators > $MGDIR/cluster-operators.txt
    fi
    $CMD get pods -n instana-agent -o wide > $MGDIR/instana-agent-pod-list.txt
    if [ $CMD == "oc" ]
    then
      $CMD get pods -n openshift-controller-manager -o wide > $MGDIR/openshift-controller-manager-pod-list.txt
    fi
    for NS in $LIST_NS; do
      export NS=$NS
      mkdir $MGDIR/$NS
      $CMD get all,events -n $NS -o wide &> $MGDIR/$NS/all-list.txt
      $CMD get pods -n $NS | awk 'NR>1{print "'$CMD' -n $NS describe pod "$1" > $MGDIR/$NS/"$1"-describe.txt && echo described "$1}' | bash
      $CMD get pods -n $NS -o go-template='{{range $i := .items}}{{range $c := $i.spec.containers}}{{println $i.metadata.name $c.name}}{{end}}{{end}}' > $MGDIR/$NS/container-list.txt
      awk '{print "'$CMD' -n $NS logs "$1" -c "$2" --tail=10000 > $MGDIR/$NS/"$1"_"$2".log && echo gathered logs of "$1"_"$2}' $MGDIR/$NS/container-list.txt | bash
      awk '{print "'$CMD' -n $NS logs "$1" -c "$2" --tail=10000 -p > $MGDIR/$NS/"$1"_"$2"_previous.log && echo gathered previous logs of "$1"_"$2}' $MGDIR/$NS/container-list.txt | bash
    done
    tar czf $MGDIR.tgz $MGDIR/

    Copy/paste the script in your cloud shell or whatever you use to manage your cluster and run with:
    bash the-name-of-the-script

    The output is a file with the name instana-mustgather-YYYY.MM.DD-hh.mm.ss.tgz. Post the file into the thread to take a look at it.

    Regards,

    Luis





    Original Message


  • 20.  RE: AKS Monitoring In Instana

    Posted Mon August 19, 2024 12:42 AM

    Hi,

    Sure, we will run the script and share the output shortly.

    Thanks,

    Sravan



    ------------------------------
    Sravankumar Chandupatla
    ------------------------------

    Original Message