webMethods

 View Only
Expand all | Collapse all

OLTP Collector Connection Issue

  • 1.  OLTP Collector Connection Issue

    Posted 28 days ago

    I am encountering an issue with the OLTP collector connection. On the Integration Server (IS) page, the status displays:
    "End-to-End Monitoring Connection Status: offline Offline."

    Below are the steps I have followed:

    1. Installed WmE2EMIntegrationAgent version 10.15.0.10.122.
    2. Updated the agent.config file with the OTLP properties, including: exporter.url=http://localhost:4317
    3. Installed otelcol-contrib.exe and updated the config.yaml accordingly.
    4. Restarted the server to apply the changes.

      Logs:
    plaintextCopy codeINFO 2024-12-17 07:58:20:447 main ServiceManager :        E2E Monitoring Agent 10.15.0.0008-0107 is starting up
...
ERROR 2024-12-17 07:59:19:466 Server Thread Pool OtlpHttpChannelManager :  Error while updating pass fields in External Config map : Cannot invoke "com.softwareag.uhm.agent.exporter.plugin.helper.TargetServerConfig.put(String, String)" because "org.apache.skywalking.apm.agent.core.remote.OtlpHttpChannelManager.externalConfig" is null
...
ERROR 2024-12-17 08:03:30:276 skywalking service shutdown thread ServiceManager :  ServiceManager try to shutdown [org.apache.skywalking.apm.agent.core.jvm.JVMService] fail.
java.lang.NullPointerException: Cannot invoke "java.util.concurrent.ScheduledFuture.cancel(boolean)" because "this.collectMetricFuture" is null


    ------------------------------
    HARI PRASANNA RAJU CH
    ------------------------------


  • 2.  RE: OLTP Collector Connection Issue

    Posted 28 days ago

    Hello, Hari.

    May you show us your agent.config and config.yaml as well? This may help on investigating.



    ------------------------------
    Gabriel Leme Solutions Architect
    Solutions Architect
    CXP Brasil
    São Paulo
    +55 (11)914734869
    ------------------------------

    Original Message


  • 3.  RE: OLTP Collector Connection Issue

    Posted 28 days ago

    Hi Gabriel,

    Attached the agent.config and config.yaml 



    ------------------------------
    HARI PRASANNA RAJU CH
    ------------------------------

    Original Message


  • 4.  RE: OLTP Collector Connection Issue

    Posted 28 days ago

    Thanks, Hari.

    I have some more questions,

    The IS you're trying to monitor is a on-premises? If so,

    1. You're able to reach 4317 and 8889 ports? (Telnet or something)
    2. Which OS you're running?
    3. You've confirmed you don't have networking/firewall restrictions?


    ------------------------------
    Gabriel Leme Solutions Architect
    Solutions Architect
    CXP Brasil
    São Paulo
    +55 (11)914734869
    ------------------------------

    Original Message


  • 5.  RE: OLTP Collector Connection Issue

    Posted 28 days ago

    The IS you're trying to monitor is a on-premises? Yes

    1. You're able to reach 4317 and 8889 ports? (Telnet or something) Unable to reac 4317, able to reach 8889
    2. Which OS you're running? Windows
    3. You've confirmed you don't have networking/firewall restrictions? No restrictions

    Note: I am trying in my local system



    ------------------------------
    Peter Vanbellingen
    ------------------------------

    Original Message


  • 6.  RE: OLTP Collector Connection Issue

    Posted 28 days ago

    Hello, Peter.

    So, that's odd, you should be able to reach 4317.

    Just to reminder, 4317 is often used to GRPC Protocol and 4318 to HTTP Protocol.

    Sounds like your OTLP is not running, before that, tries to do a telnet on 4318 port too.

    If still not working, try to do some changes on OTLP file:

    

    yaml
    

    

    

    

    

    

    receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"  
      http:
        endpoint: "0.0.0.0:4318"

    After doing this changes, try a new telnet on 4317.

    If still not working, try to change your protocol to HTTP and do some new tests:

    yaml
    


    

    

    


    


    
exporters:
      prometheus:
        endpoint: "127.0.0.1:8889"
      otlp:
        endpoint: "http://127.0.0.1:4318"
        tls:
          insecure: true


    ------------------------------
    Gabriel Leme Solutions Architect
    Solutions Architect
    CXP Brasil
    São Paulo
    +55 (11)914734869
    ------------------------------

    Original Message


  • 7.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    You need to use the HTTP port, no the gRPC one. It's very easy to miss it, but the documentation says:

    End-to-End Monitoring currently supports OTLP/HTTP that uses Protobuf payloads encoded either in binary format or in JSON format. Regardless of the encoding the Protobuf schema of the messages is the same for OTLP/HTTP. Support for OTLP/gRPC is planned in future releases.

    Don't modify the agent.config file, use environment variables instead. More readable, less error prone. Here's the set of variables I pass to my Microservices Runtime (it should work the same in an Integration Server):

    SW_AGENT_COLLECTOR_ESTABLISH_CLOUD_COMMUNICATION=false
SW_AGENT_EXTERNAL_ESTABLISH_COMMUNICATION=true
SW_AGENT_OTEL_ENDPOINT=http://otel-collector:4318/v1/traces
SW_AGENT_OTEL_HEADERS=Content-Type#application/x-protobuf
SW_AGENT_EXT_SYSTEM_USER=
SW_AGENT_EXT_SYSTEM_PASSWORD=
SW_AGENT_EXTERNAL_SUPPORT_LONG_ID=false
SW_AGENT_EXTERNAL_API_ERROR_CODES=502,503,504
SW_AGENT_OTEL_RESOURCE_ATTRIBUTES=service.name#msr-sandbox, service.namespace#default
SW_AGENT_TRACE_TRANSFER_INTERVAL=1000

    You of course need to replace the content of SW_AGENT_OTEL_ENDPOINT with the endpoint of your Open Telemetry collector. Notice the 4318 port, which is the OTLP HTTP one (as opposed to 4317 which is for OTLP gRPC.)

    SW_AGENT_OTEL_HEADERS also is very important, it needs to specify the right content type: application/x-protobuf.

    SW_AGENT_OTEL_RESOURCE_ATTRIBUTES allows you to label your spans in order to make them easier to search and analyze, you can specify whichever key-value pairs you want there.

    It's better to specify a shorter interval in SW_AGENT_TRACE_TRANSFER_INTERVAL, in your configuration the interval is set to 30 minutes.

    Don't forget to activate auditing for the services you want to trace. It's better to configure it in the extended settings using these two properties:

    settings.watt.server.audit.service.include=<to-be-modified>
settings.watt.server.audit.service.exclude=wm.,com.,pub.

    Here's also an Open Telemetry collector config file (I am using the otel/opentelemetry-collector-contrib container image):

    receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
        cors:
          allowed_origins:
            - "http://*"
            - "https://*"

exporters:  
  file/traces:
    path: /etc/logs/otlpTraces.json

  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: 
      - file/traces
      - logging

    It activates verbose logging (debug log level) and it also forwards the spans to a log file specified in the exporter section. Since you're running your collector in Windows not in containers, you'll need to change this location. Later you can also add other exporters to send spans to Jaeger, Tempo, etc.



    ------------------------------
    Stephane Tailland
    ------------------------------

    Original Message


  • 8.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    Hi Stephane ,

    I followed the above steps and found the following logs in 'e2eagent.log.' However, they do not appear in:

    exporters:
        file/traces:
            path: C:/Colruyt/softwareag/logs/otlpTraces.json

    logs I can see:

    Did I overlook anything?

    Setting the environment variable as shown below should ensure the logs are written to:
    path: .../logs/otlpTraces.json

    SW_AGENT_OTEL_ENDPOINT=http://otel-collector:4318/v1/traces

     



    ------------------------------
    Swetha Bodagal
    ------------------------------

    Original Message


  • 9.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    You probably need to set change this variable like so (the otel-collector:4318 endpoint was valid in the context of my docker compose deployment, from what I understand you have directly installed the Open Telemetry collector on your Windows workstation.)

    SW_AGENT_OTEL_ENDPOINT=http://localhost:4318/v1/traces



    ------------------------------
    Stephane Tailland
    ------------------------------

    Original Message


  • 10.  RE: OLTP Collector Connection Issue

    Posted 27 days ago
    Edited by Swetha Bodagal 27 days ago

    Hi Stephane,

    I have already set it as shown below.

    SW_AGENT_OTEL_ENDPOINT=http://localhost:4318/v1/traces



    ------------------------------
    Swetha Bodagal
    ------------------------------

    Original Message


  • 11.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    Alright, let's see if your Open Telemetry is up and running. Use this curl command to do so.

    Note this is the Posix curl format, the Windows curl has some differences so you're probably need a bit of rewriting to make it work. You'll then find a span record in  C:/Colruyt/softwareag/logs/otlpTraces.json

    curl -X POST http://localhost:4318/v1/traces \
     -H "Content-Type: application/json" \
     -d '{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "example-service" }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "example-scope"
          },
          "spans": [
            {
              "traceId": "1a2b3c4d5e6f708091a2b3c4d5e6f708",
              "spanId": "1a2b3c4d5e6f7080",
              "name": "example-span",
              "kind": 1,
              "startTimeUnixNano": "1700000000000000000",
              "endTimeUnixNano": "1700000001000000000",
              "attributes": [
                {
                  "key": "http.method",
                  "value": { "stringValue": "GET" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}'

    This command sends a span in json format, not in protobuf, but that should not make a big difference. If the OTLP/HTTP port is up then it should be capable of processing both.



    ------------------------------
    Stephane Tailland
    ------------------------------

    Original Message


  • 12.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    Hi Stephane,

    I ran the curl command got a below response as {"partialSuccess":{}}. And still data not found in C:/Colruyt/softwareag/logs/otlpTraces.json.



    ------------------------------
    Swetha Bodagal
    ------------------------------

    Original Message


  • 13.  RE: OLTP Collector Connection Issue

    Posted 27 days ago

    partialSuccess is the response you should have. So the OTEL collector is up and running.

    So we have two issues:

    1. Your IS does not connect to the OTEL collector
    2. The Otel Collector does not output the spans as expected

    In the E2EM Agent logs you gave earlier, there are multiple "Error: null" errors. Maybe the IS tries to connect to the OTEL collector and gets an unexpected response.

    Can you try with Jaeger instead of your OTEL collector? https://www.jaegertracing.io



    ------------------------------
    Stephane Tailland
    ------------------------------

    Original Message


  • 14.  RE: OLTP Collector Connection Issue

    Posted 2 days ago

    Hi Stephane,

    Thank you!

    After installing Jaeger, I was able to view the traces in the Jaeger UI at http://localhost:16686.

    Thanks,

    Swetha



    ------------------------------
    Swetha Bodagal
    ------------------------------

    Original Message