AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
#Power
 View Only

  • 1.  nextract plus to telegraf

    Posted Mon February 10, 2025 03:48 AM

    Dear community, dear Nigel.

    I'm running nimon to send data via telegraf to Prometheus and further into Grafana dashboads.
    Now, I would like to do the same with nextract. Has anyone experience or tried to set up a similar environment?

    I tried to configure nextract to send data to telegraf, but never got it working.

    Any help or hint would be appreciated. Thanks in advance.

    kind regards,
    Joerg



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------


  • 2.  RE: nextract plus to telegraf

    Posted Tue February 11, 2025 04:01 AM

    Hi Joerg,

    can you please elaborate on "never got it working"? ;-)

    • Could you get the data from the HMC?
    • Does nextract send the data to Telegraf using InfluxDB protocol?
    • Can you see the data in Telegraf using Prometheus interface or curl?



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------

    Original Message


  • 3.  RE: nextract plus to telegraf

    Posted Tue February 11, 2025 07:38 AM
    Edited by Joerg Kauke Tue February 11, 2025 07:41 AM

    Hello Andrey,

    thanks for getting into it.
    First, I run nextract plus on our NIM server, try to send the InfluxDB formated data to the telegraf container running on a SLES 15.
    Nextract config json looks like this:

    {
"hmc_hostname": "svrifahmc-02",
"hmc_user": "nextractplus",
"hmc_password": "*************",
"output_nchart": 0,
"output_json": 0,
"output_csv": 0,
"comment1": "Valid setting for output_influx = 0 (off), 1 or 2",
"output_influx": 1,
"comment2": "InfluxDB 1.x options",
"ihost": "10.1.224.31",
"iport": 8888,
"iuser": "",
"ipassword": "",
"idbname": ""
}

    and this is the error it got:

    /usr/local/scripts/nextractplus_v41/nextract_plus.py opening config file:./svrifahmc02_2_telegraf.json
-> nextract_plus version 41 saving JSON=0 Influx=1
-> Logging on to svrifahmc-02 as hmc_usernextractplus
-> Get Preferences
-> Parse Preferences
-> ALL servers:
-> Server name=svrifa922-00_SN780C540 agg=true  longterm=true  - OK
-> Server name=svrifa922-01_SN780C550 agg=true  longterm=true  - OK
-> Servers with Perf Stats

--> Server=1 Name=svrifa922-00_SN780C540 - Requesting the data ...
---> Received 12 file(s) in 0.46 seconds
----> Server Name=svrifa922-00_SN780C540 MTM + Serial Number=9223-22H*780C540
----> Server Date=2025-02-11 start=11:28:00 end=13:27:00
----> Server DataType=Processed Interval=30 seconds
----> LPAR=svrseng1-0
----> LPAR=svrsdb3-0
Adding 26397 records to InfluxDB for Server=svrifa922-00_SN780C540
Traceback (most recent call last):
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/freeware/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
  File "/opt/freeware/lib/python3.9/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/opt/freeware/lib64/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/scripts/nextractplus_v41/nextract_plus.py", line 610, in <module>
    client.write_points(entry)
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 614, in write_points
    return self._write_points(points=points,
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 692, in _write_points
    self.write(
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 424, in write
    self.request(
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 343, in request
    response = self._session.request(
  File "/opt/freeware/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/freeware/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/freeware/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

    And this is my telegraf config, like Nigel suggested for nimin.

    [[outputs.prometheus_client]]
    listen = ":8099"
    metric_version = 2
    path = "/metrics"
    expiration_interval = "120s"
    string_as_label = false

[[inputs.socket_listener]]
    service_address = "tcp://:8888"
    data_format = "influx"
    read_buffer_size = "256KiB"
    read_timeout = "2s"

    So, I'm not familiar with databases nor with telegraf in deep.
    I think it comes from the missing 'iuser / ipassword'  or 'idbname'.
    But I couldn't find any description on, how to add a database name to telegraf.
    All I found was a suggestion to save the output to a file and then read the file to a local installed telegraf instance.

    kindly regards,
    Joerg



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------



  • 4.  RE: nextract plus to telegraf

    Posted Tue February 11, 2025 09:15 AM

    Hi Jörg,

    thank you for the information! 

    Create an additional InfluxDB HTTP Listener in your telegraf.conf file like:

    [[inputs.influxdb_listener]]
service_address = ":8886"
read_timeout = "2s"

    Restart telegraf

    Reconfigure nextract to point to the listener

    Start nextract, it should send the data correctly

    Check with curl, that the data came - curl http://10.1.224.31:8099/metrics



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------

    Original Message


  • 5.  RE: nextract plus to telegraf

    Posted Thu February 13, 2025 03:50 AM

    Hello Andrey,

    once more, thanks for helping me out. After I realized I also have to put the port into the docker-compose.yml it worked :-).
    Many thanks for your support. Next time you're in Zurich, I'll pay you a beer...

    kind regards



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------

    Original Message


  • 6.  RE: nextract plus to telegraf

    Posted Tue February 25, 2025 04:59 AM

    Dear community,


    Unfortunately I have to come back to you. I'm encountering two issues with my monitoring setup and would greatly appreciate your expertise and suggestions.

    Issue 1: Limited Data Availability in Prometheus

    nextractplus script is scheduled to run hourly via cron:

    <response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

    12 * * * * /usr/local/scripts/nextractplus_v41/nextract_plus.py

    <response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

    While data is now successfully reaching Prometheus, I've noticed that for all metrics, Prometheus only seems to retain data for the last 90 seconds. For example, if I run nextractplus at 9:12 AM, I only see data in Prometheus for the time range of approximately 9:12:41 to 9:14:11.

    Is this limited data retention the expected behavior when using the prometheus_client output in Telegraf? This is quite different from what I observe in Grafana, where the historical data presentation is not limited to this short timeframe (although the data itself seems to be missing outside of this 90-second window).

    Could anyone clarify if this 90-second data retention in Prometheus is normal in this setup, or if I might be missing something in my configuration?

    Issue 2: "HTTP Request Body Too Large" Error in Production Environment

    When I attempted to collect data from our production environment, which is significantly larger than my test setup, nextractplus failed with the following error:

    <response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

    Adding 133354 records to InfluxDB for Server=svrifa1080-00_SN7870CE8
Traceback (most recent call last):
  File "/usr/local/scripts/nextractplus_v41/./nextract_plus.py", line 609, in <module>
    client.write_points(entry)
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 614, in write_points
    return self._write_points(points=points,
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 692, in _write_points
    self.write(
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 424, in write
    self.request(
  File "/opt/freeware/lib/python3.9/site-packages/influxdb/client.py", line 389, in request
    raise InfluxDBClientError(err_msg, response.status_code)
influxdb.exceptions.InfluxDBClientError: 413: {"error":"http: request body too large"}

    <response-element class="" ng-version="0.0.0-PLACEHOLDER"></response-element>

    This error message influxdb.exceptions.InfluxDBClientError: 413: {"error":"http: request body too large"} suggests that the amount of data nextractplus is trying to send at once is exceeding some limit.

    Does anyone have experience with this "request body too large" error when sending data from nextractplus to Telegraf/InfluxDB? Are there any recommended strategies to handle larger datasets, such as chunking the data or adjusting configuration parameters, either in nextractplus or Telegraf?

    Any insights or suggestions on either of these issues would be highly appreciated!

    Thank you very much for your time and support.



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------

    Original Message


  • 7.  RE: nextract plus to telegraf

    Posted Tue February 25, 2025 08:49 AM

    I could solve the second problem, 'http: request body too large', by adding a chunk_size variable to the python script.
    @Nigel Griffiths: maybe you want to implement it in you're code.
    starting at line 673

    ...
...
...
            # SET CHUNK SIZE to 5000
            chunk_size = 5000

            if output_influx == 1:
                for i in range(0, len(entry), chunk_size):
                    client.write_points(entry[i:i + chunk_size])

            if output_influx == 2:
                for i in range(0, len(entry), chunk_size):
                    write_api.write(ibucket, iorg, entry[i:i + chunk_size])

            if output_json == 1:
                for item in entry:
                    JSONfile.write(str(item) + "\n")
            entry=[] # empty the list

    # PUSH TO INFLUXDB FOR EACH SERVER
    if len(entry) > 0:

        if log:
            for line in entry:
                logfile.write(str(line))
                logfile.write("\n")

        if output_influx ==1 or output_influx == 2:
            for i in range(0, len(entry), chunk_size):
                client.write_points(entry[i:i + chunk_size])
        print("Added %d records to InfluxDB for Server=%s"%(len(entry),servername))
        measures = measures + len(entry)
        if output_json == 1:
            for item in entry:
                JSONfile.write(str(item) + "\n")

if output_json == 1:
    JSONfile.close()
if log:
    logfile.close()

print("Logging off the HMC - found %d measures"%(measures))
hmc.logoff()

     

    kind regards



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------

    Original Message


  • 8.  RE: nextract plus to telegraf

    Posted Fri February 28, 2025 10:36 AM

    Issue 2:

    Sure, I can add that to the next release of nextract but I assume you have already worked the above code out and added it to your copy.

    Issure 1:

    Sorry, I have no idea.

    Perhaps there is a Prometheous forum that might take and answer a question.



    ------------------------------
    Nigel Griffiths - IBM retired
    London, UK
    @mr_nmon
    ------------------------------

    Original Message


  • 9.  RE: nextract plus to telegraf

    Posted Thu March 06, 2025 05:21 AM

    Hello all,

    if someone is interested, I found the source of issue 1: Limited Data Availability in Prometheus
    It was just the output stream config of telegraf:

    [[outputs.prometheus_client]]
    listen = ":8099"
    metric_version = 2
    path = "/metrics"
    expiration_interval = "120s" << problem source
    string_as_label = false

    After I raised the expiration_interval to '1h', because 'nextractplus' runs just once in an hour, it worked.
    So it was just a lack of knowledge on my site.

    kind regards



    ------------------------------
    Joerg Kauke
    Unix Administrator
    COOP Switzerland
    ------------------------------

    Original Message


Related Content