for some reason its still showing 0/s on diskio and i also cannot get rid of /proc filesystem, which is always shown as 100%

Yes, 8.17.1-4

Thank you @Aditya Kamath . Could you confirm the latest version please. I can see metricbeat-8.17.1-4.ppc.

@Felipe Villagran @Sirak Shiferaw

The newer update is available in Toolbox with the fix and disk statistics [how many bytes read and written]. Please check and let us know.

@Felipe Villagran @Sirak Shiferaw

Hi,

Thank you very much for your patience. There was a bug indeed leading to incorrect user space time spent. If we see the code here https://github.com/elastic/elastic-agent-system-metrics/blob/6ebac768d46c4d1d51f200977a8f9e7c09936320/metric/cpu/metrics_aix.go#L85

The tick2msec() is not used for user space time spent. So this caused the problem. In AIX using perfstat we need to convert ticks to milliseconds always. Otherwise it will go wrong.

If we see the outputs now in stress conditions, we get

 03:22:14  0       96       4       0       0    0.25
          1       95       5       0       0    0.25
          2       94       6       0       0    0.25
          3       95       5       0       0    0.25
          4       92       8       0       0    0.25
          5       93       7       0       0    0.25
          6       90      10       0       0    0.25
          7       96       4       0       0    0.25
          -       94       6       0       0    2.00

"system": {
    "cpu": {
      "iowait": {
        "norm": {
          "pct": 0.0013
        },
        "pct": 0.0105
      },
      "cores": 8,
      "total": {
        "norm": {
          "pct": 0.9896
        },
        "pct": 7.917
      },
      "user": {
        "pct": 7.1704,
        "norm": {
          "pct": 0.8963
        }
      },
      "system": {
        "pct": 0.7465,
        "norm": {
          "pct": 0.0933
        }
      },
      "idle": {
        "pct": 0.0725,
        "norm": {
          "pct": 0.0091
        }
      }
    }
  },

There will be a little bit of difference due to sampling timing and rounding of the beats and SAR commands trying to access the data in the kernel. We will release a version with the fixes and disk-I/O statistics as soon as possible, by this week or next one at max.

@Felipe Villagran

Thank you for your patience; we have found a workload to reproduce the issue. 
Will check what happened when in stressful conditions. I can see user and system usage mismatched in metricbeat and vmstat.

Will get back about the CPU issue. We will release a 8.17.1-4 this week with diskio statistics. 

Hi @Aditya Kamath, 

yes > /dev/null &
sar 1 5

System configuration: lcpu=8 ent=0.30 mode=Uncapped

11:44:20    %usr    %sys    %wio   %idle   physc   %entc
11:44:21      20      35       0      45    1.00   332.5
11:44:22      20      36       0      44    1.00   332.2
11:44:23      21      35       0      44    1.00   333.1
11:44:24      22      36       0      42    1.00   332.6
11:44:25      22      37       0      41    1.00   333.5

Average       21      36       0      43    1.00   332.7

"system": {
      "cpu": {
        "cores": 8,
        "user": {
          "norm": {
            "pct": 0.0077
          },
          "pct": 0.0618
        },
        "system": {
          "norm": {
            "pct": 0.2008
          },
          "pct": 1.6061
        },
        "iowait": {
          "norm": {
            "pct": 0.0001
          },
          "pct": 0.0011
        },
        "total": {
          "norm": {
            "pct": 0.2085
          },
          "pct": 1.6679
        },
        "idle": {
          "norm": {
            "pct": 0.7914
          },
          "pct": 6.331
        }
      }
    },

@Sirak Shiferaw @Felipe Villagran

Hi,

Regarding disk I/O statistics, we have enabled the number of bytes read and written per disk as of now. See outputs below, and we will release a version sometime next week. 
Please see the beats output and iostat -a output below. Unfortunately, in AIX we do not have read count and write count in the persfstat_disk_t data structure like Linux. We are exploring it with the tools team. In beats, data is expected in bytes, and in iostat, we get it in KB. Let us know if this is okay for you all for now.

"system": {
    "diskio": {
      "read": {
        "count": 0,
        "time": 0,
        "bytes": 70463971328
      },
      "write": {
        "bytes": 75301294080,
        "count": 0,
        "time": 0
      },
      "io": {
        "time": 2286846
      },
      "name": "hdisk2"
    }
  },
  "ecs": {
    "version": "8.0.0"
  }
}

# iostat -a

System configuration: lcpu=32 drives=5 ent=2.00 paths=4 vdisks=1 tapes=0

tty:      tin         tout    avg-cpu: % user % sys % idle % iowait physc % entc
          0.0         14.2                0.2   0.2   99.5      0.0   0.0    0.8

Vadapter:                  Kbps      tps     bkread    bkwrtn partition-id
vscsi0                      0.0      5.0        0.9       4.1            1

Disks:              % tm_act     Kbps      tps    Kb_read   Kb_wrtn
hdisk2                 0.1       6.6       0.5   68812472  73536420
hdisk0                 0.6      24.1       3.1   42310160  474931712
hdisk3                 0.5      12.5       1.0   14957340  254527116
hdisk1                 0.1       7.3       0.4   48513104  107552348

So I was analyzing the CPU statistics. We are unfortunately not able to reproduce the issue. 

If we check the vmstat output and what Beats is giving, they are close enough. I also checked the code and we are returning what we get from the perfstat APIs. 

Elastic raw pct fields are aggregate across cores, so they look large but are correct.

Is there any other way we can reproduce the issue that you have in your mind? We are trying our best to get the issue resolved for you but have not succeeded in reproducing it so far. 

# vmstat 1 30

System configuration: lcpu=32 mem=16384MB ent=2.00

kthr    memory              page              faults              cpu          
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 3  0 1295275 546252   0   0   0   0    0   0  72  569 259  0  1 99  0  0.03   1.6
 3  0 1295275 546250   0   0   0   0    0   0  55  362 331  0  0 99  0  0.03   1.4
 3  0 1295275 546250   0   0   0   0    0   0  63  142 251  0  0 99  0  0.02   1.2
 3  0 1295275 546250   0   0   0   0    0   0  61  617 267  0  1 99  0  0.03   1.4

"system": {
    "cpu": {
      "cores": 32,
      "total": {
        "pct": 0.9664,
        "norm": {
          "pct": 0.0302
        }
      },
      "user": {
        "pct": 0.0002,
        "norm": {
          "pct": 0
        }
      },
      "system": {
        "pct": 0.9662,
        "norm": {
          "pct": 0.0302
        }
      },
      "idle": {
        "pct": 31.0336,
        "norm": {
          "pct": 0.9698
        }
      },
      "iowait": {
        "norm": {
          "pct": 0
        },
        "pct": 0
      }
    }
  },

Thanks, @Felipe Villagran. I am looking into it.

Hi @Aditya Kamath , metricbeat is not capturing or calculate the right values for CPU utilization. Thank you very much for your help.

@Sirak Shiferaw

Thanks for confirming. We are currently developing diskio features as requested and also analyzing the CPU issue. Will get back in 2 to 3 days.

Thank you for your feedback so far. 

Disk usage is fixed on 8.17-3 only problem was /proc was not excluded hence showing 100% on Disk Usage Max

on the lpar. use nmon and enter then press p to see the overall CPU usage,

or collective default monitoring topas command.


i am not sure how i can help you, or you can mail me directly and see the issue in real server

@Sirak Shiferaw

Regarding disk I/O and throughput. We can develop that and release a version 4 (8.17.1-4)for you. Give me a couple of days. 

The next thing is about 2. Can you give me more information on what is happening with CPU information and how I can reproduce the issue.

Regarding 1: The hostname is sent as is and is not in my control. Let us wait for someone's response in the community. 

@Sirak Shiferaw

Disk usage: we recently resolved that bug in 8.17.1-3.

Kindly let me know what your rpm -qa | grep metricbeat output is. If it is not the same, then kindly update. Otherwise, I need to look into this. 

Atleast in our testing the numbers came correct. Please see below. Let me know if the filesystem numbers in the release mismatch.

# df -H

Filesystem                                        Size  Used Avail Use% Mounted on

/dev/hd4                                           30G   22G  8.7G  71% /

/dev/hd2                                           11G   10G  430M  96% /usr

/dev/hd9var                                       2.1G  1.6G  436M  79% /var

/dev/hd3                                          3.8G  209M  3.6G   6% /tmp

/dev/hd1                                           26G   24G  1.9G  93% /home

/dev/hd11admin                                    135M  390k  134M   1% /admin

/ahafs                                               -     -     0    - /aha

/dev/hd10opt                                       23G   22G  1.4G  95% /opt

/dev/livedump                                     269M  377k  269M   1% /var/adm/ras/livedump

9.18.2.83:/cst-storage/awx-exp/aix_multi_updater  322G  151G  172G  47% /tmp/patch

"system": {

    "filesystem": {

      "used": {

        "pct": 0.0556,

        "bytes": 208949248

      },

      "mount_point": "/tmp",

      "device_name": "/dev/hd3",

      "type": "jfs2",

      "total": 3758096384,

      "free": 3549147136,

      "available": 3549147136

    }

  },

"system": {

    "filesystem": {

      "used": {

        "bytes": 21366697984,

        "pct": 0.942

      },

      "mount_point": "/opt",

      "device_name": "/dev/hd10opt",

      "type": "jfs2",

      "total": 22682796032,

      "free": 1316098048,

      "available": 1316098048

    }

  },

ok thank you Aditya, i actually have a bunch

1. hostname is small letter, not sure if i should raise this here or apm-java-agent, hence raised it on both https://github.com/elastic/apm-agent-java/issues/4347
2. CPU information is not even close to reality nor topas (lpar) or nmon (accross pool usage)

on elastic metricbeat its alreays < 10%

3. Disk usage is not showing correctly every filesystem is 16%

there is no DiskIOPS and Disk Throughput metrics

i will be happy to assist with additional information if requried

regards

Hi @Sirak Shiferaw

You can contribute or tell us what the issue is. We will support. 

We do have plans to make beats as supported in the community sometime this year. Until then, kindly use this post to raise issues or contribute, and we are happy to help/resolve issues in beats.

Dears, the metric collection is wrong, where do i contribute ? directly on elastic page or we have a different repository for aix porting ?

i just converted everything into a service and i don't have to worry about the requested parameter or change any os config, thank you all for the comment

mkssys \
  -s metricbeat \
  -p /opt/freeware/bin/metricbeat \
  -a "run --path.config=/opt/freeware/etc/metricbeat --path.data=/opt/freeware/var/lib/metricbeat --path.logs=/opt/freeware/var/log/metricbeat" \
  -u 0 \
  -S \
  -n 15 \
  -f 9 \
  -R \
  -G elastic

and start as a backgroud service with

lssrc -s metricbeat

startsrc -s metricbeat

@Sirak Shiferaw @Felipe Villagran

Kindly export PATH=/opt/freeware/bin:$PATH and then try. You may want to store it in .~/bashrc profile.

@Sirak Shiferaw Try with this command. It should work.

/opt/freeware/bin/metricbeat -e -c /opt/freeware/etc/metricbeat/metricbeat.yml

for some reason i need to run full command otherwise its not loading active module

./metricbeat run \
  --path.config=/opt/freeware/etc/metricbeat \
  --path.data=/opt/freeware/var/lib/metricbeat \
  --path.logs=/opt/freeware/var/log/metricbeat \
  -e

@Felipe Villagran
Update is available.

@Aditya Kamath Thank you very much for the update!

@Felipe Villagran
Wanted to inform you. 
I have given beats 8.17.1-3 with the bug fixes and swap metrics addition to upload, which will be done by this Friday (5/12/2025) or the coming Monday (8/12/2025), after which do run dnf update. 

Thanks.@Felipe Villagran

@Felipe Villagran

The numbers of the script match the beats output. So we are okay. 

Thanks for the same. Expect the fix and SWAP metric in 8.17. 1 release 3 in coming weeks. [22 Nov - 30 Nov]

@Aditya Kamath Filesystem output looks good, for SWAP metrics this should be the output. 

#!/bin/ksh
swap_info=$(lsps -s | tail -1)
total=$(echo $swap_info | awk '{print $1}' | sed 's/MB//')
used_pct=$(echo $swap_info | awk '{print $2}' | sed 's/%//')

# Convert to bytes
total_bytes=$((total * 1024 * 1024))
used_bytes=$((total_bytes * used_pct / 100))
free_bytes=$((total_bytes - used_bytes))

echo "{\"swap\": {\"total\": $total_bytes, \"free\": $free_bytes, \"used\": {\"bytes\": $used_bytes, \"pct\": $(echo "scale=3; $used_pct/100" | bc)}}}"

@Felipe Villagran

Thank you very much for reporting this. Yes, we missed this in our test buckets. 

I have quickly corrected the same. As a priority, we will release 8.17.1-3 next week. Please see the output named as corrected output below.

As a token of gratitude, we have also implemented the SWAP metrics on priority within beats to be released along the same version. Please see how the JSON will look in AIX in the output named Swap Metrics.


We want one confirmation so the requirement is clear. By swap metrics are you referring to the lsps -a or lsps -s output in AIX? If that is the one, then we have implemented the same and expect it next week. Else let us know.

Let me know. Once again, thank you for reporting the issue and using in beats in AIX. 
========= Swap Metrics ==========
"system": {
    "memory": {
      "free": 6677639168,
      "actual": {
        "free": 33434894336,
        "used": {
          "pct": 0.2215,
          "bytes": 9514778624
        }
      },
      "swap": {
        "used": {
          "bytes": 94457856,
          "pct": 0.001
        },
        "free": 93857951744,
        "total": 93952409600
      },
========== Corrected output ==================


"system": {
    "filesystem": {
      "mount_point": "/opt",
      "device_name": "/dev/hd10opt",
      "type": "jfs2",
      "total": 22682796032,
      "free": 1201262592,
      "available": 1201262592,
      "used": {
        "pct": 0.947,
        "bytes": 21481533440
      }

 "system": {
    "filesystem": {
      "mount_point": "/home",
      "device_name": "/dev/hd1",
      "type": "jfs2",
      "total": 25098715136,
      "free": 1937272832,
      "available": 1937272832,
      "used": {
        "bytes": 23161442304,
        "pct": 0.9228
      }
    }
  },


# df -H
Filesystem                                        Size  Used Avail Use% Mounted on
/dev/hd4                                           30G   20G   11G  65% /
/dev/hd2                                           11G   10G  430M  96% /usr
/dev/hd9var                                       2.1G  1.2G  880M  57% /var
/dev/hd3                                          3.8G  195M  3.6G   6% /tmp
/dev/hd1                                           26G   24G  2.0G  93% /home
/dev/hd11admin                                    135M  390k  134M   1% /admin
/ahafs                                               -     -     0    - /aha
/dev/hd10opt                                       23G   22G  1.3G  95% /opt
/dev/livedump                                     269M  377k  269M   1% /var/adm/ras/livedump
9.18.2.83:/cst-storage/awx-exp/aix_multi_updater  322G  149G  174G  47% /tmp/patch

Hi @Aditya Kamath, 

Now is showing same data for All the filesystem something was wrong with this new version. Please your help to fix this and add Swap metrics. We appreciate your support.

Disk/Filesystem    % Disk Used    Free Space    Total Space
/tmp    78.50%    275.7MB    1.3GB
/usr    78.50%    275.7MB    1.3GB
/var    78.50%    275.7MB    1.3GB

Hi @Felipe Villagran,

Yes, the fix is uploaded in toolbox. Kindly run dnf update metricbeat auditbeat filebeat. 

Hi @Aditya Kamath , Do you have news about new releases? Please your comments. Thank you.

@Felipe Villagran,

Yes, it is possible to remove unsupported modules. I have removed the garble_macho_executable file from the release 2 RPM releasing this week. Thanks.

We will need some time to develop SWAP metrics. We will plan this and get back. 

Hi @Aditya Kamath,

Thank you for your quick response.

We are currently in the testing phase and have noticed that the SWAP metrics are not being reported—only RAM metrics are visible, and SWAP data appears to be missing.

On another note, regarding Auditbeat, we had to remove the file_integrity module due to a potential threat detection. It seems this module is not compatible with AIX. Is it possible to exclude or remove unsupported modules from the AIX version of Auditbeat?

Details:

• Path: /opt/freeware/share/auditbeat/module/file_integrity/testdata/garble_macho_executable
• Threat Name: RDN/Generic.osx

Thank you in advance for your support.

@Felipe Villagran

We have made corrections in the metric beat code.

In the updated one, the mount point and device name will be shown correctly as below. Thank you for reaching out. Expect the fixed RPMS next week. 

@Felipe Villagran

Thank you for reaching out. We are analyzing. We will make it consistent in AIX and fix it in the metric beat code if that is the expected behavior. Will get back soon.  

Hi everyone,

I'm using Metricbeat on AIX servers and noticed an unexpected behavior in the system.filesystem module.

When running df -h on AIX, the output correctly shows the logical volume and its mount point, for example:

Filesystem         Mounted on
/dev/defaultlv     /opt/default

However, the data sent by Metricbeat to Elasticsearch shows the fields inverted:

• system.filesystem.device_name: /opt/default
• system.filesystem.mount_point: /dev/defaultlv

This is inconsistent with how Metricbeat behaves on Linux, where:

• device_name refers to the device (e.g., /dev/sda1)
• mount_point refers to the mount path (e.g., /)

❓ Question

Has anyone else encountered this issue on AIX?
Is there a known workaround or fix for this behavior?
Could this be a bug in the AIX module of Metricbeat?

Any help or guidance would be greatly appreciated.

Thanks,
Felipe

@Todd Winkler

You can ask your questions here. Kindly go ahead.

Who can help me with some questions on this software?