Dear Community:
We have been running nimon for over two years, collecting statistics from about 400 AIX- and Linux-based systems and retaining 105 weeks of data. The InfluxDB database & Grafana are installed on a RHEL 7.9 (3.10.0-1160.135.1.el7.x86_64 #1 SMP Tue May 13 01:53:34 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux kernel) host which was updated on June 24.
Today, when I opened Grafana, none of the dashboards were populated with data. I found the influxDB daemon was not running. I started it and checked the dashboards but still no data was displayed, so I superstitiously restarted the Grafana service. After several minutes, new data was being plotted on the dashboards but the previous 105 weeks of statistics were missing. Our polling interval is 5 minutes so that explains the delay in new statistics being graphed.
Versions:
nimon: a mix of 7.1 & 8.3
influxDB: 1.8.3
Grafana: 11.5.1
We keep track of how much NAS storage is consumed by the influxDB database & it doesn't seem like the data actually was lost:
date Filesystem 1048576-blocks Used Available Capacity Mounted on
2025-06-09 nasnfsco:/njnimon 608996 529229 79767 87% /nimon
2025-06-16 nasnfsco:/njnimon 608996 529863 79134 88% /nimon
2025-06-23 nasnfsco:/njnimon 608996 530635 78361 88% /nimon
2025-06-30 nasnfsco:/njnimon 608996 531816 77180 88% /nimon
2025-07-02 nasnfsco:/njnimon 608996 527002 81995 87% /nimon
Retention policy:
> show retention policies on njmon
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 17640h0m0s 168h0m0s 1 true
After initial setup, nimon has "just worked" all these years so I'm not sure how to begin troubleshooting. Any suggestions are welcome.
Thanks & Best Regards,
William