On Thu, Oct 19, 2023 at 03:06:27PM +0000, Vinny G via IBM TechXchange Community wrote:
> Using "errnotify ODM to send message" via email is very bad.
I complete agree. When I work with customers, we may choose to
implement email errpt forwarding and I'm up front about the potential
to spam. I've caused thousands of messages myself.
One way to handle that is to use a /.forward file to forward root's
mail, so that if there is spamming occurring you can simply rename
/.forward and stop the emails from leaving the box. You can even do
this proactively.
Unless they are implementing a syslog monitoring solution, there are
few alternatives.
At this point I tend to refer to it as "poor man's alerting", because
it has obvious deficiencies. They can spend some money to get
something better.
It still has utility if you need something immediately. Other
solutions are preferred.
In the past I have used errnotify to forward errpt to syslog, and used
a centralized syslog server for all AIX systems. Typically that's a
Linux or BSD box running syslog-ng because it can aggregate incoming
syslogs into a host and date directory hierachy. Then I can report on
it with logmuncher, which is an old grep based tool I'm fluent
in. This is still trivial to setup compared to Splunk, ELK, Greylog or
other huge database backed webapps.
> I'm not a fan of Ansible but if I use Ansible stuff on VIO, I need
> at least Python and some other software.
May I ask what you are trying to accomplish via Ansible on VIO?
I'm well aware of Ansible's requirements and it's usage. I prefer
Puppet because it has a discrete agent, a sane configuration syntax,
and does configuration by documenting the final state.
I have a poor impression of Ansible overall, because I often see
Ansible playbooks that would fail a shell scripting class. Most items
could be more easily accomplished via shell script, with less text and
confusion.
It boggles my mind that customers which would prohibit their sysadmin
running a script as root if it came from a third party vendor will
blindly allow the download of Ansible playbooks for Github to run in
their environment.
When I deploy parallel VIOs in customer environments, I do a base
install and then customize everything via script so they match
exactly. That doesn't help as much for maintenance. However an
idempotent configuration script could potentially be run after each
major change.
> I also never liked SNMP on AIX, I don't think it provides enough
> information for the work.
Hostmib and AIXmib have great information in them. You can monitor
processes, filesystems, and more. AIXmib is preferred on storage,
because hostmib wraps negative on large filesystems. I've also found
AIXmib to be more performant on systems with many thousands of
processes.
The key is SNMPv1 is insecure, and SNMPv3 does authentication and
encryption. AIX's system daemons do it well.
Nagios works well with SNMP, and it's simple enough to have a wrapper
script around snmpwalk or find an snmp plugin to check these items.
Key items I can't get from SNMP include SEA status, LACP status, and
SAN paths. I do have to use local shell scripts for those. I do use
Nagios NRDP and a tiny 300 line Perl script to submit the results to
Nagios. It also makes it trivial to monitor any shell or scheduled job
completion on a single dashboard.
------------------------------------------------------------------
Russell Adams
Russell.Adams@AdamsSystems.nlPrincipal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 10/19/2023 11:06:00 AM
From: Vinny G
Subject: RE: YUM/SUDO and Ansible on VIO
Using "errnotify ODM to send message" via email is very bad. I had taken down my mail system when we had a fast-T and put millions of errors in errpt. I'm not a fan of Ansible but if I use Ansible stuff on VIO, I need at least Python and some other software. I also never liked SNMP on AIX, I don't think it provides enough information for the work. For now, I'm using a non-privileged account to do some basic monitoring. I think you provided a lot of good, information. I will review the links.
thank you,
Vinny
------------------------------
Vinny G
------------------------------
Original Message:
Sent: Thu October 19, 2023 09:43 AM
From: Russell Adams
Subject: YUM/SUDO and Ansible on VIO
On Thu, Oct 19, 2023 at 01:28:41PM +0000, Vinny G via IBM TechXchange Community wrote:
> I'm not actually looking to run ansible on VIO. The way I'm using
> Ansible. I been installing dnf and sudo on the AIX clients. Ansible
> is using sudo to elevate rights. I want to make sure other people
> are doing it.
That's quite a bit of software to install just to do monitoring or
elevate rights.
I couldn't ever justify using a piece of unsupported OSS on AIX to
control access to the critical root account. Sudo is not a good
answer, especially with the baggage of DNF.
I recommend doing root elevation via SSH with keys. Fully supported by
IBM with no additional software, and easy to script.
https://adamssystems.nl/posts/using-ssh-instead-of-su-and-sudo/
> I was using multibos on VIO clients as it was google to back out
> disk manipulation, when upgrading EMC powerpath.. It worked great
> but IBM support was like that isn't supported. but the IBM
> instructor had told me it was ok.
You could still use an alt_disk_copy. It's less fiddly than
multibos. An alt_disk_copy works on VIO servers and clients. It's a
great step for easy rollback of upgrades.
Powerpath breaks many things. I've had consistent failures with
maintaining rootvg with Powerpath. That has caused me to use to mksysb
restores when a simple LVM migration should have worked for several
customers. Good thing it's going away and being replaced by a
PowerPath ODM plugin for MPIO.
> So I want to confirm that other people are install DNF, SUDO and
> using ansible jobs on the VIO servers. My first thought is for
> monitoring. Example Monitoring errpt, and checking file system
> space. I noticed I had to increase /var on new VIO servers.
Nope. I'd never add all of that to VIO. I don't want to have to
increase my patch schedules to satisfy cybersecurity over the DNF
updates that aren't shipped with AIX. None of that extra software
helps VIO do it's job.
For monitoring errpt, I use errnotify ODM to send messages in realtime
to syslog and via email.
https://adamssystems.nl/posts/simple-error-reporting/
For standardization and configuration, I either use scripts or custom
made LPP patches deployed by NIM. All fully supported, no additional
software.
Monitoring filesystems can be done with RMC, fully supported with no
extra software:
https://www.ibm.com/support/pages/monitoring-specific-filesystem-usage-using-resource-monitoring-control
Finally if you need to monitor more flexibly, I recommend using Nagios
and SNMPv3. The AIX snmpd, hostmibd, and aixmibd provide excellent
information in a secure fashion. I have customers where we monitor
processes and disks via SNMPv3 and it works fine with aixmibd.
------------------------------------------------------------------
Russell Adams Russell.Adams@AdamsSystems.nl
Principal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 10/19/2023 9:29:00 AM
From: Vinny G
Subject: RE: YUM/SUDO and Ansible on VIO
I'm not actually looking to run ansible on VIO. The way I'm using Ansible. I been installing dnf and sudo on the AIX clients. Ansible is using sudo to elevate rights. I want to make sure other people are doing it.
I was using multibos on VIO clients as it was google to back out disk manipulation, when upgrading EMC powerpath.. It worked great but IBM support was like that isn't supported. but the IBM instructor had told me it was ok.
So I want to confirm that other people are install DNF, SUDO and using ansible jobs on the VIO servers. My first thought is for monitoring. Example Monitoring errpt, and checking file system space. I noticed I had to increase /var on new VIO servers.
thank you.
Vinny
------------------------------
Vinny G
Original Message:
Sent: Thu October 19, 2023 01:47 AM
From: Niklas V.
Subject: YUM/SUDO and Ansible on VIO
Hi Vinny,
not sure what you are looking for, but can share some experiences. We use Ansible on all our VIOs (>200) and IBM supports this. Unfortunately this is not completely documented but if you look at the Ansible collection power_vios, there is no way around it in my view to fully authorize the Ansible user via sudo.
I tried to describe the steps in an issue: Update documentation with an initial configuration for an ansible user · Issue #31 · IBM/ansible-power-vios · GitHub
Once the preparations are complete, Ansible is very easy to use.
# Install the latest available httpd version- name: Install webserver ansible.builtin.dnf: name: - httpd state: latest# Install a specific httpd version- name: Install webserver version 2.4.54-3 ansible.builtin.dnf: name: - httpd-2.4.54-3.ppc state: latest
I am not sure if a webserver running on a VIOS is a good idea, but there is nothing that should prevent you.
------------------------------
Niklas
System Engineer UNIX and Linux on Power
Original Message:
Sent: Mon October 16, 2023 11:11 AM
From: Vinny G
Subject: YUM/SUDO and Ansible on VIO
Hi
I'm not a fan of Ansible on AIX, I would much rather do things via Korn scripts. But this old dog is trying some new tricks. I have used Ansible to call a Korn script to set up multibos, update httpd with dnf, and call a script to do a dnf update/clean up. I have ansible setup to run as non-root and escalated to install packages. It is convenient to use Ansible to push stuff out to many systems but I could have used a for loop and ssh keys to do the same.
My Ansible controller is actually my NIM server. I do not want to use my VIOS as an Ansible controller.
It sounds like it's ok to install DNF,sudo and python on the VIO servers.
thanks,
Vinny
------------------------------
Vinny G
Original Message:
Sent: Mon October 16, 2023 10:52 AM
From: Andrey Klyachkin
Subject: YUM/SUDO and Ansible on VIO
Hi Vinny,
if you want to manage VIOS using Ansible, you don't need all these funny things on VIOS. BUT they (dnf and python) make your life much more pleasant. You don't need Ansible and sudo if you want to manage VIOS with Ansible. Yum is deprecated on AIX (and on VIO), use DNF.
if you want to use VIOS as Ansible controller, then I would discourage you to do so. Use AIX or any Linux, it works better.
------------------------------
Andrey Klyachkin
https://www.power-devops.com
Original Message:
Sent: Fri October 13, 2023 03:46 PM
From: Vinny G
Subject: YUM/SUDO and Ansible on VIO
What is IBM stance on installing yum,sudo, and Ansible on VIO?
------------------------------
Vinny G
------------------------------