HMC & CMC

HMC Logging and Auditing

By HARIGANESH MURALIDHARAN posted Sun June 07, 2020 09:11 AM

  

Ever wondered about operations being performed via the HMC or what problems are found or how you can be alerted about potential problems? HMC logs information on operations performed, provides alerts on problems and provides mechanism to capture logs to a remote system. This blog provides an overview of 

  1. What data is logged in HMC.
  2. How you can view the logs/events? 
  3. What different parameters/filters are available for viewing logs/events?
  4. How to configure Remote system logging.

What’s logged?

  1. Console Events:

As an admin, I want to track what operations are performed on the HMC or on the Managed system? who has logged in and at what time? Finding it difficult to track all this information? Don’t worry, Console Events can help. Here is what is logged as Console Events, 

  • Classic UI/CLI:   All operations except for list/view operations; Example: Login/Logout, Partition Create/Delete/Activate/Shutdown, LPM, Simplified RR.
  • Rest API: CLI operations invoked via POST/PUT/Delete Operations & Jobs.
  • Enhanced+ UI : Login/Logout & Classic UI Panels (With HMC V8 R8.6.0, Enhanced UI+ operations are logged as part of the task log. To learn more about this refer to http://ibm.co/2fQIHzs.)
  1. Serviceable Events from HMC and managed servers
You can easily track what operation were performed on the HMC and on the Managed system using console event. 

But you want to know what’s happening to Managed system? Is there any problem on the Managed system? Do I need to modify partition configuration for better performance? 

No worries; Serviceable events on HMC can help answer all these questions. You can see problems on the Managed System or the HMC or for alerts when defined thresholds are crossed as a serviceable event. You can view the problem, manage problem data, call home the event to your service provider, or repair the problem.

Viewing Console event logs: 

HMC provides so many features to make your job easier. But how can I view console events?  

You can view Console events via CLI, GUI and Rest API. 

           CLI:

             Just run “lssvcevents” commands. 

           Classic UI: 

             If you are not very comfortable with CLI, there is another way.

             You can view Console events via “View Management Console Events” task under HMC Management. 

           Enhanced+ UI:

             “Console Events Log” under Serviceability.

           REST API:

             Want to develop your own app/tool to check all this data? 

             Check out our REST API Job CLI Runner which can be used to issue the lssvceventscommand

              https://www.ibm.com/support/knowledgecenter/P8ESS/p8ehl/apis/CLIRunner_ManagementConsole.htm

lssvcevents:

Here are some of the commonly used lssvcevents command variants.  You can do a lot more than this. 

If you wish to list all the console events, run the following command,

     lssvcevents -t console 

If you want to see only the events that occurred today, run the following command, 

     lssvcevents -t console -d 1

Want to know how to see 3 days’ of data?  A simple change to the previous command will provide the information.

     lssvcevents -t console -d 3

How about event that occurred recently?

     lssvcevents -t console -i 1  

The above command will list events that occurred in the last minute.

GUI

If you are not comfortable with CLI or have difficulty remembering the CLI options, HMC provides an additional means to see console events; from the Classic UI Login you can launch “View Management Console Events” task under HMC Management or "Console Events Log" option under Serviceability menu on the left side of the Enhanced UI.  The picture below gives information about the date and time when specific events occurred.

You can list console events in with a time range. Just click on “view” in View Console events panel and enter the desired time range.

Console Events GUI

View Serviceable events: 

Wondering how to check serviceable events?  It's quite easy as HMC provides three ways to view Serviceable events: via CLI, GUI and REST API. 

          CLI:

           Using lssvcevents commands.

          Classic UI: 

           “Manage Serviceable Events” task under Service Management.

          Enhanced+ UI:

           “Serviceable Events Manager” under Serviceability

          REST API:

           /rest/api/sem/ServiceableEvent

Serviceable event from CLI:

How can I view all serviceable events?

     lssvcevents will fetch the information you requested. 

     lssvcevents -t hardware

If you wish to list all serviceable events from a managed system, run the following command

     lssvcevents -t hardware –m <managed system>

If you wish to list all open serviceable events, run the following command

     lssvcevents -t hardware --filter "status=open“

Can combining status and managed system get all open serviceable events from a managed system?

Yes; if you combine status and managed system, you will get events with specific status for that managed system.

     lssvcevents -t hardware –m <managed system> --filter "status=open“

If you wish to list only the problem numbers and status of all serviceable events for the system that occurred within the last 7 days, run the following command

     lssvcevents -t hardware -m <Managed system> -d 7 -F problem_num:status

Can this command be used to list open events on HMC appliance itself?

Yes, HMC provides that option too,

     lssvcevents -t hardware -F refcode,failing_mtms,last_time --filter "status=Open"  | grep `lshmc -v | grep *TM | sed 's/*TM //'`

GUI: 

 If you wish to see serviceable events from Classic UI, you can launch “Manage Serviceable Events” task under Service Management or "Serviceable Events Manager" under Serviceability menu on the left side of the Enhanced UI. The picture below gives information about problem number, Reference code, status, reported time, failing MTMS.

Manage Serviceable Events GUI

 

Wondering what is reference code? Just click on the reference code and you will get detailed information of the reference code. You can also use a search engine like Google for the reference code to get the same information.

 

SE


Like CLI is there any way to filter these serviceable events? 

Yes! You can specify selection criteria for serviceable event, specify the criteria when “Manage Serviceable Events” task is launched. You can specify filter like Serviceable event status, problem number, Reporting MTMS , Failing MTMS ….

 

Filter SE

 

Rocket-fast system for log processing (Rsyslog for HMC)

HMC provides very good logging mechanisms, but what if you have many servers and HMCs. Wouldn't it be good if you could see logs from different HMC on to single server? 

You can do that using rsyslog. Let's learn how to configure rsyslog

Rsyslog allows syslog messages from different HMC to be forwarded to a single or multiple remote logging servers. HMC introduced support for rsyslog in release HMC V7 R7.3.0. Starting with V8 R8.2.0, support for filter on rsyslog and enhanced CLI, to list currently configured syslog server settings, were added. 

Rsyslog

HMC CLI provides support to 

  1. Enable / Disable Remote logging
  2. Configure and modify remote logging filters
  3. Import certificates for TLS encryption
  4. List the current remote syslog configuration and filter for each rsyslog server.

HMC rsyslog supports following types of connection to use for forwarding syslog messages to remote rsyslog server.

  1. UDP - Unencrypted UDP
  2. TCP - Unencrypted TCP
  3. TLS - TLS encrypted TCP

Configuring rsyslog – UDP

You can add a remote UDP-based logging destination with filter: 

chhmc -c syslog -t udp -s add -h secure.ibm.com -f   <fully qualified file name>

                       or

chhmc -c syslog -t udp -s add -a 9.1.1.1 -- input  "filter_msg_contains_discard_strings=<strings tobe discarded with comma separation>”

  • Filter parameter (--input | –f) is optional.
  • Filter File entry should start with “filter_msg_contains_discard_strings=”

What is that filter? What does if do? How to specify it?

After configuring rsyslog, you see that one HMC is logging too much information which is not needed or less important and you don’t want to list that log in the remote syslog machine. Just identify the string in that log and provide that while configuring rsyslog or modify filter so that it will not be listed in the remote syslog machine. Example: If you do not want messages containing DHCP to be listed in the remote server then just mention “filter_msg_contains_discard_strings=DHCP” or create a file with entry “filter_msg_contains_discard_strings=DHCP” and provide the input explained above. 

 

You can remove a remote UDP-based logging destination using the command

chhmc -c syslog -t udp -s remove -h secure.ibm.com

          or

chhmc -c syslog -t udp -s remove -a 9.1.1.1

What about adding a filter to an existing filter configuration? A simple variant to the existing command will do. If you are specifying with a file, then modify the existing file to add or remove filter strings. Whatever is specified in file will be taken as filter configuration. 

chhmc -c syslog -t udp -s modify -h secure.ibm.com -f <fully qualified file name>

           or       

chhmc -c syslog -t udp -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings+=<strings>” 

If there is an option to add,  then is there an option to remove? 

Yes, by specifying “-=” in the input. 

chhmc -c syslog -t udp -s modify -h secure.ibm.com -f <fully qualified file name>

           or         

chhmc -c syslog -t udp -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings-=<strings>” 

 

Configuring rsyslog – TCP

Configuring TCP is also very simple; just replace udp with tcp in the above command and you are good to go.

To add a remote TCP-based logging destination with filter,

chhmc -c syslog -t tcp -s add -h secure.ibm.com -f <fully qualified file name>         

                        or

chhmc -c syslog -t tcp -s add -a 9.1.1.1 -- input 

"filter_msg_contains_discard_strings=<strings tobe discarded with comma separation>

  • Filter parameter (--input | –f) is optional.
  • Filter File entry should start with “filter_msg_contains_discard_strings=”

To remove a remote TCP-based logging destination, 

chhmc -c syslog -t tcp -s remove -h secure.ibm.com

          or

chhmc -c syslog -t tcp -s remove -a 9.1.1.1

To add a filter to the existing configuration,

chhmc -c syslog -t tcp -s modify -h secure.ibm.com -f <fully qualified file name>

                     or        

chhmc -c syslog -t tcp -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings += <strings>” 

To remove a filter from the existing configuration,

chhmc -c syslog -t tcp -s modify -h secure.ibm.com -f <fully qualified file name>

                    or      

chhmc -c syslog -t tcp -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings -= <strings>” 

 

Configuring rsyslog – TLS 

As in the previous command, to configure TLS replace tcp with tls. However you also need to do some simple prerequisite steps before configuring TLS based rsyslog. 

Prerequisite:

Import TLS certificates

  1. CA certificate:

            getfile –t rsyslogcacert –t l –f <fully qualified file name> 

                             or

            getfile –t rsyslogcacert –t s –f <fully qualified file name> -h <host name          or ip address of secure FTP server> -u <user id> --passwd <password>

  1. HMC certificate:

            getfile –t rsysloghmccert –t l –f <fully qualified file name> 

                             or

            getfile –t rsysloghmccert –t s –f <fully qualified file name> -h <host name      or ip address of secure FTP server> -u <user id> --passwd <password>

  1. Import HMC key:

            getfile –t rsysloghmckey –t l –f <fully qualified file name> 

                 or

            getfile –t rsysloghmckey –t s –f <fully qualified file name> -h <host name       or ip address of secure FTP server> -u <user id> --passwd <password>

 

To Enable/Disable TLS Encryption: 

After copying certificates to HMC, you can execute the following command which will enable TLS encryption,

chhmc –c syslog –t tls –s enable/disable

Configuring rsyslog – Encrypted TCP Logging (TLS)

With all the prerequisite steps done, you are good to go for tls configuration. 

Before proceeding to configuration steps, let's learn why tls?

Using TLS based rsyslog provides a secured way of logging and so we support the tls option which is encrypted tcp. We recommend that you always configure tls so that your system will be more secure. Here are the steps to configure TLS:

To add a remote TLS-based logging destination with filter,

          chhmc -c syslog -t tls -s add -h secure.ibm.com -f <fully qualified file name>          

                        or

           chhmc -c syslog -t tls -s add -a 9.1.1.1 -- input "filter_msg_contains_discard_strings=<strings tobe discarded with comma separation>”

  • Filter parameter (--input | –f) is optional.
  • Filter File entry should start with “filter_msg_contains_discard_strings=”

To remove a remote TLS-based logging destination,

            chhmc -c syslog -t tls -s remove -h secure.ibm.co

          or

            chhmc -c syslog -t tls -s remove -a 9.1.1.1

To add a filter to the existing configuration,

            chhmc -c syslog -t tls -s modify -h secure.ibm.com -f <fully qualified file name

                      or

            chhmc -c syslog -t tls -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings += <strings>” 

To remove a filter from the existing configuration,

            chhmc -c syslog -t tls -s modify -h secure.ibm.com -f <fully qualified file name>

                      or           

            chhmc -c syslog -t tls -s modify -h secure.ibm.com –input “filter_msg_contains_discard_strings -= <strings>” 

 

Before we can finish, one last question. How to list the existing filter configuration and rsyslogserver from HMC? 

To list existing rsyslog server, 

             lshmc -r 

To see existing filter configuration just issue following command, 

             lshmc --syslog 

Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions


#hmc
#powervmblog
0 comments
23 views

Permalink