Before we dive into this subject a little bit about how IBM AIX patch management works.

For the more experienced readers you can skip this part.

Let me first explain a little bit about Ifixes  APARS and finally multifixes.

What are iFixes?

We as customers can discover a problem with a specific piece of code or functionality of AIX. Of course, we submit a support case for this and after IBM recognized this as problem that need to be fixed they create a APAR (Authorized Program Analysis Reports) number for this problem. This number is important and makes clear that we have official approvement by IBM that this is a problem.

Of course, we as customers, like to fix it asap. Then in most cases IBM will provide a Ifix for us. (Ifix = Interim fix) or previously called efix (emergency fix).

IBM complete description of a Ifix / efix:

Is a code update that is delivered by AIX technical support to resolve a specific known problem, or APAR, while an official PTF is undergoing the formal development process. Although Ifixes are always tested before being made available, they are generally tested in very limited scope and within very limited configurations and have not been submitted to the same testing procedures as PTFs.

PTF is a Program Temporary Fix 

Okay we as customers a happy to install this temporary solution (Ifix) on our systems. And we can deploy this temporary patch. And of course we test this solution in our environment, and report back to IBM whether the temporary patch is working or not.

If the patch solved our problem, most likely IBM would build also a PTF (Program Temporary FIX) for this problem.

What’s next?

Finaly IBM will solve the APAR in a SP, TL or new release.

With this APAR number we can track in future releases (SP’s , TL’s) if our fix is available in this AIX version.

Some words about special Critical patches APARS and Ifixes:

There are three types of those specials namely:

Security patches:

These resolve security vulnerabilities in AIX. They are reported as soon as IBM is aware and a fix is available. Tip we always look at the cve score to know the importance of and the urgency of deployment.

HIPERS (High Impact PERvasive)

These resolve defects which have a combination of both high impact and pervasiveness such that IBM thinks all customers exposed to the issues should proactively apply the fix to avoid encountering the problem. For example, to prevent a unplanned system down.

PE (PTF in Error)

These resolve issues where a prior PTF caused a regression in functionality, breaking something that used to work.

So far so good, we can deploy temporary solutions for a defect we noticed, and of course we always deploy HIPERS, and security patches, on regular basis.

And yes we use my favorite inventory tool for this FLRT (Fix Level Recommendation Tool) or with the newest version of AIX we can use emgr_check_ifixes. 

See also this nice video of my fellow Champion Andrey: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwj6mMmq_OyTAxWRSPEDHcj0Al0QtwJ6BAgTEAI&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DiQrUNoFhB38&usg=AOvVaw30QehuDPipd3sdXX3mh0X5&opi=89978449

Challenges with multiple fixes (combo fixes).

Let’s talk about what if you have more than one Ifix for the same file set.

For example, take the fileset bos.mp64 (this is the AIX kernel).

But this is also the case for other filesets of course, if you have more than one Ifix on the same fileset.

When you deploy an Ifix on a fileset then this fileset is locked by emgr and you cannot deploy another fix on that same fileset.

So this is the first challenge, for example if you have a HIPER Ifix and you have a security fix that must be deployed ASAP.

Then the next step is to raise a new case, with the request for a multi-fix. Second challenge is the ETA for this combo-fix, that can take a few weeks. For security fixes this is not ideal, also not for a HIPER fix.

Dilemmas:

What are we going to deploy meanwhile? The security patch or the HIPER?

We always look at the risk and impact for each fix. For instance, if the security fix has a very high CVE score 8 or higher, then we might choose for the security patch.

Also, if the HIPER is not likely to happen in our environment, then the decision is easy.

But what if the risk is high for both?

More challenges:

What if there are not two but four or even more Ifixes must be built into one combo fix.

Testing every fix in the combo fix will be a challenge also, but that is not the worst. What is one fix appears not to be not okay, the whole process must be done again, also on the IBM part, which will cause again a delay. Also, the dependencies between the fixes can be also an issue.

Another dilemma is how fast there will be a new SP (Service Pack) that solves all issues.

Or maybe it solves a particular problem with one of the Ifixes, then again a new combo fix is required, for the non-fixed issues.

Another challenge with combo-fixes can be security scanning software such as compliance check tools, for example cis-cat, but also PowerSC do give false positives here.

Why? They only look if a certain label of the output of emgr -l is applied.

If we installed a combo fix you only see one label, but this label can solve more than one fix! Because it a label of a combo fix.

The only way to check this correctly is with the following commands:

Before you install the combo fix:

display the combo efix file:

 emgr -d -e efix_file_to_installed -v3

after you installed the combo fix:

emgr -l -L combo_efix_label -v3

With this -v3 verbose level you get all the information you liked to see and what fixes are deployed with this label. See example below:

FILE NUMBER:      1

   LOCATION:      /usr/lib/boot/unix_64

   FILE TYPE:     Standard (file or executable)

   INSTALLER:     installp

   SIZE:          101528

   CKSUM:         06390

   ACL:           DEFAULT

   PACKAGE:       bos.mp64

   MOUNT INST:    no

Installp Prerequisite Information:

==================================

PREREQUISITE NUM:      1

   FILESET:            bos.mp64

   MINIMAL LEVEL:      7.3.3.1

   MAXIMUM LEVEL:      7.3.3.1

   TYPE:               PREREQ

   LEVEL AT INSTALL:   7.3.3.1

Efix to Efix Prerequisite Information:

======================================

No efix to efix prerequisites data.

APAR information:

=================

APAR number:      IJ56892

APAR abstract:    SYSTEM HANG IN PROCFSGETATTRCWDROOT()

APAR number:      IJ56896

APAR abstract:    system crash in process_proc_table or proc_detach_exit

APAR number:      IJ56881

APAR abstract:    CRASH DUE TO MEM CORRUPTION WHEN USING SYSLOG-NG

Description:

========================== EFIX DESCRIPTION =============================

730 TL03 SP01 level multi-ifix for the following APARs

IJ56892 - SYSTEM HANG IN PROCFSGETATTRCWDROOT()

IJ56896 - system crash in process_proc_table or proc_detach_exit

IJ56881 - CRASH DUE TO MEM CORRUPTION WHEN USING SYSLOG-NG

=========================================================================

The only tools that give you the right information is FLRT and emgr_check_ifixes.

Another common issue (this has nothing to do with combo fixes) is that for Ifixes and the replaced files the TE (Trusted Execution) database file is not completely updated.

I explained in other blog posts earlier, but let’s explain this briefly, because it’s still an IBM idea that is not completely honored.

For a security Ifix they will deliver an .sec file and the emgr fix.

With this .sec file it’s now possible to verify the Ifix package before installing it and if it’s tampered with, or if the download was not right, it will not install which is GREAT!

For other Ifixes (non security Ifixes) no .sec file is delivered, what is not right in my opinion. But what is delivered (due the partly honored IBM Idea is an updated entry in the tsd.dat (database of TE).

This for the following items: owner, group mode, size, and hash_value, the latter is also GREAT, because we can verify with trustchk -n <replaced_file>  if the hash value is correct, so the executable, file, lib etc. is not tampered with.

What is still not okay is that cert_tag = VOLATILE and   signature = VOLATILE:

Especially for HIPER fixes and combined security fixes we would like to prove to our auditors that those critical fixes are built by IBM and are not tampered with (provided with a .sig file).

End note:

As you may noticed, I like discussions to improve the security for AIX Patch management and other  AIX security subjects. Any feedback or comments on this blog is much appreciated.

Documentation used and IBM Ideas:

If you like my ideas for having also .sig files for HIPER fixes please vote:

https://ideas.ibm.com/ideas/AIX-I-779

and you like my idea for having a complete correct TE (tsd.dat) file that proves the build is done by IBM, and signed with a signature and cert_tag please vote:

https://ideas.ibm.com/ideas/AIX-I-778

Reference used:

AIX Service Strategy and Best Practices