This blog is mentioned as a friendly warning, for those who are working with and testing AIX 7.3 TL04 SP00 and are using IPsec – ipv4/ipv6 tcpip filtering.
I reported this failure already in my first blog of this year, see also the link below:
AIX 7.3 TL04 upgrade and first test results for LKU and LLU
I created a case for this as well, but meanwhile I could fix it by removing two files after upgrading to TL04: /etc/security/ ipsec_filter and ipsec_filter.vc
Removing both files was the temporary solution here. Disadvantage is that we had to apply all filter rules again.
In the same blog post I also made a promise that I will report if there was a solution available for this problem.
Short recap:
What is the problem again?
After an update (in my case I used of course LKU) we tried to load our IP filter rules again, but we run into the following problem:
ipsec_v4 Available
Can not get IPv4 default filter rule.
0519-002 libodm: The CLASS_SYMBOL does not identify a valid object class.
Check parameters, path name and permissions.
Can not change default rule for IPv4 in ODM.
Can not get IPv4 default filter rule.
0519-002 libodm: The CLASS_SYMBOL does not identify a valid object class.
Check parameters, path name and permissions.
Also afterwards when trying to apply new filter rules we got:
Filter 2 for IPv4 does not exist.
ERROR:Proceeding but failed to execute "/usr/sbin/genfilt -n 2 -v 4 -a P -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g N -c udp -O eq -P 53 -r L -w O -l N -f Y -i all -D "dns"":
So in this blog post I come back with updates about this issue:
There is good news and bad news on this subject.
Starting with the bad news:
I hoped that this problem would be solved in the upcoming SP1 for TL04, but it seems that solving this problem would cause some other issues, so the final solution will now likely be available in SP2. That means we will have to wait a long time.
But there is some good news also:
First there is a workaround, and that’s why I created this blog.
Second, we have an APAR for this now with the following description:
“Following the application of APAR IJ53189, structural modifications were made to the Object Data Manager (ODM) class definitions.” IPSEC ODM is the real root cause!
These changes were intended to resolve a defect regarding duplicate rules for ports 32768 and above. However, the schema change results in a binary incompatibility between the updated ODM libraries and existing IPsec ODM entries created under the legacy schema.
See also the technote:
IBM AIX CommApps: IPsec filter not starting due to ODM code change in AIX 7.3 TL04
The workarounds
Method 1: If you are not yet migrated to TL04:
For those LPARS that are NOT already on TL04, it is recommended that you first export all the existing filter ipv4 rules and ipv6 rules if you have them.
expfilt -r -p -f /directory_name
This exports every rule into a plain text file in the given ‘directory_name’.
After this you run the upgrade to TL04 (I used LKU of course in my case).
Then after the upgrade you remove the files:
/etc/security/ipsec_filter
/etc/security/ipsec_filter.vc
After this you import the saved filter rules: Start with the IPsec device for example the v4:
/usr/sbin/mkdev -c ipsec -t 4
Then run the import of your saved rules with:
impfilt -f /directory_name
After this your ip_filters are running again with the new ODM structure.
Alternative without backup:
If you have a script (in my case) that runs the genfilt commands, you only need to remove the /etc/security/ipsec_filter and ipsecfilter.vc
Method 2:
This method describes how you can recover your filter rules if you already upgraded to TL04.
And if you don’t have genfilt scripts or saved your filter-rules before the update.
Before you start make sure to stop your ipsec devices:
rmdev -dl ipsec_v4
rmdev -dl ipsec_v6 if you have them.
From the technote we can read:
cd /etc/security
ODMDIR=. odmget ipsec_filter > /tmp/ipsec_filter.odmadd <- export to temp odm file
rm ipsec_filter* <- removes the ipsec_filter and ipsec_filter.vc
lsfilt <- verify that there are no filters left.
ODMDIR=. odmdelete -o ipsec_filter <- removes the ODM ipsec_filter.
ODMDIR=. odmadd < /tmp/ipsec_filter.odmadd <- recreate the new ODM ipsec_filter.
This will make sure all the rules that existed previously are added back to the freshly initialized ODM.
I also received a draft rebuild script now that saves the current filters after migrating and rebuilds the ODM into the right format.
I tested this script last week and it worked fine.
See below the content of the ODM-rebuild script:
Rebuild script:
#!/usr/bin/ksh93
#1197058 : Rebuild ipsec_filter ODM to resolve schema mismatch on upgraded systems
# ODM objects use 'short' for port fields, but updated class expects
# ulong'(1186808)
ODMDIR=/etc/security
IPSEC_ODM_LOG=/var/log/ipsec_filter_odm_rebuild.log
export ODMDIR
echo "-------------------------------------------" >> $IPSEC_ODM_LOG
echo "IPSec ODM rebuild started : $(date)" >> $IPSEC_ODM_LOG
echo "-------------------------------------------" >> $IPSEC_ODM_LOG
odmshow ipsec_filter > $ODMDIR/ipsec_filter.cre 2>>$IPSEC_ODM_LOG
if [ $? -ne 0 ]; then
echo "ERROR: odmshow failed" >> $IPSEC_ODM_LOG
exit 1
fi
# Change short to ulong in C struct format
sed -e 's/short fltr_src_port;/ulong fltr_src_port;/' \
-e 's/short fltr_dst_port;/ulong fltr_dst_port;/' \
$ODMDIR/ipsec_filter.cre > $ODMDIR/ipsec_filter.cre.new
mv $ODMDIR/ipsec_filter.cre.new $ODMDIR/ipsec_filter.cre
# Validate
if grep "ulong fltr_src_port;" $ODMDIR/ipsec_filter.cre && \
grep "ulong fltr_dst_port;" $ODMDIR/ipsec_filter.cre; then
echo "Validation SUCCESS" >> $IPSEC_ODM_LOG
else
echo "Validation FAILED" >> $IPSEC_ODM_LOG
exit 1
fi
# Backup and rebuild
odmget ipsec_filter > $ODMDIR/ipsec_filter.bak 2>>$IPSEC_ODM_LOG
odmdrop -o ipsec_filter >> $IPSEC_ODM_LOG 2>&1
mkfilt -v4 -u >> $IPSEC_ODM_LOG 2>&1
odmcreate $ODMDIR/ipsec_filter.cre >> $IPSEC_ODM_LOG 2>&1
odmadd $ODMDIR/ipsec_filter.bak >> $IPSEC_ODM_LOG 2>&1
mkfilt -v4 -u >> $IPSEC_ODM_LOG 2>&1
echo "ODM rebuild completed: $(date)" >> $IPSEC_ODM_LOG
exit 0
What this above script does is basically the following:
Before the script class IPsec_filters looked like:
short fltr_src_port; /* offset: 0x52 ( 82) */
short fltr_dst_port; /* offset: 0x56 ( 86) */
After this it looks like:
ulong fltr_src_port; /* offset: 0x52 ( 82) */
ulong fltr_dst_port; /* offset: 0x56 ( 86) */
I left out the complete ODM dump but only those ODM definitions are changed.
Some final words
First some warning on the above script: it is as it is, and if you have not yet upgraded to TL04 I strongly recommend that you backup all filter rules first before the migration and use method 1 (most simple method). In my case we have a script for applying all filter rules in one go.
Also, I recommend that if you are testing with “0” levels please report your findings! We all benefit and help IBM to make it robust.
As always please send you feedback on this blog; I am always interested in a good conversation.