Last months I noticed that a lot of excellent blogs were written about how to upgrade and maintain our live support systems for virtualization on Power: the VIO Servers.
Most of the configuration for Power systems exist on at least two VIO servers, for redundancy.
This is because our VIO servers deliver both I/O most common form NPIV and networking to our virtualized logical partitions (LPARS).
Access to a VIO server is most of the time only necessary for us as system administrators and can be done in two ways, via the HMC, or if there is a local ip address configured, via ssh.
Securing your VIOS with IPsec filtering.
So the last subject brings me to this subject: why should we use network filtering at all on VIO servers?
Well, a few answers to this question. First is to prevent accidental misusage of ssh logins, of unwanted persons. For this to succeed choose at least a strong password for the padmin user.
An even a better way is to create more local users with the same padmin rights and strong passwords.
Another way is to enable LDAP on your VIOS so that you can manage your passwords centrally, EXCEPT for the padmin user in case you lose your network! Consider your padmin user as a root user and tread it security wise likewise.
Another answer to this question is that a VIO server is an IBM propriety setup version of AIX, and for most customers seen as a black box.
A standard rollout of a VIO server has many (standard) network ports open, even if you don’t use them. This is ideal for internal attackers, but also for unwanted network scans and auditors telling you that your Unix system is not save.
More solutions are possible.
To prevent unwanted network traffic to your VIO servers, you can put a dedicated firewall in your network and if you do so please choose for a stateful inspection firewall.
Second, a good way is to disable ssh login, so only a login session is possible via the HMC console or via command line HMC ‘vtmenu’.
However most of our admins like to have a ssh login to their VIOS.
Third line of defense is then configuring an external firewall.
If you don’t have the opportunity to install an extra firewall then the IP filter setup is a good alternative.
First a few words about ‘ip_sec’ IP filtering:
IP filtering is a way of blocking and allowing specific ip addresses and certain port numbers to your system.
NOTE: this is not the same as ‘iptables’. The difference between iptables and ip-filtering is that iptables is a statefull inspection firewall and ip-filtering is NOT.
Lets look on our VIOS what we can use to setup IP-filtering.
First let me explain why you should choose for one of those three possible options.
This strongly depends on how secure you’d like to setup your ip-filtering, but also depends on the number of network interfaces you have.
See little flow below:
Option one:
Option one is the most easy way to setup your ip-filtering.
But this option has a few limitations.
This option is make use of the template /home/ios/security/viosecure.ctl
See below a part of this configuration file:
With this option you can configure the ports that should be open, the other ports are automatically blocked.
After editing this configuration file you can enable ip-filtering with:
viosecure -firewall on -reload -lpm -icmp
This line enables the ip filter for IPv4. It also configures the ephemeral ports for LPM (Live Partition Mobility) and the required icmp protocol (both are necessary for LPM).
Pitfall: PLEASE always enable the ip-filtering via an HMC console session or via vtmenu.
This to prevent that you are locked out after you activated the ip-filters if you made a mistake.
With the command: viosecure -firewall view
you can verify your current configuration and check if the firewall (ip-filters) are enabled.
Limitations of this option:
1. This option is only a good option if you have only one (virtual) network interface, it’s not suitable for multihomed (more than one network interface) VIOS servers.
If you use this option on multi homed VIOS It will apply the setting on ALL the interfaces.
2. It is not possible to filter an ip address or ip address ranges. It uses the ‘0.0.0.0’-adresses that will allow every network address.
See below a part of the control configuration file: /home/ios/security/viosecure.ctl
Service Port Remote/Source Service name
#20 ftp-data
#21 ftp
22 Both ssh # SSH Remote Login Protocol
#23 telnet
53 Remote domain # DNS
#80 www # World Wide Web HTTP
123 Both ntp
427 Both svrloc # Server Location
443 https # http protocol over TLS/SSL
636 Both ldapssl # ldap ssl client
657 Both rmc
12601 Both rmc-tls # secure RMC for supported in HMC V10R3
#2279 xmquery # xmquery (PTX/PAIDE protocol)
5987 wbem-rmi # WBEM RMI
5988 wbem-http # WBEM HTTP
5989 wbem-https # WBEM HTTPS
1019 Both rsh
1020 Both rsh
1021 Both rsh
1022 Both rsh
1023 Both rsh # rsh attempts to bind a port from 1023 ~ 513
3901 Both nimsh # nimsh (IANA) primary, i.e. stdin & stdout
3902 Both nimsh-2 # nimsh (IANA) secondary, i.e. stderr
#67 bootp-server
#68 Remote bootp-client
#69 Both tftp # client and server exchange the TID ports
2049 Both nfsd
111 Both port-mapper
1058 Both nim
1059 Both nimreg
Explanation fields of this configuration file:
Service Port = port number to open
Remote/Source = field is default source, but can be Remote or Both
Both creates two ip filter lines one for incoming traffic and one for outgoing traffic.
Remote allows the remote port to be queried (for example for DNS).
Service name = the ports number or the name in the etc/services configuration file.
Option 2:
This option is a combination of option 1 and the VIOS command ‘viosecure -firewall’.
This option overcomes the shortcomings of option1, and can be used for multihomed VIOS servers, and also to specify specific hosts for access.
Only short coming of this option is that you cannot specify address ranges.
Also a bit of a disadvantage is that it’s better to script this.
The good news is that I will log an new IBM Idea for this, to extend the ‘/home/ios/security/viosecure.ctl’ with extra fields for interface, address and a description field. See link below:
https://ideas.ibm.com/ideas/PVMV-I-132
But for now we created a little script, that first uses option1, but does not use the -lpm option when it reloads the ip-filter configuration. This is because we’d like to steer the ephemeral ports on a specific interface.
Use case:
We created a script for the following use case:
We have two virtual network on each VIO server
Network 1: We have on non-routed network that connects to the Power infrastructure, HMC, PowerVC, NIM servers. On this network all the traffic is allowed for maintaining our cluster, in other words RMC traffic and traffic for LPM and nim is allowed.
Network 2: this network is only used for other more general services, such as NTP, Ldaps, and DNS.
We like to hide all the management ports from this network so only necessary services are allowed.
Requirement was that we would like to use as much as possible the ‘viosecure -firewall’ commands, but liked to run this script under ‘oem_setup_env’.
See below the script we created so for this:
#!/bin/bash
###################################################################
# Header
clear
echo "============================================================================"
echo " Setting IP filters on VIOS"
echo "============================================================================"
echo ""
###################################################################
ioscli="/usr/ios/cli/ioscli"
date=$(date +%Y\%m\%d\%H%M)
backup(){
printf "Backing up default config file..."
cp ${ioscli} ${ioscli}-${date}
echo "OK"
}
reload_viosecure(){
printf "Reloading default config..."
${ioscli} viosecure -firewall on -reload -icmp
echo "OK"
}
removing_lpm_ports(){
printf "Removing LPM rules..."
${ioscli} viosecure -firewall deny -port 40001
${ioscli} viosecure -firewall deny -port 40001 -remote
${ioscli} viosecure -firewall deny -port 40002
${ioscli} viosecure -firewall deny -port 40002 -remote
${ioscli} viosecure -firewall deny -port 40003
${ioscli} viosecure -firewall deny -port 40003 -remote
${ioscli} viosecure -firewall deny -port 40004
${ioscli} viosecure -firewall deny -port 40004 -remote
${ioscli} viosecure -firewall deny -port 40005
${ioscli} viosecure -firewall deny -port 40005 -remote
${ioscli} viosecure -firewall deny -port 40006
${ioscli} viosecure -firewall deny -port 40006 -remote
${ioscli} viosecure -firewall deny -port 40007
${ioscli} viosecure -firewall deny -port 40007 -remote
${ioscli} viosecure -firewall deny -port 40008
${ioscli} viosecure -firewall deny -port 40008 -remote
${ioscli} viosecure -firewall deny -port 40009
${ioscli} viosecure -firewall deny -port 40009 -remote
${ioscli} viosecure -firewall deny -port 40010
${ioscli} viosecure -firewall deny -port 40010 -remote
echo "OK"
}
import_rules_admin_lan(){
printf "Importing admin rules..."
# SSH:
${ioscli} viosecure -firewall allow -port 22 -interface en4
${ioscli} viosecure -firewall allow -port 22 -interface en4 -remote
# RMC
${ioscli} viosecure -firewall allow -port 657 -interface en4
${ioscli} viosecure -firewall allow -port 657 -interface en4 -remote
# RMC with TLS
${ioscli} viosecure -firewall allow -port 12601 -interface en4
${ioscli} viosecure -firewall allow -port 12601 -interface en4 -remote
# LPM
${ioscli} viosecure -firewall allow -port 40001 -interface en4
${ioscli} viosecure -firewall allow -port 40001 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40002 -interface en4
${ioscli} viosecure -firewall allow -port 40002 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40003 -interface en4
${ioscli} viosecure -firewall allow -port 40003 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40004 -interface en4
${ioscli} viosecure -firewall allow -port 40004 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40005 -interface en4
${ioscli} viosecure -firewall allow -port 40005 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40006 -interface en4
${ioscli} viosecure -firewall allow -port 40006 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40007 -interface en4
${ioscli} viosecure -firewall allow -port 40007 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40008 -interface en4
${ioscli} viosecure -firewall allow -port 40008 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40009 -interface en4
${ioscli} viosecure -firewall allow -port 40009 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 40010 -interface en4
${ioscli} viosecure -firewall allow -port 40010 -interface en4 -remote
# NIMSH
${ioscli} viosecure -firewall allow -port 3901 -interface en4
${ioscli} viosecure -firewall allow -port 3901 -interface en4 -remote
${ioscli} viosecure -firewall allow -port 3902 -interface en4
${ioscli} viosecure -firewall allow -port 3902 -interface en4 -remote
# NFS
${ioscli} viosecure -firewall allow -port 2049 -interface en4 -remote
# NFS (portmapper)
${ioscli} viosecure -firewall allow -port 111 -interface en4
${ioscli} viosecure -firewall allow -port 111 -interface en4 -remote
# NIM
${ioscli} viosecure -firewall allow -port 1058 -interface en4 -remote
# NIMREG
${ioscli} viosecure -firewall allow -port 1059 -interface en4 -remote
echo "OK"
}
import_rules_prod_lan(){
printf "Importing production rules..."
# SSH
${ioscli} viosecure -firewall allow -port 22 -interface en5 -address server1.x.y
${ioscli} viosecure -firewall allow -port 22 -interface en5 -address server1.x.y
# NTP
${ioscli} viosecure -firewall allow -port 123 -interface en5 -address ntp.xx.xx.xx
${ioscli} viosecure -firewall allow -port 123 -interface en5 -address ntp.xx.xx.xx -remote
${ioscli} viosecure -firewall allow -port 123 -interface en5 -address ntp.yy.xx.yy
${ioscli} viosecure -firewall allow -port 123 -interface en5 -address ntp.yy.xx.yy -remote
# LDAP
${ioscli} viosecure -firewall allow -port 636 -interface en5 -address ldap1.xx.xx.xx
${ioscli} viosecure -firewall allow -port 636 -interface en5 -address ldap1.xx.xx.xx -remote
${ioscli} viosecure -firewall allow -port 636 -interface en5 -address ldap2.xx.xx.xx
${ioscli} viosecure -firewall allow -port 636 -interface en5 -address ldap2.xx.xx.xx -remote
# DNS
${ioscli} viosecure -firewall allow -port 53 -interface en5 -address dns.someone.xx.yy -remote
echo "OK"
}
cleanup(){
printf "Cleaning up..."
${ioscli} viosecure -firewall deny -port 22
${ioscli} viosecure -firewall deny -port 657
${ioscli} viosecure -firewall deny -port 657 -remote
${ioscli} viosecure -firewall deny -port 12601
${ioscli} viosecure -firewall deny -port 12601 -remote
${ioscli} viosecure -firewall deny -port 3901
${ioscli} viosecure -firewall deny -port 3901 -remote
${ioscli} viosecure -firewall deny -port 3902
${ioscli} viosecure -firewall deny -port 3902 -remote
${ioscli} viosecure -firewall deny -port 1019
${ioscli} viosecure -firewall deny -port 1019 -remote
${ioscli} viosecure -firewall deny -port 1020
${ioscli} viosecure -firewall deny -port 1020 -remote
${ioscli} viosecure -firewall deny -port 1021
${ioscli} viosecure -firewall deny -port 1021 -remote
${ioscli} viosecure -firewall deny -port 1022
${ioscli} viosecure -firewall deny -port 1022 -remote
${ioscli} viosecure -firewall deny -port 1023
${ioscli} viosecure -firewall deny -port 1023 -remote
${ioscli} viosecure -firewall deny -port 2049
${ioscli} viosecure -firewall deny -port 2049 -remote
${ioscli} viosecure -firewall deny -port 111
${ioscli} viosecure -firewall deny -port 111 -remote
${ioscli} viosecure -firewall deny -port 1058
${ioscli} viosecure -firewall deny -port 1058 -remote
${ioscli} viosecure -firewall deny -port 1059
${ioscli} viosecure -firewall deny -port 1059 -remote
${ioscli} viosecure -firewall deny -port 123
${ioscli} viosecure -firewall deny -port 123 -remote
${ioscli} viosecure -firewall deny -port 636
${ioscli} viosecure -firewall deny -port 636 -remote
${ioscli} viosecure -firewall deny -port 53 -remote
${ioscli} viosecure -firewall deny -port 427
${ioscli} viosecure -firewall deny -port 427 -remote
${ioscli} viosecure -firewall deny -port 443
## ${ioscli} viosecure -firewall deny -port 443 - remote
${ioscli} viosecure -firewall deny -port 5987
## ${ioscli} viosecure -firewall deny -port 5987 -remote
${ioscli} viosecure -firewall deny -port 5988
## ${ioscli} viosecure -firewall deny -port 5988 -remote
${ioscli} viosecure -firewall deny -port 5989
## ${ioscli} viosecure -firewall deny -port 5989 -remote
${ioscli} viosecure -firewall deny -port 6181
${ioscli} viosecure -firewall deny -port 6181 -remote
${ioscli} viosecure -firewall deny -port 16191
${ioscli} viosecure -firewall deny -port 16191 -remote
${ioscli} viosecure -firewall deny -port 42112
${ioscli} viosecure -firewall deny -port 42112 -remote
${ioscli} viosecure -firewall deny -port 3192
${ioscli} viosecure -firewall deny -port 3192 -remote
echo "OK"
}
view_result(){
printf "Showing result:"
${ioscli} viosecure -firewall view
}
###################################################################
# Executing functions
#
backup
reload_viosecure
removing_lpm_ports
import_rules_admin_lan
import_rules_prod_lan
cleanup
view_result
###################################################################
# Ending
echo "";
echo "============================================================================";
echo "Goodbye";
echo "";
exit
How does it work?
1. This little script configures first via viosecure -firewall the basic setup (see option1) but does not use the -lpm flag.
This option enables the open ports on all interfaces.
2. After this it configures the first network interface in this test case en4 with all the required ports open; no specific addresses specified as this is LAN is completely separate from any other network anyway. (non-routeable network).
3. Then it configures the next interface en5 with only the necessary ports ports open to the right addresses as this is the interface to ‘the outside world’ (routed network).
4. Finale step is to remove all the rules created in step one.
Why we still use option one? Into this script, well the idea behind this that if you do an upgrade or new VIOS install, that you can reuse this template to set default the right ports open (only port filter).
And yes it would be nice if this template had more options so that we do not need this option.
Additional information form man page viosecure -firewall
viosecure -firewall {on [-reload [-force]] [-lpm][-icmp] | off} [-ip6]
Configures the default firewall settings from the filter rules that are defined in Object Data Manager (ODM). If you use the reload option, the ODM rules are deleted and the default values are loaded from the /home/ios/security/viosecure.ctl file.
If the viosecure.ctl file does not exist, the force option uses the default firewall settings. The -ip6 flag specifies to use the IPv6 version of this command.
If the -ip6 flag is not used, IPv4 is used as the default version.
If the -lpm flag is specified, the visosecure command adds the TCP ephemeral ports from the
vioslpm0 pseudo device to permit Live Partition Mobility (LPM) operations.
If the -icmp flag is specified, the ports to allow ICMP packets are opened. Both the -lpm flag and
the -icmp flag must be specified to perform the partition migration operation.
Option 3
This last option describes an option, that you can use for maximum flexibility.
This is however a bit more complicated option but most advanced.
With this option we complete ignore the template used described in option 1 and 2
However the technology used in all the tree options is the same.
All the three option make use of IPSec that is part of the AIX operating system.
And all the options are using, under the hood, the command genfilt to put filter rules into the ODM of AIX.
In other words with the most advanced commands ‘genfilt’, ‘rmfilt’and ‘lsfilt’ you are very flexible.
How to use it?
With this option we configure everything manually and for most of the VIOS users this option is not useful, unless you have to build advanced filter rules.
Because we don’t use the padmin’s ‘viosecure -firewall’-option we have to do everything manually in an script.
First step that we have to do is enable ‘ipsec’, and more specificcally, combined with which version, e.g.
/usr/sbin/mkdev -c ipsec -t 4
/usr/sbin/mkdev -c ipsec -t 6
Whereby the 4 stands for IPv4 and 6 stands for IPv6.
After this we can create filter rules and below I share some examples how to do this.
If you need more detail for all the flags, please look at the man page for ipfilt which is excellent documentation.
See below a few examples on how to create with genfilt ip filtering rule:
For incoming access you can create with genfilt the following rules
# NTP access:
/usr/sbin/genfilt \
-v 4 -a P -s 10.210.252.136 -m 255.255.255.255 \
-d 0.0.0.0 -M 0.0.0.0 -g N -c tcp -p 123 -o any \
-O eq -P 123 -r L -w I -i en5 -D ntp_hostname -n 88
# DNS access:
/usr/sbin/genfilt -v 4 \
-a P -s 10.255.19.224 -m 255.255.255.255 \
-d 0.0.0.0 -M 0.0.0.0 -g N -c tcp -p 53 -o any \
-O eq -P 53 -r L -w I -i en5 -D dns -n 88
Explanation of the used flags:
-a P = permit traffic
-s = Specifies the source address. It can be an IP address or a host name
-m = specifies the source subnet mask e.g. can be used for ranges of addresses.
-d = destination address 0.0.0.0 is any address
-M = Specifies the destination subnet mask
-g N = Apply to source routing? In this case set to No.
-c = protocol in this case tcp, can be udp, icmp, icmpv6, tcp, tcp/ack, ospf, ipip, esp, ah, and all
-p = Specifies the source port
-o any = specifies source port
-O eq = Specifies the destination port
-P = Destination port -O eq -P 123 = ntp destination poort.
-r L = routing local
-w I = directon I= incoming O= outgoing B=Both
-i = interface name
-D = description field
-n = Specifies the filter rule ID. The new rule will be added BEFORE line 88
Displaying filter rules
With the command ‘lsfilt -v -O’ you will display all the filter rules in a list, starting with a filter line number. Without the -O flag the filters are listed in blocks.
Removing filter rules from the ODM
For removing filters you can use the ‘rmfilt’ command
rmfilt -v 4|6 -n fid | all [-f]
Very easy to use, but of cause be careful.
To remove ip filter rule 88 e.g., you can use
rmfilt -v4 -n88
For removing all the filter rules again be careful and make sure that you saved the ‘lsfilt’ output first.
You can use ‘rmfilt -v4 all’
Conclusion:
If you have configured local network (virtual) interfaces on you vio server for network access, you should use some form of network protection, such as external firewall an or an internal ip-filter.
For simple single network interfaces I personally liked the idea of one single control filer (template) that could be adjusted, but this has now a few shortcomings. Therefore I logged already an new IBM Idea for this subject.
If you are a more experienced user, than use the genfilt ip filter commands, with this you are very flexible, but most likely requires more testing also, on the other hand if you are already on an AIX LPAR familiar with this, this is you best option.
Documentation used:
Man pages viosecure, genfilt.
https://www.ibm.com/support/pages/ibm-vios-how-create-custom-viosecure-rules
http://gibsonnet.net/blog/dwarchive/LPM%20and%20viosecure%20(Chris%27s%20AIX%20Blog).html
Endnote:
As you may noticed, I like discussions to approve the security for AIX Patch management AIX Live Update and other AIX (security) subjects.
Any feedback or comments on this blog and other blogs is much appreciated.
#IBMChampion
