PowerVM

 View Only

Power Systems Flexible Service Processor (FSP) Security

By Ratan Gupta posted Thu June 11, 2020 11:54 PM

  
Security vulnerabilities in the network elements (Compute servers, interconnect routers and switches, gateways etc.) have been exploited to launch Denial of Service (DOS) attacks and inflict loss of customer data and more importantly cause breach of trust. In order to protect ones network from such malicious attacks, the system admin must have a thorough understanding of the security controls provided by the network components. It is also critical to understand the inherent hardware and firmware/software limitations in each of the network elements to effectively prevent an intrusion.

All IBM Power Systems that support PowerVM are equipped with a Flexible Service Processor (FSP) which is an always-on management processor helping in out-of-band management of the server. Thus the FSP is the external face of a POWER system providing various platform management interfaces. Consequently, it is critical to understand the potential vulnerabilities these interfaces can expose the system to and adopt best practices to minimize the attack surface.

Hide them if you can

POWER servers are typically deployed in a private network. Most customer deployments chose to deploy their servers in this manner in order to eliminate external intrusions. In the deployment below, the eth0 interface is configured with a 192.168.10.0/24 subnet address while the eth1 interface is configured with a 192.168.20.0/24 subnet address. There is one Hardware Management Console (HMC) responsible for each of the subnets. While the HMC (in the demilitarized zone) is accessible from the WAN, the FSP interfaces in the private network are not.

FSP Security Figure 2

While this deployment vastly reduces the attack surface, some customers may have the need to remotely manage the POWER servers, thereby justifying the need to provision WAN accessible addresses on the FSP's ethernet interfaces.

Opening up the stack

FSP uses a Linux distribution based on the 2.6.32 kernel which provides the architecture specific drivers. The firmware relies on the OpenSSL for secure access and utilizes open source as well as custom built software to provide NIST SP800-131a compliant security.
FSP Security Figure 2

Isn't SSL vulnerable?

Various vulnerabilities have been identified with SSL protocol, most well known being POODLE. While SSL is deprecated in POWER8 it is user configurable in previous generations of systems in order to maintain backward compatibility. National Institute of Standards and Technology (NIST) published requirements (NIST SP800-131a) on the cryptographic algorithms and key lengths to be supported and provided a deadline for implementations to comply with them. Compliance to NIST SP800-131a is not be supported on legacy firmware versions (prior to FW770) due to lack of dependent software packages. The following table defines the matrix for support of SSLv3 and compliance to NIST SP800-131a.
Feature FW350, FW730, FW740, FW760 Release FW770, FW780 Release All POWER8 Releases
SSLv3 Support Yes Yes Deprecated (TLSv1.2 only)
NIST Compliance No Yes Yes


The security configuration interface depends on whether the system is NIST SP800-131a capable or not. Security configuration sub-menu can be accessed from the System Configuration main menu on the ASMI on P7 systems.

On P7 systems below FW770 firmware level, the security configuration menu allows to provision the support for SSLv3. The default configuration on these systems is to have SSLv3 enabled for the HMC communication and disable it for all other services. The system admin can chose to disable SSLv3 for *all* communication interfaces (including that of the HMC) after the HMC is upgraded to a firmware level that supports TLS.

Security configuration on P7 systems with FW770+ firmware levels allows the provisioning of NIST compatibility (SSLv3 support and strength of cipher suites)

  • legacy
    • SSLv3 is enabled on all service processor communication interfaces. This mode provides a basic security level and is suitable for interoperability with the hardware management console (HMC), CIM client and browsers with no TLS support.
  • nist_compat
    • SSLv3 is enabled only on service processor communication interface with the HMC. This mode enables interoperability with HMC having no TLS support while providing a high security level. Choose this mode when the HMC has no TLS support and the CIM client and browsers are capable of communication over TLSv1.2.
  • nist_sp800_131a
    • SSLv3 is disabled on all service processor communication interfaces. This setting provides the security strength in compliance with NIST SP800-131A-recommendations. This mode uses TLSv1.2 to provide a very high security level. Choose this mode when the hardware management console (HMC), CIM client and browsers are all capable of communications over TLSv1.2.

SSLv3 support is deprecated on P8 systems and NIST SP800-131a compliance is mandated and hence the menu for security configuration is not available on these systems.

The following interfaces provide guidance for the security configuration on those P7 systems.
FSP Security Figure 3
               FW770/FW780 - NIST Supported                                                             FW350/FW730/FW740/FW760 

What's in a port?

The open services and ports on the FSP are detailed in the table below.
FSP Security Figure 4

But my network doesn't run / need CIM support...

It is certainly a good idea to keep the attack windows closed, such as disabling a service when it's not used. FSP provides the interface to selectively enable / disable the management interfaces to suit the deployment. The figure below show the associated ASMI screens to selectively enable/disable External Services.
FSP Security Figure 5

How about Risks of using IPMI protocol?

The US-CERT published Alert (TA13-207A) to identify the risks of using IPMI. While certain vulnerabilities are inherent to the protocol, the implementation of IPMI protocol on the FSP mitigates others as highlighted in the table below.
Risks identified in TA13-207A Risk mitigation in FSP based systems
Passwords for IPMI authentication are saved in clear text. Passwords are encrypted and persisted in the flash in FSP based systems.
Knowledge of one IPMI password gives you the password for all computers in the IPMI managed group. This vulnerability does not apply to FSP based systems since passwords are encrypted prior to being saved. Deployments must discourage reuse of password across different systems.
Root access on an IPMI system grants complete control over hardware, software, firmware on the system. Console access to the FSP is disabled. User can open an IPMI SOL console to a partition and log in with root credentials ( need password here ). Once user has logged in and then if SOL is deactivated and activated, the password is not prompted. Hence, if a root user has logged in to partition and gets the session deactivated, that session can be re-activated by anybody else. IPMI protocol requirements for SOL deactivation itself is silent on this.
BMCs often run excess and older network services that may be vulnerable. In FSP based systems user could disable network services via ASMI. In general, this may not really be an IPMI specific security issue.
IPMI access may also grant remote console access to the system, resulting in access to the BIOS. While this is a feature of IPMI, user still needs the IPMI session password to activate the SOL session.
There are few, if any, monitoring tools available to detect if the BMC is compromised. FSP firmware is validated using security test tools such as Nessus and Qualys.
Certain types of traffic to and from the BMC are not encrypted. This is a usage issue. User could choose to have an un-encrypted SOL session.
Unclear documentation on how to sanitize IPMI passwords without destruction of the motherboard. Not applicable to FSP based systems since passwords are never stored in clear text.
IPMI 2.0 Cipher Type 0 - Authentication Bypass Vulnerability. FSP rejects if client attempts to open session via Cipher 0.

Okay, Is there a possibility of an *Identity Leak*?

SLP protocol on the FSP advertises (multicasts) its capabilities for management clients to discover useful information for operations such as asset identification, firmware upgrades, etc. Evidently, availability of system information in the WAN may provide an attack surface for a malicious user with sufficient knowledge of the vulnerabilities. While the user may chose to disable SLP protocol (see above), it is worthwhile to be aware of the info shared while it is enabled.
Protocol Parameter Value
SLP Key Advertisement Attributes (type=cec-service-processor),(serial-number=106CE87),(machinetype- model=9119-MME),(fru-serial-number=0),(name=b101a),(frame- number=0),(cage-number=0),(ip-address=9.3.115.124,169.254.3.147), (web-url=https://9.3.115.124:473),(slot=1),(bpc-machinetype-model=0), bpc-serial-number=0),(Image=fips840/b0210b_1507.840),(cimom- namespace=root/ibmsd),(secure-cimom-port=5989),(firmware-image- info=TC840_003),(mtms=9119-MME106CE87), (MicroActiveCodeImage=TC840_003), (MicroInactiveCodeImage=TC840_003),(IPv6- ipaddress=fe80:0000:0000:0000:6eae:8bff:fe7c:c914,),(IPv4- enabled=true),(IPv6-enabled=true),(level=4)

Is ssh/telnet enabled on the FSP?

The management interfaces are limited to ASMI, NETC, IPMI & CIM on the FSP. ssh/telnet access is NOT enabled as a security measure thus preventing root hacks and privilege escalation attacks.

Feature Description
Command Line Access Not Available
SSH/telnet Not Available
Firewall Configuration Allows traffic to/from whitelisted ports only.

Moreover firewall (CANNOT be disabled) setting allows only white-listed ports to be accessed on the FSP.

Why does the browser flag certificate warning when accessing the ASMI?


POWER systems are shipped with certificate self signed by IBM internal Certification Authority (CA). Those certificates must be imported and trusted in the browser to stop the certificate warnings from being flagged. The certificates on the FSP can be modified to those signed by a trusted CA. Certificates may also be refreshed when those currently deployed have or about to expire. The ASMI screen shot below shows how the available options.
FSP Security Figure 6

Certificate management interface provides access to :

Is a login required to access the ASMI functions?


Yes it is. Furthermore the number of concurrent login sessions is limited to prevent a DOS attack and repeated failed login attempts are followed by a cool off period before which the user can login again.

Session/Login Management Attributes Value
Session ID 128 bit Random Number
Failed Logon Attempts Session Auto-Logout Time 5 , Unlocks after 5 mins.
Value 15 mins
128 bit Random Number 5 , Unlocks after 5 mins. 15 mins 3
Maximum Parallel Logins Allowed Session ID generated on logon and deleted on logout
Session ID Life Yes
Cache-Control Secure Yes
httpOnly Yes
Cookie Creation Only on successful login

How are the OWASP Top 10 vulnerabilities addressed in the FSP?

The Open Web Application Security Project (OWASP) is a non-profit organization that provides unbiased information about application security. The OWASP Top 10 represents a broad consensus on the most critical web application security flaws. Vulnerability

Vulnerability Applicability Reason
Injection None Injection is possible either as LDAP/OS/SQL. Service Processor has no LDAP server running. It has no remote shell/command line access for OS commands. As per SQL, it does not store any credential information in database and also does not have any user interfacing operation which interacts with database.
Broken Authentication & Session Management Organizational control for Password Policy Password strengths are allowed as per organization control. Currently Service Processor does not enforce any specific password policy requirements.
Cross Site Scripting No Service Processor does not use client side scripts and has taken care of input validation
Insecure Direct Object References No Service Processor does not have this functionality
Security Mis-configuration Customer option to choose security configuration based on interoperability requirement. Security configuration as security modes are provided and recommended to be used based on customer interoperability requirements. There are no other user configurable options allowed.
Sensitive Data Exposure No All confidential data in motion and rest is encrypted
Missing Function level Privilege No Users are provided interfaces for operation based on their logged in user id privileges.
Cross Site Request Forgery No Cross-Site Request Forgery (CSRF) protection is provided by implementing session based CSRF Token.
Using components with Known Vulnerabilities No All third party packages are updated with latest security patches
Unvalidated Redirects & Forwards No Service Processor does not use this functionality

What tools are used to validate the security compliance?

FSP uses the same tools used by system admins of customer networks to validate the firmware. Nessus and Qualys are two of the widely used security vulnerability test tools used to validate the FSP firmware. Each firmware release is subjected to exhaustive security test coverage. An illustrative test report below from a Nessus scan shows that there are no high severity vulnerabilities discovered and the two medium severity vulnerabilities need to be resolved at the customer deployment via suitable security certificate configuration.
FSP Security Figure 7

How to allow only certain IP to access FSP?


ASMi provides a menu "Network Access" which can be used to allow/deny certain IP addresses. "ALL" is a value for the IP address, which can be used to allow/deny any IP addresses.

Example: Update HMC IP address in the scan "allowed" list, and update "ALL" in the
denied list. This will allow only the configured HMC IP address to access the FSP. Any other IP will not be allowed to access the FSP.

Warning: If FSP is configured to allow only a single HMC IP address, in the event that the HMC loses its IP address the FSP will no longer be accessible. To prevent FSP lockouts, we recommend having a backup IP address added to the Allow list.
FSP Security Figure 8

FAQs

  • Which ports are opened on the FSP?
    • FSP provides multiple management interfaces like HMC, ASMI, CIM & IPMI. Thereby specific ports are opened as required by these interfaces.
  • How does FSP protect itself against vulnerabilities?
    • FSP provides defense in depth security by:
      • Enabling (whitelisting) only required ports via firewall; all others are disabled
      • Access via ports which are opened is only through management interfaces and require authentication Web application is protected from popular security vulnerabilities as defined by OWASP
      • FSP does not allow any command line support like telnet/SSH
      • FSP has controlled access for users depending on the type of user
  • How does the FSP authenticate accesses?
    • Access is restricted to specific users and follows Unix based authentication.
  • How does FSP identify an HMC?
    • Management interface communicating to FSP is identifiable by its *tool type* which is specific to the particular management interface.
    • Service Processor does client authentication by verifying HMC certificate.
  • What prevents a hacker from getting into the FSP?
    • FSP provides no local command line support via SSH/telnet
    • FSP provides only limited remote access via management interface
    • FSP controls remote access via proper authentication/authorization support
    • FSP maintains all the protocol/services supported for management interfaces to their latest security patches/fixes
    • FSP communication to management interface is over secure channel
    • Web Application Vulnerabilities like CSRF, CXX, SQL injection are either fixed or not applicable on Service processor web application ASMI
    • Session Management / Cookie Management is strongly taken care to address any identified vulnerabilities

Contacting the PowerVM Team

Have questions for the PowerVM team or want to learn more?  Follow our discussion group on LinkedIn IBM PowerVM or IBM Community Discussions

#PowerVM
#powervmblog
0 comments
43 views

Permalink