Introduction
NOTE: PLEASE MAKE SURE YOUR LOAD BALANCER IS CORRECTLY CONFIGURED TO FORWARD THE CLIENTS IPS OTHERWISE THIS TUTORIAL WILL NOT WORK. THIS IS NOT SOMETHING WE CAN ASSIST ON AND IT'S DOWN TO YOU TO LOOK AT THE DOCUMENTATION OF YOUR LOAD BALANCER TO ENABLE THIS.
Having multiple load balancers and reverse proxies in your upstream network can cause them to be banned instead of the client IP if the perimeter security module is triggered on the Developer Portal. In 10.0.5.2 and 10.0.1.9 and above we have added Trusted Reverse Proxies which allows a Drupal module to control reverse proxies list making sure the correct IP gets banned.
X-Forwarded-For: <client>, <proxy1>, <proxy2> the first ip should be your client-ip and this is the ip that will be banned, this blog will help with some of the common ways you should setup your network to facilitate this.
In versions older than those, you will have to add all the Load Balancers IPs manually using the Portal Admin CLI after configuring the ingress to forward all IPs https://www.ibm.com/docs/en/api-connect/10.0.5.x_lts?topic=allowlist-apic-service-ip-allowlistadd
VMware / OVA Steps
Configure NGINX Ingress to Forward IPs
First you need to ssh into your Portal OVA
Next edit the configmap kubectl edit configmap -n kube-system ingress-nginx-ingress-controller and add the following options to the data object.
Setting these options makes the ingress controller pass through the inbound X-Forwarded-For header instead of overwriting them. As Drupal needs to know the full chain of IPs so it know what client IP to block.
Enabling Trusted Reverse Proxies
To enable this feature we need to add a environment variable to the Portal Cluster. First you will need to create an extra values yaml file that contains the following
spec:
template:
- containers:
- env:
- name: ENABLE_TRUSTED_REVERSE_PROXY
value: "true"
name: admin
name: www
Next set the extra-values in the project directory to point to this file.
apicup subsys set <portal_subsystem_name> extra-values-file extravalues.yaml
Finally deploy the changes to the Portal Cluster by running
apicup subsys install <portal_subsystem_name>
Kubernetes Steps
Configure NGINX Ingress to Forward IPs
Locate the config map for your NGINX ingress controller. In this example we have installed ours into kube-system yours maybe installed elsewhere.
Next edit the configmap kubectl edit configmap -n kube-system ingress-controller-ingress-nginx-controller and add the following options to the data object.
Setting these options makes the ingress controller pass through the inbound X-Forwarded-For header instead of overwriting them. As Drupal needs to know the full chain of IPs so it know what client IP to block.
Enabling Trusted Reverse Proxies
To enable this feature we need to add a environment variable to the Portal Cluster. First you will need to get the name of you portal cluster by running kubectl get ptl
No that you have the name of your Portal cluster you need to edit it and add the following to the spec object. kubectl edit ptl portal
template:
- containers:
- env:
- name: ENABLE_TRUSTED_REVERSE_PROXY
value: "true"
name: admin
name: www
Openshift (OCP / CP4i) Steps
Infrastructure Node running haproxy located infront of an APIC Instance and NGINX Ingress
We have noticed that this setup is fairly common as well so here are some tips on how to correctly setup haProxy so that it forwards all IP addresses. In the example we will be using an OCP cluster but same can apply for CP4I.
Our haProxy is configured in tcp mode with the following config:
frontend ingress-https
bind *:443
default_backend ingress-https
mode tcp
option tcplog
backend ingress-https
balance source
mode tcp
server master0 <redacted-ip>:443 check
server master1 <redacted-ip>:443 check
server master2 <redacted-ip>443 check
server worker0 <redacted-ip>:443 check
server worker1 <redacted-ip>:443 check
server worker2 <redacted-ip>:443 check
server worker3 <redacted-ip>:443 check
server worker4 <redacted-ip>:443 check
server worker5 <redacted-ip>:443 check
server worker6 <redacted-ip>:443 check
As documented here - How to manage IP security in the Developer Portal, you can get the client ip passed through from the external haProxy using the send-proxy directive 'provided both the load balancer and the ingress controller are compatible with the proxy protocol, and have the protocol enabled'. In the case of our OCP cluster the ingress controller doesn't have the PROXY protocol enabled by default.
First modify the external haProxy config to add send-proxy on to the end of the server lines:
Once you have saved your changes you will need to reload haProxy `systemctl reload haproxy`
Next modify the ingress controller definition with the following block:
First set your project to location of the ingress controller our is located in oc project openshift-ingress-operator
Then edit the Ingress Controller by running oc edit IngressController and adding the following snippet to the spec object
endpointPublishingStrategy:
hostNetwork:
protocol: PROXY
type: HostNetwork
Enabling Trusted Reverse Proxies
To enable this feature we need to add a environment variable to the Portal Cluster. First you will need to get the name of you portal cluster by running oc get ptl or if you are using the top level CR oc get apiconnectcluster
No that you have the name of your Portal cluster you need to edit it and add the following to the spec object. oc edit ptl <portal-name> or if you are using top level like me oc edit apiconnectcluster production.
Clearing Banned IPs
If you have accidentally banned your load balancer IPs and need to clear the bans. Then you can use the Portal Admin CLI command security:clear-bans documented at https://www.ibm.com/docs/en/api-connect/10.0.5.x_lts?topic=security-apic-securityclear-bans
Conclusion
Once you have saved the changes and added the template overrides the www pods should restart with the environment variables set on them and the Trusted Reverse Proxy feature should now be enabled. Testing the perimeter module now you should see the Client IP being banned and not your load balancer’s.
There are some warnings that come with enabling Trusted Reverse Proxy - If clients are able to spoof the x-forwarded-for header, IP bans and other such features on your site could be circumvented. Only enable if you know and understand your upstream network.
#APIConnect #developerportal #portal #drupal