How to bypass OpenShift OVS by using hostnetwork

Introduction

With the setting hostNetwork=true, the pod shares the host node’s network namespace rather than being isolated behind a virtual network interface. This means the pod uses the same network interface and IP address as the node itself by bypassing the OpenShift OVS layer.

Using hostNetwork=true can be considered when pods require direct access to host network interfaces, such as binding to reserved ports or reducing network latency for performance-critical applications.

In this article, it introduces how to bypass the OpenShift OVS layer using hostNetwork for troubleshooting a network problem encountered by the wd-discovery-crawler pod.

Note:

1.The purpose of this article is to demonstrate how to bypass the OpenShift OVS layer using hostNetwork=true. It takes wd-discovery-crawler pod as an example. While, it doesn’t indicate any suggestions or recommendations for wd-discovery-crawler  deployment.

2.Use hostNetwork=true sparingly due to security risks, port conflicts, and potential disruption in multi-tenant cluster environments.

1.Backup the K8s resources & configuration of the crawler pod

1.1 Backup the crawler pod yaml files

mkdir backup_wd_crawler

cd backup_wd_crawle

export PROJECT_CPD_INST_OPERANDS=cpd-operands

echo $PROJECT_CPD_INST_OPERANDS

oc project $PROJECT_CPD_INST_OPERANDS 


 for p in $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | awk '{print $1}') ;do oc get pod $p -o yaml -n ${PROJECT_CPD_INST_OPERANDS} > $p.yaml;done 

1.2 Backup the crawler deployment yaml file

 oc get deployment wd-discovery-crawler -o yaml > wd-discovery-crawler-deploy.yaml 

1.3 Backup the /etc/resolv.conf of the crawler pod

 oc exec -it $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | head -n 1 | awk '{print $1}') -- cat /etc/resolv.conf > etc_resolv_pod.conf 

For example, the content of the etc_resolv_pod.conf looks like this.

search cpd-operands.svc.cluster.local svc.cluster.local cluster.local cp.test.example.com vzapicluster.cp.test.example.com 

2.Scale down the Watson Discovery operator

 export PROJECT_CPD_INST_OPERATORS=cpd-operators

oc -n $PROJECT_CPD_INST_OPERATORS

scale deploy wd-discovery-operator --replicas=0 

3. Create the allow-from-hostnetwork network policy

3.1 Set environment variables

export PROJECT_CPD_INST_OPERATORS=cpd-operators

export PROJECT_CPD_INST_OPERANDS=cpd-operands

export PROJECT_CERT_MANAGER=cert-manager

export PROJECT_LICENSE_SERVICE=ibm-licensing

export PROJECT_SCHEDULING_SERVICE=cpd-scheduler 

3.2 Make sure the environment variables are set properly

echo $PROJECT_CPD_INST_OPERATORS

echo $PROJECT_CPD_INST_OPERANDS

echo $PROJECT_CERT_MANAGER

echo $PROJECT_LICENSE_SERVICE

echo $PROJECT_SCHEDULING_SERVICE 

3.3 Prepare the network policy yaml

cat <<EOF > allow-from-hostnetwork.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-hostnetwork
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          network.openshift.io/policy-group: ingress
    - namespaceSelector:
        matchExpressions:
          - key: kubernetes.io/metadata.name
            operator: In
            values:
              - ${PROJECT_CPD_INST_OPERATORS}
              - ${PROJECT_CPD_INST_OPERANDS}
              - ${PROJECT_CERT_MANAGER}
              - ${PROJECT_LICENSE_SERVICE}
              - ${PROJECT_SCHEDULING_SERVICE}
    - namespaceSelector:
        matchLabels:
          policy-group.network.openshift.io/host-network: ""
  podSelector: {}
  policyTypes:
  - Ingress
EOF

3.4 Apply the network policy

oc project $PROJECT_CPD_INST_OPERANDS

oc apply -f allow-from-hostnetwork.yaml 

3.5 Validate the network policy

 oc get netpol allow-from-hostnetwork -o yaml 

4.Add the SCC hostnetwork to the service account wd-discovery-admin

Granting the service account the necessary permissions to create and run pods that use the host's network namespace.

oc project $PROJECT_CPD_INST_OPERANDS

oc adm policy add-scc-to-user hostnetwork-v2 -z wd-discovery-admin 

5.Patch the WD Crawler with new network settings

5.1 Get the default DNS server IP address of the OpenShift cluster 

export OCP_DNS_NAMESERVER=$(oc get svc dns-default -n openshift-dns --no-headers | awk '{print $3}') 
echo $OCP_DNS_NAMESERVER 

5.2 Patch the wd-discovery-crawler deployment with the new network settings 

Note:

Refer to the output of the step 5.1 and 1.3 and change the values of the nameservers and searches in below command accordingly.

Make sure the correct values are specified.

 oc patch deployment wd-discovery-crawler -n ${PROJECT_CPD_INST_OPERANDS} --patch '{"spec": { "template": {"spec": { "hostNetwork": true,        "dnsPolicy": "None", "dnsConfig": { "nameservers": ["172.30.0.10"], "searches": ["cpd-operands.svc.cluster.local","svc.cluster.local", "cluster.local", "cp.test.example.com", "vzapicluster.cp.test.example.com"] } } } }}' 

5.3 Validate and ensure the WD Crawler pods restarted successfully

Check the status of the WD Crawler pod status and ensure all of them are up and running.

 oc -n ${PROJECT_CPD_INST_OPERANDS} get pods -o wide | grep crawler 

Check the hostNetwork settings of the WD Crawler pod. The hostNetwork value should be 'true'.

for p in $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | awk '{print $1}') ;do oc get pod $p -o yaml | grep -i hostNetwork ;done 

Check the DNS settings of the WD Crawler pod

 for p in $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | awk '{print $1}') ;do oc get pod $p -o yaml | grep -i dnsConfig -A7 ;done 

How to revert the changes

1. Remove the SCC hostnetwork from the service account wd-discovery-admin

export PROJECT_CPD_INST_OPERANDS=cpd-operands

oc project $PROJECT_CPD_INST_OPERANDS 

oc adm policy remove-scc-from-user hostnetwork-v2 -z wd-discovery-admin 

2.Patch the network settings of WD Crawler deployment

 oc patch deployment wd-discovery-crawler -n ${PROJECT_CPD_INST_OPERANDS} --patch '{  "spec": {"template": {"spec": {"dnsPolicy": "ClusterFirst",        "dnsConfig": null,"hostNetwork": false}}}}' 

Validate and ensure the WD Crawler pods restarted successfully.

Check the status of the WD Crawler pod status and ensure all of them are up and running.

 oc -n ${PROJECT_CPD_INST_OPERANDS} get pods -o wide | grep crawler 

Check the hostNetwork settings of the WD Crawler pod. There should be no results returned when the hostNetwork set to be 'false'.

 for p in $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | awk '{print $1}') ;do oc get pod $p -o yaml | grep -i hostNetwork ;done 

#Check the DNS settings of the WD Crawler pod

 for p in $(oc get pod -n ${PROJECT_CPD_INST_OPERANDS} | grep crawler | awk '{print $1}') ;do oc get pod $p -o yaml | grep -i dnsPolicy ;done 

3.Delete the allow-from-hostnetwork network policy

oc delete NetworkPolicy allow-from-hostnetwork 

4.Scale back the Watson Discovery operator

 export PROJECT_CPD_INST_OPERATORS=cpd-operators oc -n PROJECT_CPD_INST_OPERATORS scale deploy wd-discovery-operator --replicas=1 

Reference

Connections between host-network and normal pods failing after creating NetworkPolicy

https://access.redhat.com/solutions/7018836