Cloud Platform as a Service

 View Only

Configuring an IP firewall in Cloud Internet Services to increase protection of your Code Engine workloads

By Enrico Regge posted Mon December 02, 2024 09:09 AM

  

Written by
Sascha Schwarze (schwarzs@de.ibm.com)
Senior Software Engineer - IBM Cloud Code Engine

Enrico Regge (reggeenr@de.ibm.com)
Architect - IBM Cloud Code Engine

Abstract

IBM Cloud Code Engine is a fully managed, serverless platform that runs your containerized workloads, including web apps, REST APIs, microservices, event-driven functions or batch jobs. In this article, we're demonstrating how to increase the protection of Code Engine apps and functions, that are exposed to the public internet through a custom domain, by a firewall provided by IBM Cloud Internet Services.

In this blog post, we are going to demonstrate how to setup the Cloud Internet Services and Code Engine to achieve the following objectives:

1. Exposing the clients IP address to the Code Engine app so that its source code can make use of it.

2. Enable a firewall in Cloud Internet Services (CIS) that blocks or allows specific IP address to access an app.

3. Setup a domain lockdown in CIS that only allows certain IP ranges to access an app.

4. Harden the solution by making sure that requests are always coming through CIS. 

We'll provide instructions for the IBM Cloud Console, and the IBM Cloud CLI to setup our examples. No matter which of these interfaces is your favourite, we've got you covered 🚀.

Prerequisites

As a prerequisite, you'll need to purchase a custom domain, delegate its management to CIS and configured a load balancer that points to the Code Engine project that contains your custom domain mapping and the app or function that should respond to HTTP requests.

In case you are not there yet, we have your back: In our blog post about "Deploy a Globally Distributed Web App on Your Domain with Code Engine and Cloud Internet Services" we guide you through all necessary steps, including the creation of Cloud Internet Services "Global load balancer" and its's corresponding origin pools, to hook up your web app hosted on IBM Cloud Code Engine with a domain managed by Cloud Internet Services.

In order to verify whether the IP address reaches the deployed app, we'll use a Code Engine sample that echos HTTP request headers to the calling user. The app can be deployed in Code Engine using the following commands:

# Select the Code Engine project
ibmcloud ce project select --name <your-project>

# Create an app based on the Code Engine sample image
ibmcloud ce app create --name myapp --image icr.io/codeengine/hello --cpu 0.125 --memory 0.25G --min-scale 1 

Once the app is deployed, you'll be able to access it on the URL path "/debug" through its public system domain using your browser, or cURL in your command-line, or any other HTTPS client of your choice:

curl --silent "$(ibmcloud ce app get --name myapp --output url)/debug" | jq

{
  "headers": {
    "host": "myapp.1mr3jprdj7mi.eu-es.codeengine.appdomain.cloud",
    "user-agent": "curl/8.7.1",
    "accept": "*/*",
    "forwarded": "for=172.30.115.128;proto=https, for=127.0.0.6",
    "x-forwarded-for": "172.30.115.128, 127.0.0.6, 127.0.0.6",
    "x-forwarded-proto": "https",
    "x-request-id": "d8bdd2c6-7376-4964-8cf2-114d80ac416e",
    …
  },
  "url": "/debug",
  "method": "GET",
  "env": {
    …
  }
}

Note: To improve formatting, the command above pipes the JSON data output of the cURL operation to the tool jq.

As mentioned before, the sample app "hello" can be used to observe the request input provided to the app. The listed header "x-forwarded-for", is a well-known source for identifying the originating IP address of the client that connects to a web server. But in our case it does only list Code Engine owned internal and external IP addresses. To prove this, use an online services of your choice, like ifconfig.me, ip.me, or whatismyip.com, to obtain your public internet IP address and check whether it appears in the apps debug output… It will not.

curl -4 ifconfig.me                                      

158.175.119.4

At the time we are writing this blog post, our public IP address was "158.175.119.4" 

In the following section, we'll make sure that the clients IP address gets exposed to the app, too.

Configuring CIS to forward the users IP address to your app

In general, there are multiple ways to order a TLS certificate for a custom domain, such as using Certbot and Let's Encrypt, or by ordering an origin certificate in CIS, or through your well established certificate creation initiation/renewal process. For this blog post we chose to use CIS origin certificates. But the choice of how certificates are managed, has no effect on the security features that allow to filter for IP addresses.

In the IBM Cloud Console, following steps can be used to configure your domain properly:

  1. In the Cloud Console, navigate to the "Resource list" and filter for your "Internet Services" instance. Click on the name of the target instance to navigate to its detail page

  2. On the CIS instance details page navigate to "Security > Origin" and Click "Order"

  3. In the "Order origin certificate" panel, keep the defaults and enter the domain name that you want to have handled, e.g. "myapp.example.com". Confirm by clicking "Order"

  4. Copy and paste the ordered "Origin certificate" and "Private key" values into temporary local files

  5. Navigate to the Serverless projects page in Code Engine and click on an existing project, or create a new one

  6. On the project details page, navigate to "Applications" and click "Create"

  7. Choose "myapp" as a name and specifcy "icr.io/codeengine/hello" as the image reference to use. Confirm by clicking "Create"

  8. On the apps detail page, navigate to "Domain mappings" and click "Create"

  9. In the "Create domain mapping" panel, paste the Certificate and private key and enter the fully-qualified domain name, e.g. "myapp.example.com". Confirm by clicking "Create", after copying the CNAME target value into the clipboard

  10. On the same page, select the option "No external system domain mapping" to deactivate the public system domain mapping. Confirm the modal by clicking "Apply".

  11. Navigate to the CIS instance details page (see step 1)

  12. On the CIS instance details page navigate to "Reliability > DNS", scroll to "DNS records" and click "Add"

  13. In "Add record" panel, choose "CNAME" as the type, set your subdomain; e.g. "myapp" as name and copy the CNAME target captured previously into the Alias domain name. Make sure the details output mentions something similar than "myapp.example.com is an alias of custom.<id>.<region>.codeengine.appdomain.cloud.", before confirming the dialogue by clicking "Add"

  14. Once all steps have been completed, the DNS record should look as shown in the screenshot below

The steps in the IBM Cloud CLI are depicted below. Please note, that the commands listed will work on most Linux and macOS distributions. If for some reason they don't work on yours, or if you are using Windows, please consider to use the IBM Cloud Console to create the sample configuration.

# Define your target hostname that should receive traffic on your custom domain and direct it to the Code Engine app 
BASE_DOMAIN=example.com
SUB_DOMAIN=myapp
DOMAIN_NAME=${SUB_DOMAIN}.${BASE_DOMAIN}
CE_APP=myapp

# Install Cloud Internet Services plugin 
ibmcloud plugin install cis -f

# List all CIS instances in your account. 
ibmcloud cis instances

# Set the context, by using the ID of the previous command
ibmcloud cis instance-set <ID>

# Obtain the DNS domain ID
DNS_DOMAIN_ID=$(ibmcloud cis domains --output json |jq -r --arg base_domain $BASE_DOMAIN '.[]|select(.name==$base_domain)|.id')

# Create a new origin certificate that is used to protected the traffic between CIS and the Code Engine app
ibmcloud cis origin-certificate-create $DNS_DOMAIN_ID --cert-type origin-rsa --hostnames $DOMAIN_NAME --output json

# Store the output of the "certificate" and "private_key" properties in individual files (make sure to preserve the trailing line feeds)
echo "<certificate_value>" > myapp.crt
echo "<private_key_value>" > myapp.key

# Create a Code Engine TLS secret
ibmcloud ce secret create --name "${CE_APP}-tls" -format tls --private-key-file myapp.key --cert-chain-file myapp.crt
rm myapp.crt
rm myapp.key

# Create a Code Engine domain mapping
ibmcloud ce domainmapping create --name $DOMAIN_NAME --target $CE_APP --target-type application --tls-secret "${CE_APP}-tls"
CNAME_TARGET="$(ibmcloud ce domainmapping get --name $DOMAIN_NAME --output json | jq -r '.cname_target')"

# Update the app configuration and deactivate its system-generated public endpoint
ibmcloud ce app update --name $CE_APP --cluster-local

# Create the DNS record in CIS to make sure that traffic for the domain is routed to Code Engine
ibmcloud cis dns-record-create $DNS_DOMAIN_ID --type CNAME --name $DOMAIN_NAME --content $CNAME_TARGET --proxied true

It is important to point out, that we enabled the proxy mode on the created DNS record in CIS, as this setting makes sure that traffic flows through the security and performance functions on CIS. In essence, this means that the clients IP address will be extracted and exposed into request headers that are forwarded to the Code Engine app.

Now, let us check what kind of headers are exposed to the app, after completing the configuration. We'll use the same tooling as before but this time we are calling the external domain, instead of the public system domain provided by Code Engine.

curl --silent https://myapp.example.com/debug | jq

{
  "headers": {
    "host": "myapp.example.com",
    "user-agent": "curl/8.7.1",
    "accept": "*/*",
    "accept-encoding": "gzip, br",
    "cdn-loop": "cloudflare; loops=1",
    "cf-connecting-ip": "158.175.119.4",
    "cf-ipcountry": "DE",
    "cf-ray": "8dd7289a6cfbd2d2-FRA",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "forwarded": "for=\"[158.175.119.4]\";proto=https, for=172.30.207.64, for=172.30.217.195, for=127.0.0.6",
    "k-proxy-request": "activator",
    "x-envoy-attempt-count": "1",
    "x-envoy-external-address": "172.30.207.64",
    "x-forwarded-for": "158.175.119.4,172.30.207.64,172.30.217.195, 127.0.0.6, 127.0.0.6",
    "x-forwarded-proto": "https",
    "x-request-id": "f24a0d54-7668-4236-84a4-9fb635a62c0e"
  },
  "url": "/debug",
  "method": "GET",
  "env": {
    …
  }
}

As you can see, the IP address that is exposed in the app, now maps to our own IP address 🎉.

In the next section, we'll configure a firewall in CIS that only allows certain calling IP addresses.

Enable CIS firewall capabilities

CIS firewall rules offer power and flexibility by targeting HTTP traffic and applying custom criteria to block, challenge, log, or allow certain requests.

To demonstrate that a firewall actually blocks traffic, let us start with a rule that blocks just our current IP address:

  1. In the CIS instance, navigate to "Security > Firewall rules" and click "Create"

  2. In the "Create firewall rule" panel

    • Enter a name, e.g. "block-me"

    • Click on the Disabled toggle, to activate the rule after creation

    • Choose "Hostname" as field , "equals" as operator and as value put in the hostname, e.g. "myapp.example.com"

    • Click "And" to add a second row

    • Choose "IP source address" as field , "equals" as operator and as value put in your IP address, e.g. "158.175.119.4"

    • As response action choose "Block"

    • Complete the panel by clicking "Create"

  3. Use cURL or your browser to verify whether you can still access your domain

Now, let us turn this around. Our goal is to establish an allow list and block all others from accessing our app.

Adjust the existing firewall rule, as depicted in the screenshot below:

  • Set the name to "block-all-others-but-me"

  • Set the operator of the "IP source address" field to "does not equal" and re-enter your IP address, e.g. "158.175.119.4"

  • Confirm the update by clicking "Save"

Almost immediately after updating the rule, you should be able to access the app again. Okay, we just proved that the firewall is capable of blocking certain IPs and are now using the inverse rule to make sure that all other IP addresses are blocked. By further playing around with the field types, you'll notice that firewall rules are quite powerful as you can use various characteristics besides the users IP address to allow or reject traffic.

The following command demonstrates how to configure the same CIS firewall rule through the CLI:

# Create a firewall rule that is applied 
ibmcloud cis firewall-rule-create $DNS_DOMAIN_ID \
    --description block-all-others-but-me \
    --action block \
    --expression "(http.host eq \"$DOMAIN_NAME\" and ip.src ne 158.175.119.4)"

How to lockdown a domain for certain IP ranges

Some real world scenarios, require to allow-list hundreds or thousands of IP addresses. To make that happen, one can make use of so called IP ranges (aka CIDR blocks). To apply such IP range filter to our domain, we'll make use of different security feature, called IP firewall and more specifically, we'll make use of the feature called "Domain lockdown"

Note: Before we proceed, please make sure to deactivate the firewall rule that we created in the previous section to ensure that it does not interfere with the domain lockdown configuration that we'll add next.

In the IBM Cloud Console,

  1. In the CIS instance,

    • Navigate to "Security > IP firewall"

    • Click on tab "Domain lockdown"

    • Click "Create"

  2. In the "Create a domain lockdown rule" panel

    • Enter a name, e.g. "my-company-network"

    • Provide the base URLs that should be in scope, e.g. "myapp.example.com” and "myapp.example.com/*"

    • Enter one or more IP ranges that should be able to access your domain, e.g. "158.175.119.4/32”

    • Complete the panel by clicking "Create"

Using the IBM Cloud CLI, the same rule can be created as follows

# Create a local file that contains the specification of the lockdown rule
cat >ip-firewall-domain-lockdown.json <<EOF
{
  "description": "my-company-network",
   "urls": [ "$DOMAIN_NAME", "$DOMAIN_NAME/*" ],
   "configurations": [{
     "target": "ip_range",
     "value": "158.175.119.4/32"
   }],
   "paused": false
}
EOF

# Create the lockdown rule
ibmcloud cis firewall-create --domain $DNS_DOMAIN_ID --type lockdowns --json @ip-firewall-domain-lockdown.json

# Cleanup the local file
rm ip-firewall-domain-lockdown.json

Once applied, all non-legit callers will be greeted with an error page similar to the following one.

Harden the setup

You have completed a setup which you can use to restrict access to your app endpoint using IP address filters. This is a useful addition to traditional authentication and authorization that you will likely still require for use cases where you want to allow callers to perform different actions depending on their role.

Having mapped your domain to Cloud Internet Services and Proxy mode enabled, you have actually hidden that the backend of this domain is hosted on Code Engine. Still, users with insider knowledge who somehow know that your backend is running in Code Engine, can figure out how to by-pass your Cloud Internet Services instance and its firewall rules.

To exclude those from accessing your app, you can extend your setup with an Edge Function that injects an HTTP header with a secret value that is only known to you. You can then verify the presence of this header in your app and reject all calls that are missing it.

Next, you'll deploy the following piece of source code as an Edge Function. But before doing that, replace the sample value "Some-Secret" with a secret that only you should know and store the entire source code snippet in a local file, called "ce-cis--edge-function.js".

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Set a custom header
 * @param {Request} request
 */
async function handleRequest(request) {
  // Best practice is to always use the original request to construct the new request
  const newRequest = new Request(request);

  // Inject a secret that is only known to the Code Engine app
  newRequest.headers.set("x-cis-secret", "Some-Secret");
  return fetch(newRequest);
}

In the IBM Cloud Console,

  1. In Code Engine

    • Navigate to the "Secrets and configmaps" on the project overview page

    • Create a generic secret named "cis-secret" and add a key-value pair "CIS_SECRET=Some-Secret"

    • Navigate to the details page of app "myapp"

    • Click on "Configuration" and switch to tab "Environment variables"

    • Click "Add environment variable"

    • In the "Add environment variable" panel select "Reference to full secret" and select the "cis-secret" and click "Add"

    • Update the apps configuration by clicking "Deploy"

  2. In the CIS instance
    • Navigate to "Edge Function"

    • Click on tab "Actions",

    • Click "Create",

    • Replace the sample source code with the content of locally stored file "ce-cis--edge-function.js" and click "Save"

    • Click on tab "Triggers" and click "Create"

    • Provide your domain URL as "Trigger URL”, e.g. "myapp.example.com”

    • Select the action that has been created in the previous step and confirm by clickling "Save"

    • Click "Create" to create a second trigger

    • Provide the domain including a slash and a wildcard character has "Trigger URL", e.g. "myapp.example.com/*"

    • Select the action that has been created in the previous step and confirm by clickling "Save"

Using the IBM Cloud CLI, the same rule can be created as follows

# Adjust the app by introducing a secret 
# make sure to replace "Some-Secret" accordingly with the value that you stored in "ce-cis--edge-function.js"
ibmcloud ce secret create --name cis-secret --from-literal CIS_SECRET="Some-Secret"
ibmcloud ce app update --name myapp --env-from-secret cis-secret

# Create the CIS Edge Function along with the two triggers
ibmcloud cis edge-functions-action-create --name add-a-secret-header --javascript-file ./ce-cis--edge-function.js
ibmcloud cis edge-functions-trigger-create $DNS_DOMAIN_ID "$DOMAIN_NAME" --name add-a-secret-header
ibmcloud cis edge-functions-trigger-create $DNS_DOMAIN_ID "$DOMAIN_NAME/*" --name add-a-secret-header

Alright, everything is in place. Let us check whether it works as expected. As you can see below, the header value, which has been injected by the CIS Edge Function, and the environment variable value are both exposed to the app instance. 

curl --silent -4 https://myapp.example.com/debug |jq

{
  "headers": {
    "x-cis-secret": "Some-Secret",
    …
  },
  "url": "/debug",
  "method": "GET",
  "env": {
    ...
    "CIS_SECRET": "Some-Secret",
  }
}

Last thing to do is to adjust the apps source code by adding a check that makes sure the injected header value equals the environment variable value. In case these values are not the same, the caller has not been routed through the CIS instance and hence the request should get rejected by the app. In case you'll want to apply the same mechanism to a Code Engine Function, the following documentation page provides details on how to access headers. For a Node.js app, the following code snippet can be used to harden your solution:

// If the app has been configured to check the CIS secret, 
// make sure that the request header value 'x-cis-secret' matches the configured secret.
// If it doesn't match, assume that the request bypassed the CIS firewall and reject it.
if(process.env.CIS_SECRET && request.headers['x-cis-secret'] !== process.env.CIS_SECRET){
  response.writeHead(403);
  return response.end();
}

And that is it 🎉

Conclusion

We have configured an IBM Cloud Internet Services instance in a way that allows to pass the clients IP address to an IBM Cloud Code Engine app. Furthermore, we demonstrated how allow list and block single IP addresses as well as entire IP ranges on an entire domain, using Cloud Internet Services security features.

What's next?

If you have feedback, suggestions, or questions about this post, please reach out to us; e.g. on LinkedIn Enrico Regge or via E-Mail.

0 comments
14 views

Permalink