Written By:

Felix Janakow (felix.janakow@ibm.com)

Master@IBM Student - Technical Sales Public Cloud Platform 

Introduction

Based on an existing customer scenario, I was asked to help build a tool that handles the simple but important and repetitive task of performing an automatic public TLS certificate rotation. Since Secrets Manager is capable of rotating the public TLS certificate whenever it exceeds a given time period, you would have to update the renewed certificate within your Code Engine project manually, because Code Engine will not recognize the change in the TLS certificate value that resides in a Secrets Manager instance by itself.

The core principle of the automatic certificate renewal is already described in this blog post I want to give credit to. What was added to the approach described in the article by my colleagues is the principle of least privilege by using private endpoints as well as trusted profiles instead of IAM API keys, and a simple TLS certificate chain assembly mechanism.

The SM‑TLS‑Sync system is an event‑driven pipeline that propagates a rotated public TLS certificate from IBM Cloud Secrets Manager into a Code Engine project secret with minimal always‑on footprint. Everything activates only when a real rotation happens.

As an IBMer, you can find the code and full documentation within this GitHub repository. If you are external and interested in using this asset, please write an DM  or email to me. I will then provide you with the code in a public repo.

Architecture

The architecture is as simple as that:

1. An event notification service listens for rotation events within your Secrets Manager instance and posts the events via a private endpoint to the Code Engine Rotator App, which is essentially just a Flask app that scales up in your Code Engine project when an event is received.
   
2. The received payload will then be disassembled by the Python script to retrieve the Secret_ID of your public TLS certificate in your Secrets Manager.
   
3. With the Secret_ID in hand, the code will authenticate to the Secrets Manager using a trusted profile to locate and retrieve the certificate value.
   
4. The now assembled certificate chain will then be replaced within your Code Engine project, with authentication also done using a trusted profile.
   
   
5. Your secret should now be rotated. The rotation app in Code Engine will scale back down and wait for another event. 
   
   
   

Prerequisites

IBM Cloud Account Setup:

• Active IBM Cloud
• Appropriate IAM permissions for:
  • Creating and managing Secrets Manager instances
  • Creating and managing Event Notification services
  • Creating and managing Code Engine projects
  • Creating trusted profiles and service IDs

Required IBM Cloud Services

IBM Secrets Manager: 

• Secrets Manager instance provisioned
• A public TLS-certificate secret stored in the instance with active roation in place

IBM Code Engine:

• Code Engine project with the app the secret will be rotated for
• TLS certificate in Secrets and configmaps stored within the Code Engine project where the app resides

Components Overview

Secrets Manager (Certificate Origin):

Stores the rotating public TLS certificate. After a rotation it can emit a “secret rotated” event (with secret_id, name, version metadata) only if the instance is first linked to an Event Notifications service in its Settings > Event Notifications.

Without that one‑time connection no events leave Secrets Manager and the pipeline won’t trigger. Once connected, eligible rotations are routed through Event Notifications to the sm‑tls‑sync app.

To find the secret_id, the application will disassemble the received payroll like this:

@app.route("/", methods=["POST"])
def handle_notification():
    
    # Check for the required authentication header
    expected_header_name = "Auth-SM-TLS-Sync"
    expected_header_value = "SM-TLS-Sync-Header-Verify"
    
    received_value = request.headers.get(expected_header_name)
    
    if received_value != expected_header_value:
        abort(403, "Forbidden: Invalid or missing authentication header")

    logging.info("Received notification")
    payload = request.get_json()

    # ===================== Debug Logs =====================
    #WARNING: This exposes sensitive data. For testing only!
    #logging.info("Payload: %s", json.dumps(payload, indent=2))

    try:
        secret_id = payload["data"]["secrets"][0]["secret_id"]  # <--- Payroll disassemble operation
    except (KeyError, IndexError) as e:
        logging.error("Invalid payload format")
        return jsonify({"error": "Invalid payload"}), 400

    service = Service()

Event Notification Service:

Acts as the broker between producer (Secrets Manager) and consumer (your rotation app). A Topic can narrow events with:

• Event type filter (e.g. Secret rotated)
• Advanced JSON condition (e.g. match a specific certificate name)
  A Destination of type “IBM Cloud Code Engine (Application)” points to the private URL of the rotator app and injects the required custom auth header.

Example:

$.data.secrets[0].secret_name == 'your_secret_name'
# Replace your_secret_name with the actual name of your secret within the stings

Code Engine Application (sm‑tls‑sync):

A tiny Flask service that’s normally scaled to zero. First incoming POST:

• Cold start container
• Validate custom header
• Parse JSON: extract secret_id
• Invoke downstream IBM Cloud APIs using short‑lived credentials
• Replace the target TLS secret in the project
• Return 201 → idle → scale back to zero

Code Engine Secret and Configmap store (Distribution point):

Stores the TLS secret in format=tls (cert chain + key). Any app in the same project referencing that secret gains the rotated material on next container start (or reread, depending on usage).

Components Workflow

High Level Sequence:

Data Touchpoints:

• Event Payload (subset):
  • data.secrets[0].secret_id
  • data.secrets[0].secret_name
  • (Optional) rotation metadata / version

• Secrets Manager Response (public_cert):
  • certificate
  • intermediate 
  • private_key
  • expiration_date (can be logged for observability)

Certificate Assembly Logic

1. Start with certificate
2. Append intermediate (if present) separated by a newline
3. Keep private_key isolated
4. Write as tls_cert / tls_key fields in a replace_secret call

        # Combine certificate with intermediate certificate chain if available
        full_certificate = certificate
        if intermediate:
            full_certificate = certificate.rstrip() + "\n" + intermediate.strip()
            logging.info("Combined certificate with intermediate certificate")
        
        logging.info("Preparing certificate with chain for Code Engine")
        
        tls_data = SecretDataTLSSecretData(
            tls_cert=full_certificate,
            tls_key=private_key
        )

        self.ce_client.replace_secret(
            project_id=project_id,
            name=ce_secret_name,
            if_match="*",
            format="tls",
            data=tls_data
        )
        logging.info("Secret updated in Code Engine")

Security & Isolation Layers:

• Network: Private service endpoints; Code Engine app marked private (no public ingress).
• Request Gate: Static header secret (lightweight, acts as a first barrier).
• Identity: Trusted profile eliminates long‑lived API keys and restricts blast radius.
• Principle of Least Privilege: Only read specific class of secrets + write a project secret, nothing else.

Setup:

If you're interested in setting up automatic rotation, please refer to the GitHub project.
The setup process is described in detail there.

As an IBMer, you can access the code and full documentation directly within the internal GitHub repository.
If you're external and would like to use this asset, feel free to send me a direct message or email. I’ll then provide access to the code via a public repository.



If you have feedback, suggestions, or questions about this post, please reach out to me; e.g. via E-Mail.