• Login to IBM Cloud via the CLI and target your preferred region and resource group

REGION=ca-tor
RESOURCE_GROUP=Default
ibmcloud login -r ${REGION} -g ${RESOURCE_GROUP}

• Create the Code Engine project

CE_INSTANCE_NAME=cos-to-sql--ce
ibmcloud code-engine project create --name ${CE_INSTANCE_NAME}

CE_INSTANCE_GUID=$(ibmcloud ce project current -o json | jq -r .guid)
CE_INSTANCE_ID=$(ibmcloud resource service-instance ${CE_INSTANCE_NAME} --output json | jq -r '.[0] | .id')

• Create the COS instance and the bucket

COS_INSTANCE_NAME=cos-to-sql--cos
ibmcloud resource service-instance-create ${COS_INSTANCE_NAME} cloud-object-storage standard global

COS_INSTANCE_ID=$(ibmcloud resource service-instance ${COS_INSTANCE_NAME} --output json | jq -r '.[0] | .id')

ibmcloud cos config crn --crn ${COS_INSTANCE_ID} --force
ibmcloud cos config auth --method IAM
ibmcloud cos config region --region ${REGION}
ibmcloud cos config endpoint-url --url s3.${REGION}.cloud-object-storage.appdomain.cloud
COS_BUCKET_NAME=${CE_INSTANCE_GUID}-csv-to-sql
ibmcloud cos bucket-create \
    --class smart \
    --bucket $COS_BUCKET_NAME

• Create the PostgreSQL instance

DB_INSTANCE_NAME=cos-to-sql--pg
ibmcloud resource service-instance-create $DB_INSTANCE_NAME databases-for-postgresql standard ${REGION} --service-endpoints private -p \
'{
    "disk_encryption_instance_crn": "none",
    "disk_encryption_key_crn": "none",
    "members_cpu_allocation_count": "0 cores",
    "members_disk_allocation_mb": "10240MB",
    "members_host_flavor": "multitenant",
    "members_members_allocation_count": 2,
    "members_memory_allocation_mb": "8192MB",
    "service-endpoints": "private",
    "version": "16"
}'

DB_INSTANCE_ID=$(ibmcloud resource service-instance $DB_INSTANCE_NAME --location ${REGION} --output json | jq -r '.[0] | .id')

• Create the Secrets Manager instance. Note: To be able to create secret through the CLI running on your local workstation, we are creating the SecretsManager instance with private and public endpoints enabled. For production use, we strongly recommend to specify “allowed_network: private-only“

SM_INSTANCE_NAME=cos-to-sql--sm
ibmcloud resource service-instance-create $SM_INSTANCE_NAME secrets-manager 7713c3a8-3be8-4a9a-81bb-ee822fcaac3d ${REGION} -p \
'{
    "allowed_network": "public-and-private"
}'

SM_INSTANCE_ID=$(ibmcloud resource service-instance $SM_INSTANCE_NAME --location ${REGION} --output json | jq -r '.[0] | .id')
SM_INSTANCE_GUID=$(ibmcloud resource service-instance $SM_INSTANCE_NAME --location ${REGION} --output json | jq -r '.[0] | .guid')
SECRETS_MANAGER_URL_PRIVATE=https://${SM_INSTANCE_GUID}.private.${REGION}.secrets-manager.appdomain.cloud

• Create an IAM Authorization policy "Key Manager" between SM and the DB

ibmcloud iam authorization-policy-create secrets-manager databases-for-postgresql \
    "Key Manager" \
    --source-service-instance-id $SM_INSTANCE_ID \
    --target-service-instance-id $DB_INSTANCE_ID

• Create the service credential to access the PostgreSQL instance

SM_SECRET_FOR_PG_NAME=pg-access-credentials
ibmcloud secrets-manager config set instance-id $SM_INSTANCE_GUID
ibmcloud secrets-manager config set region $REGION
ibmcloud secrets-manager config set service-url https://$SM_INSTANCE_GUID.$REGION.secrets-manager.appdomain.cloud
SM_SECRET_FOR_PG_ID=$(ibmcloud secrets-manager secret-create \
    --secret-type="service_credentials" \
    --secret-name="$SM_SECRET_FOR_PG_NAME" \
    --secret-source-service="{\"instance\": {\"crn\": \"$DB_INSTANCE_ID\"},\"parameters\": {},\"role\": {\"crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Writer\"}}" \
    --output JSON|jq -r '.id')

Trusted profiles setup

In this section, we are going to create the Code Engine app. On top of this, we’ll make sure that it has everything needed to access the COS, as well as the Secrets Manager instances. So far, the steps to achieve this would have either incorporated making use of Code Engine service bindings, or manually creating a serviceID, access policies, an API key and a Code Engine secret to store it. Neither of this is needed anymore. 

• Create the Code Engine app
  • The parameter --trusted-profiles-enabled="true" causes Code Engine to make a file /var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token available while your container runs. This file contains a short-living compute resource token, which shall be used to authenticate towards the IAM API, as part of login operations. Code Engine will update the content of this file automatically to make sure the token used by your source code is valid at all times. As the compute resource token has a short time-to-live (TTL), make sure that your source code re-reads the file whenever it needs to operate with the token value.
  • The environment variables COS_TRUSTED_PROFILE_NAME and SM_TRUSTED_PROFILE_NAME are used to expose the names of the trusted profiles that the source code should assume in order to gain access to the respective cloud resources.
  • The environment variable SM_PG_SECRET_ID is used to identify the secret of type service credentials needed to access the database.

CE_APP_NAME=csv-to-sql
TRUSTED_PROFILE_FOR_COS_NAME=cos-to-sql--ce-to-cos-access
TRUSTED_PROFILE_FOR_SM_NAME=cos-to-sql--ce-to-sm-access

ibmcloud code-engine app create \
    --name ${CE_APP_NAME} \
    --build-source https://github.com/IBM/CodeEngine \
    --build-context-dir cos-to-sql/ \
    --trusted-profiles-enabled="true" \
    --probe-ready type=http \
    --probe-ready path=/readiness \
    --probe-ready interval=30 \
    --env COS_REGION=${REGION} \
    --env COS_TRUSTED_PROFILE_NAME=${TRUSTED_PROFILE_FOR_COS_NAME} \
    --env SM_TRUSTED_PROFILE_NAME=${TRUSTED_PROFILE_FOR_SM_NAME} \
    --env SM_SERVICE_URL=${SECRETS_MANAGER_URL_PRIVATE} \
    --env SM_PG_SECRET_ID=${SM_SECRET_FOR_PG_ID}

• Create a trusted profile that grants the Code Engine app read access to your COS bucket

ibmcloud iam trusted-profile-create ${TRUSTED_PROFILE_FOR_COS_NAME}
ibmcloud iam trusted-profile-link-create ${TRUSTED_PROFILE_FOR_COS_NAME} \
    --name ce-app-${CE_APP_NAME} \
    --cr-type CE --link-crn ${CE_INSTANCE_ID} \
    --link-component-type application \
    --link-component-name ${CE_APP_NAME}
ibmcloud iam trusted-profile-policy-create ${TRUSTED_PROFILE_FOR_COS_NAME} \
    --roles "Content Reader" \
    --service-name cloud-object-storage \
    --service-instance ${COS_INSTANCE_ID} \
    --resource-type bucket \
    --resource ${COS_BUCKET_NAME}

• Create the trusted profile that grants the Code Engine app read access to secrets managed by the created Secrets Manager instance. Note: The access permission could be further scoped to a single secret group.

ibmcloud iam trusted-profile-create ${TRUSTED_PROFILE_FOR_SM_NAME}
ibmcloud iam trusted-profile-link-create ${TRUSTED_PROFILE_FOR_SM_NAME} \
    --name ce-app-${CE_APP_NAME} \
    --cr-type CE --link-crn ${CE_INSTANCE_ID} \
    --link-component-type application \
    --link-component-name ${CE_APP_NAME}
ibmcloud iam trusted-profile-policy-create ${TRUSTED_PROFILE_FOR_SM_NAME} \
    --roles "SecretsReader" \
    --service-name secrets-manager \
    --service-instance ${SM_INSTANCE_ID}

Once the trusted profiles have been created, they can be reviewed, and managed in the IBM Cloud Console https://cloud.ibm.com/iam/trusted-profiles.

Note: IAM offers quite some flexibility to organize trusted profiles, their links and associated access permissions. For our example, we used two different trusted profiles to demonstrate how source code would need to look like to work with two individual profiles. It is also be possible to put all access permissions into a single trusted profile and link all legit Code Engine components with it. In the end, you can decide what works best for your account setup, use cases, performance requirements, and your security controls.

Event subscription setup

In this section, we’ll complete the setup by adding a Code Engine event subscription that will trigger the application on each object create or update within the COS bucket. It is intentional that the destination of such events is an app. In many scenarios that we see in the field, COS buckets are used to store a huge amount of objects, causing quite a volume of events as each update operation is delivered as a single event. Given the fact that apps can handle multiple requests concurrently, as well as being able to scale dynamically new instances if needed, apps are much better suited than jobs for such event-driven solutions.

• Create an IAM Authorization policy to allow the Code Engine project receive events from COS

ibmcloud iam authorization-policy-create codeengine cloud-object-storage \
    "Notifications Manager" \
    --source-service-instance-id ${CE_INSTANCE_ID} \
    --target-service-instance-id ${COS_INSTANCE_ID}

• Create the subscription for all create and update COS events related to the bucket

ibmcloud ce sub cos create \
    --name "coswatch-${CE_APP_NAME}" \
    --bucket ${COS_BUCKET_NAME} \
    --event-type "write" \
    --destination-type app \
    --destination ${CE_APP_NAME} \
    --path /cos-to-sql

Verifying the sample

• Open a browser and call the default API endpoint. You can use “ibmcloud ce app get --name csv-to-sql --output url” to obtain the public system generated URL of the Code Engine app

• Review the CSV file that we are going to import

curl --silent --location --request GET 'https://raw.githubusercontent.com/IBM/CodeEngine/main/cos-to-sql/samples/users.csv' > CodeEngine-sample-users.csv

cat ./CodeEngine-sample-users.csv

"Firstname","Lastname"
John,Doe
Jane,Doe
"John",Smith
Jane,"Smith"
"A. N.","Other"

• Upload a CSV file to COS, to initiate an event that triggers the application to convert the CSV file contents to SQL statements

ibmcloud cos object-put \
    --bucket ${COS_BUCKET_NAME} \
    --key users.csv \
    --body ./CodeEngine-sample-users.csv \
    --content-type text/csv

Re-render the root API endpoint of the deployed app. After a few seconds the output should contain the additional names that were extracted out of the CSV file that we uploaded to COS

How does that work?

By enabling the app for trusted profiles, Code Engine injects a compute resource token as a file into the file system of each app instance. The compute resource token has a relatively short lifetime of ~60 minutes. The token identifies the running workload of the Code Engine component (application, function, job) it was created from, and in the context it runs in: the project, the region, the resource group. It can be used to obtain an IAM access token, if a proper trusted profile is assumed in the corresponding login operation.

Once the IAM access token has been obtained, the application source code can use it as authentication token towards IBM Cloud APIs, that are fully IAM enabled. For such scenarios, the trusted profile needs to grant access to the respective IBM Cloud service.

Note: It is recommended to cache the obtained IAM access token and only perform the trusted profile login procedure, once the access token exceeded its time-to-live (TTL). Furthermore, the application source code should assume that the injected compute resource token got renewed, too. In consequence, the compute resource token file should be re-read prior performing a trusted profile login procedure.

For those IBM services, which are not fully IAM-authentication enabled, such a token can be used to initiate calls towards a credential store, such as Secrets Manager, to obtain specific credentials for IBM Cloud services or third-party APIs. For such scenarios, the trusted profile needs to grant access to the credentials store. The access permissions granted to specific secret groups define whether a caller has access to all, or only a specific set of stored credentials.

Note: The service credentials, provided by the credentials store, should also be cached locally for performance reasons. Furthermore they should only be re-fetched, if the authentication towards the target API fails with an indication that the credentials are not longer valid. On that aspect, also keep in mind that the access token, that you’ve obtained from IAM in exchange for the compute resource token expires, too. While obtaining the access token, you should grab the property expires_in from the IAM response payload and remember it so that you can initiate a refresh before the token expires.

Now, you might get the feeling that dealing with all these tokens, their expiry dates, and the need to refresh them can easily get quite complex. And you are right 😅

Having said that, best practice is to use official IBM Cloud SDKs, which are available for all IBM Cloud services to access the services API endpoints. By using the ContainerAuthenticator, the authentication, caching and re-authentication procedures are handled automatically, which simplifies your source code a lot, like described in part 1 of this blog post series.

The following code snippet demonstrates how to obtain a service credential secret from Secrets Manager, that contains required information to access an IBM Cloud Databased for PostgreSQL instance.

As input parameters, it requires:

1. the trusted profile name that grants read access to the Secrets Manager instance
2. the API URL of Secrets Manager instance
3. the unique identifier of the service credential secret, that contains the certificate and the connection string to authenticate towards the PostgreSQL database

// Libraries needed to materialize the authentication to SecretsManager
// and to read service credentials from SecretsManager
import SecretsManager from "@ibm-cloud/secrets-manager/secrets-manager/v2.js";
import { ContainerAuthenticator } from "ibm-cloud-sdk-core";

...

// Required environment variables
const smTrustedProfileName = process.env.SM_TRUSTED_PROFILE_NAME;
const smServiceURL = process.env.SM_SERVICE_URL;
const smPgSecretId = process.env.SM_PG_SECRET_ID;

// Create an authenticator to access the SecretsManager instance based on a trusted profile
const smAuthenticator = new ContainerAuthenticator({
  iamProfileName: smTrustedProfileName,
});

// Initialize Secrets Manager SDK
const secretsManager = new SecretsManager({
  authenticator: smAuthenticator,
  serviceUrl: smServiceURL,
});

...

// Use the Secrets Manager API to get the secret using the secret ID
const res = await secretsManager.getSecret({
  id: smPgSecretId,
});

...

// Extract service credentials needed to connect to the PostgreSQL database
const pgCaCert = Buffer.from(res.result.credentials.connection.postgres.certificate.certificate_base64, "base64");
const pgConnectionString = res.result.credentials.connection.postgres.composed[0];

Conclusions and next steps

As part of this blog post, we demonstrated two typical flows that leverage trusted profiles, as a secure authentication and authorization mechanism towards IBM Cloud IAM. On top of this, we proofed that Code Engine workload can be integrated with various IBM Cloud services, such as COS, PostgreSQL and SecretsManager without requiring an API key to be stored in a Code Engine secret.

Trusted profiles enhance the security of Code Engine deployments a lot, as it doesn’t require to store long-living API keys in a Code Engine component. Also, it eliminates the need for complex procedures for API key rotations for those APIs that support IAM authentication. Furthermore, it allows to define fine-granular access permissions scoped to the Code Engine component level. In consequence, we strongly recommend to adopt trusted profiles in combination with Secrets Manager wherever possible. Having said that, existing solutions, which are based on Code Engine service bindings, should be adjusted to adopt trusted profiles, if possible.

What's next?

• You can further explore features that Secrets Manager provide to provide credentials for third-party APIs
• You can explore best practices for rotating and locking secrets managed in Secrets Manager

If you have feedback, suggestions, or questions about this post, please reach out to us; e.g. via E-Mail.