The content on this page was originally published on Instana.com and has been migrated to the community as a historical asset. As such, it may contain outdated information on our products and features. Please comment if you have questions about the content.
What is a Zero Width Space?
A few days ago I learned that the Unicode character for ‘ZERO WIDTH SPACE’ is U+200B.
“The zero-width space is a non-printing character used in computerized typesetting to indicate word boundaries to text processing systems.” Source: https://en.wikipedia.org/wiki/Zero-width_space"
How Slack Broke My Kubernetes Deployment
It all started with a built-in alert (from Instana APM) that I got paged on. At first sight it looked like the container of our UI component was not able to start. The issue contains the starting time, the GKE node pool, GKE node name and the K8s pod that was not able to start.
So I started checking the pod status and it came up as “Pending”.
The “Conditions” tab for the pod shows a new scheduled deployment.
Checking the “Events” tab displays the full error message. For each event Instana shows the event type, reason, message, namespace and time so I can easily identify the event that I need to investigate next. The latest event contained a detailed error message that helped me to identify the root cause of the issue.
Here is the error message in a more readable format. As you can see the “zero width space” Unicode character somehow made it into the container image tag. Redeploying the UI component with the correct image version fixed the issue.
Failed to apply default image tag
"/ui-client:2.176.12\u200b7-0": couldn't parse image reference
"/ui-client:2.176.12\u200b7-0": invalid reference format
So how did it get there? It turns out we received a hotfix deployment request for this UI component in our internal chat system (Slack). We copied the container image version from the message and used the version for the deployment. These “zero width space” characters are added for formatting reasons in Slack. Since this character has zero width, we did not see the hidden character when deploying the hotfix.
Searching the Internet it seems we are not the only ones experiencing this issue.
Lessons Learned – OR – How To Prevent Zero Width Space From Causing Problems
To avoid similar issues in the future we added regex checks to our deployment automation, so only valid characters are allowed when deploying components.
We use dedicated Jenkins servers for production deployments. Only SRE team member have access and can deploy releases and hotfixes to production. Jenkins gives us a nice audit trail, who and when components got deployed.
The Validating String Parameter Jenkins plugin allows us to specify valid parameters using regular expressions. This resolves the issue of whitespace characters ending up in the VERSION parameter, see https://plugins.jenkins.io/validating-string-parameter
Small enhancement for Jenkins Job DSL plugin
We are using the Job DSL plugin for Jenkins to automatically generate all Jenkins jobs. The latest version does not support the validating string parameter yet. Therefore we created a pull request to add support for the validating string parameter plugin (https://github.com/jenkinsci/job-dsl-plugin/pull/1226).
Here is an example of what a job description looks like using validatingStringParam().
job('deploy-component') {
parameters {
validatingStringParam(VERSION, '', 'instana-release-\d+-\d+|1.\d+.\d+', '', '')
}
steps {
shell('echo ${VERSION}')
}
}
