Co-Authors：
Hai Jun Xu (xuhjbj@cn.ibm.com) - Developer for Instana Synthetic PoP
Rong Zhu Shang(shangrz@cn.ibm.com) - Developer for Instana Synthetic PoP
Rui Yin (yinrui@cn.ibm.com) - Tech lead for Instana Synthetic
What is Synthetic Monitoring?
Instana Synthetic monitoring, also known as proactive monitoring, can simulate actions that an end-user takes on an application from different locations. , and continuously monitors at a specific interval for performance characteristics such as availability and response time. Synthetic monitoring is built on Instana, and it works seamlessly with other Instana capabilities. With the Synthetic monitoring features, you can create Synthetic tests to monitor an application.
What is Synthetic PoP?
Instana Synthetic PoP (Point of Presence) is the agent in which the Synthetic tests are run. Synthetic PoP can be deployed by using the Helm charts.
An air-gapped environment is a network environment that does not have internet access.
This blog covers deploying an Instana Synthetic PoP in an air gapped environment.
Deploying Synthetic PoP in an Air-Gapped Environment
Before you deploy Synthetic PoP, ensure that following prerequisites are met:
· Two Linux virtual machines with the following configuration:
operation system: Ubuntu 20.04
host name: airgap1.fyre.ibm.com
The VM must have internet access.
Operation system: Ubuntu 22.04
Host name: testversion1.fyre.ibm.com
The VM is the target host with no internet connectivity.
Make sure the following tools are installed on both VMs:
You can use the following command to install Docker 20.10 on the main Linux distributions such as SUSE Linux Enterprise or Ubuntu:
curl https://releases.rancher.com/install-docker/20.10.sh | sh
https://helm.sh/
- Synthetic PoP support different Kubernetes distributions such as ocp, minkube, and microk8s. Microk8s is used as an example in the document. Microk8s: https://microk8s.io/
References for installing tools on VM B:
· Installing MicroK8s Offline or in an air gapped environment
· Installing Helm from the Binary releases
root@testversion1:~# tar -zxvf helm-v3.12.2-linux-amd64.tar.gz
linux-amd64/ linux-amd64/helm linux-amd64/LICENSE
root@testversion1:~# mv linux-amd64/helm /usr/local/bin/helm
root@testversion1:~# helm version
version.BuildInfo{Version:"v3.12.2", GitCommit:"1e210a2c8cc5117d1055bfaa5d40f51bbc2e345e", GitTreeState:"clean", GoVersion:"go1.20.5"}
Complete the following steps to deploy an Instana Synthetic PoP in an air-gapped environment:
1. Prepare and get access to the instana docker image registry from VM A: Manually pull all the synthetic pop images by using the username and password. The username is ”_”. Use download key from VM A as the password.
root@airgap1:~/airgap/images# docker login containers.instana.io -u _ -p <downloadkey>
2. Download the necessary images: (Here are using the image.tag from the default values in pop helm chart , different engines with different values.)
docker pull containers.instana.io/synthetic/synthetic-pop-controller:1.264.0
docker save containers.instana.io/synthetic/synthetic-pop-controller:1.264.0 > synthetic-pop-controller.tar
docker pull containers.instana.io/synthetic/synthetic-playback-http:1.264.0
docker save containers.instana.io/synthetic/synthetic-playback-http:1.264.0 > synthetic-playback-http.tar
docker pull containers.instana.io/synthetic/synthetic-playback-javascript:1.264.0
docker save containers.instana.io/synthetic/synthetic-playback-javascript:1.264.0 > synthetic-playback-javascript.tar
docker pull containers.instana.io/synthetic/synthetic-playback-browserscript:1.264.0
docker save containers.instana.io/synthetic/synthetic-playback-browserscript:1.264.0 > synthetic-playback-browserscript.tar
docker pull containers.instana.io/synthetic/redis:1.263.0
docker save containers.instana.io/synthetic/redis:1.263.0 > redis.tar
3. Check whether the following files are available in the images directory:
root@testversion1:~/airgap/images#
-rw-r--r-- 1 root root 851139584 Jul 11 02:54 pop-controller.tar
-rw-r--r-- 1 root root 259670528 Jul 11 02:58 redis.tar
-rw-r--r-- 1 root root 797607424 Jul 11 02:56 synthetic-playback-http.tar
-rw-r--r-- 1 root root 797607424 Jul 11 02:57 synthetic-playback-javascript.tar
4. Archive the image directory:
tar -cvf data.tar ~/airgap/images
5. Copy the tar file to the local registry of the VM B (host machine):
The tar file is around 2.6 GB.
6. Copy the images on VM B, move the data.tar file and release the images:
cd /root/airgap/
tar -xvf data.tar
Skip step 7, if you have a local registry.
7. Create a registry server on VM B:
a. Obtain a certificate authority：
In a production environment, you must obtain a certificate from a CA. In a test or development environment, you can generate your own CA. To generate a CA, complete the following steps:
i. Generate a Certificate Authority Certificate:
1). Generate a CA certificate private key:
openssl genrsa -out ca.key 4096
2). Generate the CA certificate:
Adapt the values in the -subj option to reflect your organization. use the FQDN to connect registry host, specify it as the common name (CN) attribute.
openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=testversion1.fyre.ibm.com" \
-key ca.key \
-out ca.crt
ii. Generate a server certificate:
1). Generate a CA certificate private key:
openssl genrsa -out testversion1.fyre.ibm.com.key 4096
2). Generate the CA certificate:
openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=testversion1.fyre.ibm.com" \
-key testversion1.fyre.ibm.com.key \
-out testversion1.fyre.ibm.com.csr
3). Generate an x509 v3 extension file:
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=testversion1.fyre.ibm.com
DNS.2=testversion1.fyre.ibm
DNS.3=testversion1.fyre.ibm.com
EOF
Use the v3.ext file to generate a certificate for registry host. CRS and CRT file names are same with the registry host name.
openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in testversion1.fyre.ibm.com.csr \
-out testversion1.fyre.ibm.com.crt
b. Provide the certificates to Registry and Docker:
When using authentication, some versions of Docker also require you to trust the certificate at the operating system level.
openssl s_client -showcerts -connect testversion1.fyre.ibm.com:5000 </dev/null 2>/dev/null|openssl x509 -outform PEM >mycertfile.pem
cp mycertfile.pem /usr/local/share/ca-certificates/testversion1.fyre.ibm.com.crt
update-ca-certificates
8. Restrict access:
The simplest way to achieve access restriction is through basic authentication. It is very similar to the basic authentication mechanism on the other web server. The following example shows storing the secrets with htpasswd by using native basic authentication.
Create a password file with one entry for the user “testuser”, with the password “testpassword”:
mkdir auth
docker run \
--entrypoint htpasswd \
httpd:2 -Bbn testuser testpassword > auth/htpasswd
9. Check the prepared items:
root@testversion1:~/airgap#
auth\
htpasswd
mycerts\
ca.crt ca.key
testversion1.fyre.ibm.com.crt
testversion1.fyre.ibm.com.key
10. Start the registry server:
docker run -d \
--restart=always \
--name registry \
-v "$(pwd)"/auth:/auth \
-e "REGISTRY_AUTH=htpasswd" \
-e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
-v "$(pwd)"/mycerts:/certs \
-e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/testversion1.fyre.ibm.com.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/testversion1.fyre.ibm.com.key \
-p 5000:5000 \
registry:2
11. Check the registry server:
root@testversion1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34eadb5814f9 registry:2 "/entrypoint.sh /etc…" 2 days ago Up 2 days 0.0.0.0:5000->5000/tcp registry
12. Load the images to the local registry:
Check whether Docker clients can pull and push to your registry by using its external address, run the following commands:
docker load < synthetic-pop-controller.tar
docker tag containers.instana.io/synthetic/synthetic-pop-controller:1.264.0 localhost:5000/synthetic/synthetic-pop-controller:1.264.0
docker push localhost:5000/synthetic/synthetic-pop-controller:1.264.0
docker load < synthetic-playback-http.tar
docker tag containers.instana.io/synthetic/synthetic-playback-http:1.264.0 localhost:5000/synthetic/synthetic-playback-http:1.264.0
docker push localhost:5000/synthetic/synthetic-playback-http:1.264.0
docker load < synthetic-playback-javascript.tar
docker tag containers.instana.io/synthetic/synthetic-playback-javascript:1.264.0 localhost:5000/synthetic/synthetic-playback-javascript:1.264.0
docker push localhost:5000/synthetic/synthetic-playback-javascript:1.264.0
docker load < synthetic-playback-browserscript.tar
docker tag containers.instana.io/synthetic/synthetic-playback-browserscript:1.264.0 localhost:5000/synthetic/synthetic-playback-browserscript:1.264.0
docker push localhost:5000/synthetic/synthetic-playback-browserscript:1.264.0
docker load < redis.tar
docker tag containers.instana.io/synthetic/redis:1.263.0 localhost:5000/synthetic/redis:1.263.0
docker push localhost:5000/synthetic/redis:1.263.0
13. Check local registry images:
curl -u testuser:testpassword -X GET https://testversion1.fyre.ibm.com:5000/v2/_catalog
{"repositories":["synthetic/redis","synthetic/synthetic-playback-browserscript","synthetic/synthetic-playback-http","synthetic/synthetic-playback-javascript","synthetic/synthetic-pop-controller"]}
14. Install PoP:
a. Download helm charts, click the https://artifacthub.io/packages/helm/instana/synthetic-pop?modal=install link in helm on VM A and then copy the charts into VM B.
The following is an example of the helm chart:
root@testversion1:~/airgap# tar -zxvf synthetic-pop-1.1.4.tgz
root@testversion1:~/airgap# ls
synthetic-pop synthetic-pop-1.0.14.tgz
b. Create secret:
Create a secret in with namespace synap which is used to access the local docker registry. Set the right credentials for docker username and docker password:
kubectl create secret docker-registry my-secret \
--docker-server=testversion1.fyre.ibm.com:5000 \
--docker-username=testuser \
--docker-password=testpassword \
--docker-email=mymail@example.com \
--namespace=synap
c. Run the helm command:
Run the helm command to deploy Synthetic PoP on namespace synap. You can set the imagePath and downloadSecret with Synthetic PoP helm chart.
The following example shows the default values in the values.yaml file:
# Image common path, for instana production, it is "containers.instana.io/synthetic"
imagePath: "containers.instana.io/synthetic"
# Required, either downloadKey or downloadSecret must be provided to pull images
# The string of download key, this value can be found by clicking on "Deploy a PoP" within the Instana Synthetic Monitoring user interface page.
downloadKey: ""
# The name of secret of kubernetes.io/dockerconfigjson type used to pull images from the image registry.
downloadSecret: "instana-io"
Because VM B is an AirGap environment with local docker registry, enter the values shown in the following example:
The value for downloadSecret is my-secret created in the previous step;
The local docker registry server is "testversion1.fyre.ibm.com:5000". Change the value "containers.instana.io" to "testversion1.fyre.ibm.com:5000".
root@testversion1:~/airgap#helm install synthetic-pop \
./synthetic-pop \
--namespace synap \
--create-namespace \
--set controller.location="R254;R254;China;Beijing;39.54;116.23;R246Double on Microk8s" \
--set controller.instanaKey="*****" \
--set controller.instanaSyntheticEndpoint="https://synsfbd1.fyre.ibm.com/synthetics" \
--set redis.tls.enabled=false \
--set redis.password="a1fc5d01bcbb" \
--set imagePath="testversion1.fyre.ibm.com:5000/synthetic" \
--set downloadSecret="my-secret"
15. Check whether Instana Synthetic PoP is deployed successfully, run the following commands:
root@testversion1:~/airgap# kubectl get pods -n synap
NAME READY STATUS RESTARTS AGE
synthetic-pop-redis-66c6c6f9b-p7s5l 1/1 Running 0 4d3h
synthetic-pop-javascript-playback-engine-849bb8bb6d-wxbt7 1/1 Running 0 4d3h
synthetic-pop-http-playback-engine-7bcc8b5c76-bxz4d 1/1 Running 0 4d3h
synthetic-pop-controller-6b57859f58-6vbdc 1/1 Running 0 162m
synthetic-pop-browserscript-playback-engine-595f45594c-vjg25 1/1 Running 0 4d7m
When you complete the procedure to deploy Synthetic PoP in an air-gapped environment, the operational efficiency improves significantly.
#synthetic-pop