View Only

Instana backend upgrade best practices(on-premises)

By Xing Tian posted Mon October 17, 2022 02:19 AM

Like

Instana backend upgrade best practices(on-premises)

Co-Authors:

Sunny Tian(tianxing@cn.ibm.com) - Developer for Instana Integration, CP4MCM

Lu Xiao (xiaolu@cn.ibm.com) - Developer for Instana Integration, CP4MCM

Wang XiaoHang(xhwangbj@cn.ibm.com) - GCG CSM Architect

Introduction

To ensure business continuity, system administrators need to do maintenance in daily work, such as doing an upgrade when new features are added. In addition, they need a robust, tested disaster recovery plan when an unexpected outage occurs. Backup and disaster recovery are used to achieve this goal.

Backup is the process of making an extra copy (or multiple copies) of data. You back up data to protect it. You might need to restore backup data if you encounter an accidental deletion, database corruption, or a problem with a software upgrade.
Disaster recovery, on the other hand, refers to the plan and processes for quickly re-establishing access to applications, data, and IT resources after an outage. This plan might involve switching over to a redundant set of servers and storage systems until your primary data center is functional again.

This blog mainly focuses on introducing the upgrade best practices of Instana backend (on-premises) during daily maintenance, including backup and upgrade steps.

Instana environment prerequisites

Ensure that you have a running on-premises Instana backend.

Upgrade process

Back up data and configurations
Download images
Import images
Upgrade Instana backend
Restore Instana if upgrade fails

Backup data and configurations

Before an upgrade, you need to back up data and configurations to avoid ~~the~~ important data is going missing ~~when~~ if the upgrade fails. Follow these steps:

1. If the Instana backend is running on VM, you can take a snapshot of the VM to back up both the data and the configuration.

2. Back up data, which means copying files.

Note: You need to stop Instana by running the command instana stop before you copy the Instana data files.
See the following example procedure:

instana stop
tar -cf ./data_backup/data.tar /mnt/data/
tar -cf ./data_backup/metric.tar /mnt/metric/
tar -cf ./data_backup/traces.tar /mnt/traces/
cp settings.hcl ./data_backup/
instana start

For more information, refer to the configuration guide here.

3. Back up configurations by using a tool that can back up self-defined configurations such as events, dashboards, etc. For more information, refer to the guide here.

a. Install the tools by running the following commands:

wget https://nodejs.org/dist/v14.17.1/node-v14.17.1-linux-x64.tar.xz
scp -r root@9.46.72.72:/root/images/instana-move .
ln -s /root/images/node-v14.17.1-linux-x64/bin/node /usr/bin/
which node
ln -s /root/images/node-v14.17.1-linux-x64/bin/npm /usr/bin/
npm -v

b. Execute the tool to back up the configurations by running the following commands:

/root/ServerBackup/instana-move
npm install
npm run build
node . export all ~/ServerBackup

c. Use the tool to back up configurations by running the following commands:

Note: "export" is used for backup, "import" is used for restore, "export" is used to export application configurations

$ node . export application {writable local directory path for exported files}

import application configurations

$ node . import application {writable local directory path for exported files}

delete application configurations exported

$ node . delete application {local directory path of json app files to delete}

export website and website alert configurations

$ node . export website {writable local directory path for exported files}

import website and website alert configurations

$ node . import website { local directory path of json files to import}

delete website and website alert configurations exported

$ node . delete website {local directory path of json files to delete}

Download images

Find a box that can connect to the Instana website to download images.

To download new package images, you need to confirm the exact upgrade version first. Use the command yum --showduplicates list instana-console to get the target release version.

yum --showduplicates list instana-console
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 11 kB/s | 2.4 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 15 kB/s | 2.8 kB 00:00
Installed Packages
instana-console.x86_64 227-1 @instana-product
Available Packages
instana-console.x86_64 170-0 instana-product
instana-console.x86_64 170-1 instana-product
instana-console.x86_64 170-2 instana-product
..............................................
instana-console.x86_64 225-4 instana-product
instana-console.x86_64 227-0 instana-product
instana-console.x86_64 227-1 instana-product
instana-console.x86_64 227-2 instana-product
instana-console.x86_64 227-3 instana-product
instana-console.x86_64 227-4 instana-product
instana-console.x86_64 227-5 instana-product
instana-console.x86_64 229-0 instana-product
instana-console.x86_64 229-1 instana-product
instana-console.x86_64 229-2 instana-product
instana-console.x86_64 229-3 instana-product
instana-console.x86_64 231-0 instana-product
instana-console.x86_64 231-1 instana-product
instana-console.x86_64 233-0 instana-product

2. Update Instana console to a specified version.

Confirm the specified version as "231-1". Pay attention to the small version "-1", it is needed to specify the version.And then upgrade instana console to this specified version with the following command:
yum update instana-console-231-1 # use update if you already have lower version Instana-console
yum install instana-console-231-1. # use install if Instana-console is already removed

3. Download images with the download key by using the command "instana images export -k DOWNLOAD_KEY". The downloaded version will be the same with the Instana console version.

instana images export -k $DOWNLOAD_KEY
Images export ✓
File is stored at /root/LubanInstana/APM225/export-231-1.tar

Import images

1. Copy images to the Instana backend machine, and then import images by using the command as follows:

instana images import -f ./export-231-1.tar

Images import ⣷ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e] 6.144kB/6.144kB","idImages import ⣾ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e] 6.144kB/6.144kB","idImages import ⣽ {"status":"Loading layer","progressDetail":{"current":6144,"total":6144},"progress":"[==================================================\u003e] 6.144kB/6.144kB","idImages import ✓

2. Check the Instana version and instana images version.

instana version
Instana self-hosted: 227-1 3.227.192-0 (e70cb126bc21e38c1256072355d6eaab2af7ef4b)

Upgrade Instana backend

After images are imported, the upgrade can be done by using the command ./instana update -f <the location of the settings.hcl file>. See the following example:

./instana update -f /root/instana/Instana_APM193/docker-installer/settings.hcl
Setup host environment ✓
Handle certificates ✓
Ensure images ✓
Clean docker containers ✓
Check data directories ✓
Create configurations ✓
Migrate data stores ✓
Run components ✓
Check components ✓
Setup environment urls ✓
Run nginx ✓
Welcome to the World of Automatic Infrastructure and Application Monitoring
Note: Sometimes the upgrade process might be very slow if a new version database is migrated during upgrade, so you need to wait until the upgrade process ends.

Restoring Instana

If the upgrade fails, a restore is needed. This restore procedure has different use cases. It can be used as a regular restore from a snapshot in an emergency case or used to restore EC2 instances inside auto-scaling groups for an automated restore after termination.

If there is a snapshot for the Instana backend VM, directly restore VM and start Instana to see whether the Instana backend is recovered.

If there is no snapshot, follow the steps:
1. Restore Instana backend configurations by using the command " instana restore -f ./settings.hcl". You need to restore Instana on an empty instance. The restore procedure consists of the following steps:

（1）Set up the instana-console repository.
（2）Install Docker.
（3）Install instana-console in the same version as on the original instance.
（4）Import the data backups to the mount paths of the file settings.hcl.
（5）Run the command instana restore -f <path-to-settings.hcl>.

2.Restore the backup configurations with Instana-move tool, refer the the guide here.

Some useful commands for troubleshooting

Sometimes the upgrade can’t complete sucessfully. Here are some common problems and commands, which can help you debug and fix the problems.

1. The upgrade needs to be done in two neighboring versions.

For example, it’s not supported to upgrade 227 to 233. In this case, you can upgrade 227 to 229, upgrade 229 to 231, and then upgrade 231 to 233.

[root@luban-instana APM225]# instana update -f ./settings.hcl
Setup host environment ✓
Handle certificates ✓
A direct update from version 227 to 233 is not supported. The latest supported version is 229

2. If the upgrade fails due to database problems, try to run the following commands several times. The problem might be fixed.

instana repair
instana restore -f ./settings.hcl

3. To remove Instana completely for re-installation if the upgrade fails, follow these steps:

Remove Instana-console.
Delete instana images by running the command "instana images prune".

4. If the Instana images fail to be displayed or deleted, run the command as follows:

docker login https://containers.instana.io-u="_" -p="$DOWNLOAD_KEY"
After the Docker login is done, the problem might be fixed.

5. To check whether there are network problems with the instana server, run the following command:

curl -X GET -k -H 'Authorization: apiToken $apiToken' -i ' https://luban-instana.luban.cdl.ibm.com/api/settings/rabc/groups/'

Reference

0 comments

36 views

IBM IT Automation Community

Come for answers, stay for best practices. All we're missing is you.

Instana