With the PowerVC 2.0.2 release, IBM Cloud Object Storage support has now been integrated with PowerVC. Now, you can register an IBM COS bucket on PowerVC and use it to upload or download image OVAs between PowerVC and IBM Cloud. (For more information on what image OVAs are, see Inside a Backup node exported image OVA).
Currently, only IBM Cloud Object Storage is supported. Support for other external Cloud Object Storages may be available in future releases.Now, let's dive in to this blog and see how you can leverage this new support for IBM Cloud on PowerVC, and move a step further from the traditional image import/export to a wider scope of use - importing and exporting images between different PowerVCs via the IBM Cloud !First things first
To begin working with IBM COS on PowerVC, you need to have:
- An IBM Cloud Account and an Active Cloud Object Storage (COS) service instance.
- A bucket on the IBM COS service instance.
If you are new to IBM Cloud, it may help to go through this IBM COS getting started documentation.
Registering an IBM COS on PowerVCFor PowerVC to be able to access your COS bucket, the IBM COS first needs to be registered on PowerVC. Every IBM COS instance has service credentials associated with it, which help locate the cloud resource and allows other agents to authenticate, interact and perform operations on it such as uploading and downloading files from it.Steps for registering a COS instance on PowerVC:
1. Get the service credentials for the IBM COS instance and the bucket name from IBM Cloud as shown in the below example. These values need to be picked up and will be used in the subsequent steps when registering the COS on PowerVC:
Getting your IBM COS Service credentials from IBM Cloud
cos_api_key_id : New API key that is created for the Service ID.
cos_auth_endpoint: An authentication token is generated for the request. Use https://iam.cloud.ibm.com/identity/token.
cos_endpoint: It defines the bucket's resiliency. If Cross Regional - data spreads across several metropolitan areas, if Regional - then only one area.
cos_resource_crn: Unique identifier for the instance of Object Storage the credential accesses.
2. For getting the COS endpoint value for the COS bucket, the endpoint needs to be picked according to the location of the bucket.
i. If the bucket location is Regional e.g. us-south, then from the Endpoints page – the Regional endpoint for that bucket location needs to be picked :
Regional Endpoints on IBM Cloud for your IBM COS Instance
Note: The location of the bucket can be fetched from the ‘Location’ value for the particular bucket in the Buckets page on IBM Cloud.
ii. If the bucket location is Cross-regional e.g. us-geo or ap-geo – the Cross-Regional endpoint for that bucket location needs to be picked :
Cross-Regional Endpoints on IBM Cloud for your IBM COS Instance
3. Go to the COS tab on the Backup nodes page on PowerVC and click on ‘Add COS’ :
You can now provide the Service credentials and Endpoint got from the earlier steps from IBM Cloud as well as the bucket name - to register the IBM COS on PowerVC as shown below :
Name : Display name for the COS
Bucket name: Name of the bucket as it appears on IBM Cloud
Type : Type of the Cloud Object Storage (only IBM Cloud Object Storage is supported currently)
API key ID : 'apikey' value from the IBM COS Service credentials
Resource CRN : 'resource_instance_id' value from the IBM COS Service credentials
Endpoint : COS endpoint value got from the Endpoints page for the bucket location
Authentication endpoint : Always set as "https://iam.cloud.ibm.com/identity/token" for IBM cloud
An IBM COS can also be registered on PowerVC using the COS Registration API. Note: All COS CRUD operations, upload/download of images and listing of the COS images in a bucket, can also be done using the COS REST APIs.Viewing images on IBM COSAfter a COS is registered on PowerVC, you can click on the COS and go to the ‘image list’ tab for that COS to view the image OVAs present on that COS bucket on IBM Cloud along with the size of the image in GB. You can also use the COS List images API.
List of image OVAs in a registered IBM COS bucket on PowerVC
Note: The size of the image OVA is rounded off to the nearest GB. If the size of the volume is a small one e.g. 1 GB, then after the level 9 pigz compression of the backup files is done, it is possible to have an image OVA with a size in KB/MB. In which case, it will be displayed as 0. The image OVA can still be downloaded/imported and the image will get restored to its original size.Editing COS detailsThe Service credentials for a registered COS can be edited by clicking on the ‘Edit Connection’ option for the COS. The COS name can be edited by going to the details page for that particular COS and editing the name field. You can also use the COS Update API to do the same.De-registering a COSTo de-register a COS from PowerVC, you can select the COS and click on the ‘Remove’ option, or use the COS Delete API.
Note: Registration and De-registration of COS buckets from PowerVC does not create/delete the COS bucket from IBM Cloud. These operations are only w.r.t to PowerVC. Same applies for COS update operations.Upload and download images between IBM COS and PowerVCUploading an image to IBM COSWhen an image is exported, the image OVA is created on the backup node as per the backup_ova_path ( ova location ) set in the cinder.conf file of the Backup node. This image OVA can now be uploaded from the Backup node to an IBM COS bucket. Once the image is uploaded, the image OVA file is deleted from the backup node. The uploaded image OVA can be seen listed in the IBM Cloud console – in the bucket’s Objects list.Steps to Upload an image OVA to IBM COS:1. Select an Available ( or upload_failed ) image backup from the Backup node Images list and click on ‘Upload to COS’. The image backup can also be selected from the ‘Image backups’ tab of the Backup Nodes page.In the example below, an image that was exported (without upload) to a Backup node - has been selected, and we’ll upload it to IBM COS.
Choose an exported image backup to upload to your IBM COS bucket2. Select the COS bucket to which the image OVA should be uploaded. If no bucket is selected, the COS default of the backup node will be used. When registering a backup node, we can set the Default COS value i.e. the name of a COS bucket registered on PowerVC, that we want to use as the default bucket for all uploads from a particular Backup node. If neither a COS bucket nor a COS default is selected/associated with the backup node – the upload operation will fail. ( If you do not wish to upload the image to COS when exporting, de-select the 'Upload to COS' option or set it to False ).
Provide the required parameters for the Upload3. Once the image is uploaded, the image backup can be seen in the ‘uploaded’ state in the Backup node’s image list and the container value will be updated to the <COS-bucket-name>/<image-OVA-name.ova> to reflect the current location of the image OVA :
An image backup uploaded to IBM COS4. On the IBM Cloud console, the uploaded image OVA can be seen listed under the Objects list of the bucket. You can also use the COS Upload API to upload images to COS :
The uploaded image OVA in the bucket's Objects list on IBM CloudDownloading an image from IBM COSAn image OVA can be downloaded from the IBM COS bucket to any registered backup node. It will be downloaded at the OVA location in the backup node. This downloaded image OVA can later be imported to PowerVC. You can also retry download for an image backup if the download fails for some reason ( it can be seen in the 'download_failed' state ). Same applies for the retry of upload using the Upload to COS option, if the upload fails.Steps to download an image from IBM COS: 1. Select an image in the uploaded (or download_failed) state from the image backups in the Backup Node Image list.
Choose an uploaded image backup for download.An image backup for download can also be selected from the COS Image list or the Image backups tab of the Backup Nodes page :
The COS Image list for a bucket on PowerVC shows all image OVAs residing in that COS bucket. Images can be also downloaded from here.2. Click on ‘Download’ provide a display name for the image, and optionally choose a backup node to download it to and a description, if required :Note: When downloading an image OVA, any registered backup node can be selected for download, irrespective of which backup node that image OVA was uploaded from. If no backup node is chosen, the cinder scheduler will select the most suitable backup node from the available backup nodes - based on the space available, status of the cinder backup service and which backup node is the least used one i.e. having the least number of image backups on it.
Provide the required parameters for Download
3. After the download is complete, the image backup can be seen in the Available state - reflecting that it is now available for import :
A downloaded image backup
And the image OVA file can be seen at the OVA location on the Backup node which is reflected in the ‘container’ value. You can also use the COS Download API to download images from COS :
So, this was all about how you can work with IBM Cloud Object Storage on PowerVC 2.0.2. Troubleshooting for COSWhen performing COS operations or import/export with an upload or download - the first level of issues can be checked at the /var/log/cinder/api.log on the PowerVC management node. For more detailed information as to what the exact COS errors are - such as Client errors - see the /var/log/cinder/backup.log on the respective backup node on which the upload/download was attempted.When working with COS, you can run into errors like the ones below. Usually, they occur due to one or the other combination of incorrect COS service credentials, endpoints or bucket name.
Note: When a COS is registered or updated on PowerVC, a connection check to the COS instance is done using the service credentials provided - which fails if any of them is incorrect. So it is unlikely to run into errors due to the above reason in a normal scenario. However, if the service credentials are updated manually in the PowerVC DB or some out-of-band operations are performed on the IBM COS instance at the IBM Cloud backend, which lead to changes in the connection information, these errors may occur.
Some of the possible errors are listed below.1. If in case, while uploading/downloading an image from IBM COS, you see a ‘NoSuchKey’ error like the one below:
An error occurred (NoSuchKey) when calling the PutObject operation: The specified key does not exist. : ibm_boto3.exceptions.S3UploadFailedError: Failed to upload /var/opt/ibm/powervc/ova/singleIMG_Export_13.ova to vk-bucket01/singleIMG_Export_13.ova: An error occurred (NoSuchKey) when calling the PutObject operation: The specified key does not exist.
Check if the cos endpoint is Regional if the bucket location is regional, and similarly for Cross-Regional buckets and endpoints. When the cos endpoint provided for a COS instance is incorrect, the resource cannot be located and the COS cannot be connected to, which causes errors like the ‘NoSuchKey’ or the ‘NoSuchBucket’ error as shown below:
ibm_botocore.exceptions.ClientError: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist.
For more details, see NoSuchKey/NoSuchBucket error when uploading/downloading images from COS.2. In case of an ‘InvalidArgument’ error as so:
ibm_botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the ListBuckets operation: Invalid Argument
Make sure the COS resource crn provided is the ‘resource_instance_id’ value from the IBM COS Service credentials and not the ‘iam_serviceid_crn’ value. Even though both crn values my look similar, but the ‘iam_serviceid_crn’ value is an unique identifier for the Service ID, whereas the ‘resource_instance_id’ is an unique identifier for the COS instance. At all times, it is the ‘resource_instance_id’ value that you need to pick.
For more details, see Invalid Argument Client error when doing COS Operations.
3. For an ‘AccessDenied’ error check if you have exceeded the usage quota limit for your IBM COS Service instance :
A Client Error occurred while Downloading the Image OVA upload-MultiVOl_2.ova from COS bucket new.bucket90 : An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
IBM Cloud has a free tier Lite plan available for creating a COS service instance - which allows you to transfer up to 25 GB/month of data to COS. There is also a Standard Plan available which can offer a higher usage quota on a pay-per-use basis. Learn more about the available options at IBM Cloud service plans.
An ‘AccessDenied’ error can also occur if the access policies for the COS bucket are set incorrectly. Every COS bucket has access policies such as Reader/Writer policies having different levels of permissions to perform operations on the bucket.
For more details, see Access Denied error when uploading an image to COS.
4. Only image OVAs exported from PowerVC 2.0.2 can be imported and restored on PowerVC 2.0.2 using the import/export API or from the PowerVC GUI. If you upload an old format image OVA and try to download and import it to PowerVC 2.0.2 using the Import API, the import will fail. This is due to the difference in the OVA formats between the old PowerVC image OVAs and the image OVAs created using the PowerVC 2.0.2 Export API. The older image OVAs use the OVF file for storing the image properties whereas the new format image OVAs store the same information in a JSON file. For more details, see Inside a Backup node exported image OVA .
If you wish to import an image from an older version of PowerVC to PowerVC 2.0.2, you can use the powervc-image CLI to import that image. And then do an export operation from PowerVC 2.0.2 using the Export API or the PowerVC 2.0.2 GUI - to create the new format image OVA for that image. You can find more details in this blog on image import/export.
Conclusion
The integration of IBM Cloud Object Storage with PowerVC 2.0.2 has unleashed a plethora of capabilities and a lot more possibilities for the future releases, enabling PowerVC for Hybrid Cloud.
Hope you enjoyed reading this blog, as much as I enjoyed writing it ! Feel free to reach out to me in case of any queries. See you, until next time : )
PowerVC Knowledge Center Link: PowerVC
Keep watching our social outlets for more interesting information about PowerVC ! Find and follow us on Facebook, LinkedIn, YouTube and Twitter.
Blog Author:
Suman Shaw