Global Storage Forum

Introduction

This blog post provides information on how to use IBM Storage Insights using REST API.

Overview

IBM Storage Insights (SI) is a software-as-a-service (SaaS) solution designed to help organizations manage their storage environments more efficiently and effectively. It provides a unified dashboard for monitoring and managing various storage systems from IBM and other vendors, including IBM block and file storage, object storage, and virtualization systems. It is built as a cloud-native application composed of a collection of micro services running in a cloud-native environment.

A simplified architecture of SI is shown below. Storage systems running on customer's datacentre provide their performance and configuration data periodically to SI where they are processed. The SI GUI offers many functionalities, some of the key features are:

1. Real-time performance monitoring: Get visibility into the health and performance of storage systems, including metrics like latency, throughput, and capacity utilization. 
2. Predictive analytics: Leverage machine learning algorithms to predict potential issues before they become problems, reducing downtime and improving overall system reliability.

Some of the capabilities available on SI GUI can also be performed using SI REST API that we'll illustrate this blog.

References

You can refer to the following documents for information for the comprehensive list of all API that are available.

• OpenAPI documentation
• REST API Documentation

SI REST API comes in two flavors - one for ecosystem partners, e.g., IBM Defender, who are interested in some aspect of the storage system identified by their MTM (Machine Type Model) and serial number, and secondly for SI tenants who want to leverage REST API to get information about their systems.

REST API for Ecosystem Partners

The REST API meant for ecosystem partners requires an API Key that serves as an authentication mechanism. Partners should open a ticket in SI service portal, https://ibm.biz/si-service-request and furnish details to generate an API key. A new API key is sent to the partner by DevOps. The process is illustrated below.

The API key can then be used to call relevant API by passing it in the header in every request. The following example illustrates how to use an API to acknowledge an SI alert.

curl --location 'https://dev.insights.ibm.com/restapi/v1/alerts/<alert-id>/ack' \
--header 'x-api-key: <insert-api-key-here>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "alert": {
        "source": {
            "systemName" : "2107.75KMP11",	
            "systemModel" : "998",
            "systemType" : "5331",
            "systemSerial" : "75KMP11"
        },
        "method": "FCM4"
    },
    "acked": {
        "by": "john.doe@ibm.com",
        "timeInMs": 1704346271967,
        "action": "ACKNOWLEDGE",
        "details": {}
    }
}'

Response 202 Accepted
{
    "result": {},
    "metadata": {
        "rc": "success",
        "took": 3350
    }
}

REST API for IBM Storage Insights Tenants

SI tenants can log into SI GUI with their credentials. The tenant users are authenticated by IBM Security Verify. A user with an Admin role in SI, once logged in, can navigate to the Configuration page where they can create up to 5 API keys per user. A key contains the identity information of the user and has following characteristics:

• Validity. A key can be valid from a day up to a maximum of 2 years
• IBM ID of the user. This allows a key to be used for accessing different types of data depending on user's role.
• Purpose. Keys can be created to be used for different purposes/applications interacting with SI.

A Key contains information to identify the caller. The caller also needs to be authorized by SI to perform the desired operation. This is accomplished by getting a short-lived API token from SI passing the API key. The token is valid for 15 minutes and contains the role and authorization information. Following snippet shows how to create an API token from API key.

curl -XPOST --location 'https://dev.insights.ibm.com/restapi/v1/tenants/<tenant-id>/token' \
--header 'x-api-key: <insert-api-key-here>' \
--header 'Content-Type: application/json' \
--data '{}'

Response 201 Created:
{
    "result": {
        "token": "<token>",
        "expiration": 1710859776000
    },
    "metadata": {
        "rc": "success",
        "took": 1594
    }
}

API token can then be used to call a REST API.

curl --location --request GET 'https://dev.insights.ibm.com/restapi/v1/tenants/<tenant-id>/storage-systems?storage-type=block' \
--header 'x-api-token: <token>' \
--header 'Content-Type: application/json' \
--data '{}'

Response 200 OK

{
    "tenantId": "<tenant-id>",
    "storageType": "block",
    "timeStamp": 1709806458481,
    "data": [
        {
            "written_capacity_limit_bytes": 33121714044928,
            "storage_type": "block",
            "vdisk_mirrors_count": 2,
            "unmapped_capacity_percent": 60.281784,
            "last_successful_probe": 1709791897433,
            "provisioned_written_capacity_percent": 75.94178,
            "capacity_savings_bytes": 2159941779456,
            "raw_capacity_bytes": 103115535679488,
            "provisioned_capacity_percent": 27.106022,
            "mapped_capacity_percent": 39.718216,
            "available_written_capacity_bytes": 21849572376576,
            "mapped_capacity_bytes": 3565039910912,
            "drive_compression_savings_percent": 0.0,
            "probe_status": "successful",
            "available_volume_capacity_bytes": 3750743507968,
            "capacity_savings_percent": 24.058218,
            "remaining_unallocated_capacity_bytes": 2860359151616,
            "overhead_capacity_bytes": 6044906364928,
            "pool_compression_savings_bytes": 0,
            "compression_savings_bytes": 0,
            "compression_savings_percent": 0.0,
            "events_status": "critical",
            "unmapped_capacity_bytes": 5410791416832,
            "last_successful_monitor": 1709806457145,
            "ip_ports_count": "28",
            "overprovisioned_capacity_bytes": 890384356352,
            "remote_relationships_count": "4",
            "condition": "error",
            "capacity_bytes": 33121714044928,
            "used_written_capacity_percent": 34.032482,
            "pools_count": 21,
            "pm_status": "running",
            "unallocated_volume_capacity_bytes": 890384356352,
            "shortfall_percent": 23.738876,
            "used_written_capacity_bytes": 11272141668352,
            "available_system_capacity_bytes": 23004918579200,
            "managed_disks_count": 72,
            "used_capacity_bytes": 10116795465728,
            "volumes_count": 2718,
            "data_collection": "running",
            "available_capacity_bytes": 23004918579200,
            "used_capacity_percent": 30.5443,
            "flashcopy_count": 45,
            "data_reduction_savings_percent": 0.0,
            "disks_count": 48,
            "unprotected_volumes_count": "2662",
            "drive_compression_savings_bytes": 0,
            "provisioned_capacity_bytes": 8977978811392,
            "data_reduction_savings_bytes": 0,
            "available_system_capacity_percent": 69.4557,
            "storage_system_id": "6eeb4d50-cf4d-11ee-81d5-a57b49f560b8",
            "serial_number": "0000020320212D52",
            "pool_compression_savings_percent": 0.0,
            "ip_address": "9.11.98.38",
            "type": "SAN Volume Controller - 2145",
            "model": "SV1",
            "drive_compression_ratio": 1.0,
            "topology": "hyperswap",
            "cluster_id_alias": "0000020320212D52",
            "tenant_id": "01ec8d7b-db37-1be0-a8e5-51dee329c35c",
            "pool_compression_ratio": 1.0,
            "vendor": "IBM",
            "firmware": "8.5.0.11 (build 157.28.2401051201000)",
            "recent_fill_rate": 0.0,
            "name": "SVC-svc5",
            "recent_growth": 0.0,
            "time_zone": "US/Arizona",
            "fc_ports_count": 64,
            "staas_environment": false,
            "total_compression_ratio": 1.0,
            "element_manager_url": "https://9.11.98.38",
            "probe_schedule": "{\"interval\":1,\"intervalUnit\":\"DAY\",\"enabled\":true,\"frequency\":\"REPEATEDLY\"}",
            "acknowledged": false,
            "safe_guarded_capacity_bytes": 0,
            "compressed": false,
            "callhome_system": false
        }
    ],
    "status": 200,
    "message": "success",
    "path": "/restapi/v1/tenants/<tenant-id>/storage-systems?storage-type=block",
    "method": "GET"
}

Using REST API

As shown above, to call a REST API, we need to pass a token in each call. An application integrating with SI over REST APIs should automate the  token generation part so that the REST API can be used uninterruptedly. Following Python code illustrates how to accomplish token creation automatically while calling REST API.

This function gets a token by passing API Key.

def get_token(rest_api_host, api_key, tenant_id):
    """Fetches token from IBM SI RestApi service"""

    url = f'{rest_api_host}/restapi/v1/tenants/{tenant_id}/token'
    req_headers = dict()
    req_headers['x-api-key'] = api_key
    req_headers['Accept'] = 'application/json'

    try:
        with requests.Session() as s:
            retry_count = RETRY_COUNT
            if retry_count > 0:
                ha = requests.adapters.HTTPAdapter(max_retries=retry_count)
                s.mount("https://", ha)
            if DEBUG:
                click.secho("Sending request to %s with headers %s (retries: %s)"
                            % (url, req_headers, retry_count), fg="blue")
            r = s.post(url, headers=req_headers, timeout=DEF_TIMEOUT)
            if r.status_code != 201:
                click.secho("Token creation failed with status %d (%s)" % (r.status_code, r.text), fg="bright_red")
                return None
            click.secho("Token creation successful, token valid until %s" % datetime.datetime.fromtimestamp(
                r.json()['result']['expiration'] / 1000.0), fg="green")
            return r.json()
    except requests.exceptions.Timeout:
        click.secho("Request timed out!", fg="bright_red")
        return None
    except Exception as ex:
        click.secho("Unexpected error: %s" % str(ex), fg="bright_red")
        return None

The following function is a wrapper function that first checks if the previously fetched token is still valid. If yes, it uses it, else gets a new token.

def get_token(rest_api_host, api_key, tenant_id):
    """Fetch API token and save it locally till it expires"""
    try:
        if os.path.exists(TOKEN_FILE):
            with open(TOKEN_FILE, 'r') as token_file:
                stored_token = json.load(token_file)
                token_expiry = datetime.utcfromtimestamp(stored_token['result']['expiration']/1000.0)
                if datetime.utcnow() < token_expiry:
                    click.secho('Token is valid, re-using it', fg='green')
                    return stored_token['result']['token']
                else:
                    # Token expired, get a fresh token
                    click.secho('Token expired, getting a fresh token', fg='red')
                    token_response = si_rest.get_token(rest_api_host, api_key, tenant_id)
                    if token_response:
                        with open(TOKEN_FILE, 'w') as token_file_w:
                            json.dump(token_response, token_file_w)
                        return token_response['result']['token']
                    else:
                        return None
        else:
            # fetch token for the first time
            click.secho('Token not found, fetching token', fg='blue')
            token_response = si_rest.get_token(rest_api_host, api_key, tenant_id)
            if token_response:
                with open(TOKEN_FILE, 'w') as token_file:
                    json.dump(token_response, token_file)
                return token_response['result']['token']
            else:
                return None
    except Exception as ex:
        click.secho("Unexpected error: %s" % str(ex), fg="bright_red")

This function can be called to get a token that can be passed in the header as shown below.

        try:
            rest_api_host, api_key, tenant_id = si_rest.setup()
            token = get_token(rest_api_host, api_key, tenant_id)
            if token is None:
                raise Exception('Token creation failed')
            storage_systems = si_rest.list_storage_systems(rest_api_host, token, tenant_id, args.system_type)
            click.secho('Found (%d) storage systems' % len(storage_systems['data']), fg='green')
        except Exception as ex:
            click.secho("Unexpected error: %s" % str(ex), fg="bright_red")

The code is available on GitHub.

REST API Specification

A developer who wants to create a new REST API needs to specify the REST API so as to reach a common understanding with all the stakeholders. A REST API specification describes the API and the different aspects of the API as illustrated below.

Resource Name:  SI Alert

{
    "alert": {
        "source": {
            "systemName" : "2107.75KMP11",	
            "systemModel" : "998",
            "systemType" : "5331",
            "systemSerial" : "75KMP11"
        },
        "method": "FCM4"
    },
    "acked": {
        "by": "john.doe@ibm.com",
        "timeInMs": 1704346271967,
        "action": "ACKNOWLEDGE",
        "details": {}
    }
}

Success Response 202 Accepted

{
    "result": {},
    "metadata": {
        "rc": "success",
        "input": {},
        "took": 3350
    }
}

Failure Response 400 Bad Request

{
    "metadata": {
        "rc": "error",
        "message": "Alert not found",
        "type": "WebApplicationException"
    }
}

Query Parameters

Result Codes

A clear specification helps the caller to code correctly handling any errors that might be returned when calling the API.

#IBMStorageInsights

#RESTAPI

#ibmstorageinsightsRESTAPI