Storage

View Only

The new IBM Storage Archive Enterprise Edition REST API

By Nils Haustein posted 21 days ago

Authored by Nils Haustein

In this blog article I share my experience installing, configuring, and using the new IBM Storage Archive Enterprise Edition (Storage Archive EE) REST API.

Introduction

IBM Storage Archive EE provides a REST API. With Storage Archive EE version 1.3.5.0 the REST API allows managing resources (nodes, drives, tapes, libraries) and controlling file migration and recall [1].

In this article I first show how to install and configure the REST API. Next, I explain the API user concept. Finally, I demonstrate how to use the Storage Archive EE REST API.

I installed and configured the IBM Storage Archive REST API with Storage Archive version 1.3.5.0, RHEL 8.8 and IBM Storage Scale version 5.1.9.3. Furthermore, I had Python 3.8 installed, aside of Python 3.6.

Installation and configuration

The essential steps to install and configure the Storage Archive EE REST API are outlined in IBM Documentation [2]. The REST API is provided by an RPM located in ~/rpm/RHEL8/ibmsa-rest*.rpm.

Prior to installation of the Storage Archive EE REST API package, prerequisite software must be installed [3]. Some of the required software can be installed as RPM-package and some of the software must be installed as python module using pip3.8. The prerequisite software and the installation commands I used are listed below:

Apache httpd: dnf install http
Python3.8: dnf install python3.8
Python3.8 mod_wsgi: dnf install python38-mod_wsgi
mod_ssl (for https): dnf install mod_ssl
Python3.8 cryptography: dnf install python38-cryptography.x86_64
Flask and flask-jwt-extended: pip3.8 install flask flask-jwt-extended

Note, I purposely installed a separate python3.8 environment on my Storage Archive EE server because the python3 environment was used by Storage Scale.

To install the Storage Archive REST API package, I set the python3.8 environment and installed the RPM:

# cd ~/rpm/RHEL8
# export PYTHON_BIN="/usr/bin/python3.8"
# dnf install ibmsa-rest-1.3.5.0-53242-RHEL8.x86_64.rpm

Now, the Storage Archive REST API is installed:

# rpm -qa | grep ibmsa-rest
ibmsa-rest-1.3.5.0-53242.x86_64

Configure the REST API

The Storage Archive EE REST API web-service is provided by the Apache httpd-service. The httpd-service configuration can be adjusted in configuration file /etc/httpd/conf.d/ibmsa-rest-httpd.conf that is provided by the Storage Archive EE REST API [2].

To use the REST API with http on the default http-port 7100, no further adjustments on the configuration file are needed.

The httpd-service runs under the user apache that was created automatically during the installation of the httpd-package. To grant this user required permissions, run the following script:

# /opt/ibm/ltfsee/rest/conf/apache-config.sh

Make sure the permissions for directory /opt/ibm/ltfsee/rest/uploads/ are set for the apache user:

drwxr-xr-x 2 apache root 58 Jan 31 12:07 uploads

If this is not the case, then set the permission of the uploads directory for the apache user:

# chown apache:root /opt/ibm/ltfsee/rest/uploads

Certain security and logging options can be configured in file /opt/ibm/ltfsee/rest/conf/config.ini. This includes the following setting:

Token expiration time (default is 8 h)
Upload path for the httpd server (default is /opt/ibm/ltfsee/rest/uploads)
CSRF (Cross Site Request Forgery) protection (default is true)
Log level (default is INFO, set to DEBUG when debugging issues)
Secret for symmetric key (authentication)
Private and public keys for symmetric encryption (SSL)
Key algorithms

I just reviewed this configuration file and did not make any changes. To edit this file, run:

# vi /opt/ibm/ltfsee/rest/conf/config.ini

If you want to use http, then go to section Start the REST API service. If you want to configure https, then follow the next section Enable SSL (https).

Enable SSL (https)

To enable SSL for the use with https, make sure that the mod_ssl package was installed. To use SSL a key-file and a certificate-file must be available. Both files must be accessible by the user apache. This user was created during the installation of the httpd package.

In my example the server certificate (server-cert.pem) and the server key (server-key.pem) are stored in /opt/ibm/ltfsee/rest/conf/.

To enable SSL, the file /etc/httpd/conf.d/ibmsa-rest-httpd.conf must be adjusted. Find below a snippet of this file setting the https-port to 7143:

…
# set https port
Listen 7143 https

WSGIRestrictEmbedded On
# virtual host definition for https port
<VirtualHost _default_:7143>

# provide a fully qualified server name
ServerName eenode.mydomain.com
DocumentRoot "/opt/ibm/ltfsee/rest"
# set SSL on and provide paths to key and certificate
SSLEngine on
SSLCertificateFile /opt/ibm/ltfsee/rest/conf/server-cert.pem
SSLCertificateKeyFile /opt/ibm/ltfsee/rest/conf/server-key.pem
…

These changes are on top of the file ibmsa-rest-httpd.conf, no further changes are required

The REST API service is configured, now let’s start it.

Start the REST API service

The Storage Archive EE REST API service is provided by httpd. To start httpd-service:

# systemctl start httpd

Check the journal of the httpd-service:

# journalctl -u httpd

Check the ports the httpd-service is listening at:

# netstat -antp | grep httpd

If the service is also listening on port 443, the open the file /etc/httpd/conf.d/ssl.conf and hash the line:

# Listen 443 https

The log-file of the Storage Archive REST API is located in /var/log/ltfsee_rest/rest_app.log. It contains information about API requests.

Using the REST API

In this chapter I explain the authentication concepts of the Storage Archive REST API and demonstrate how to view information and manage resources.

User Authentication

To use the Storage Archive EE REST API services, authentication is required. There are two alternate ways to authenticate with the REST API [4]:

Username and password of a user registered in the Storage Archive EE node hosting the REST API
Private key stored in the Storage Archive EE node hosting the REST API in combination with key passphrase.

The authentication methods are explained below.

Authentication with username

To authenticate with username and password, the user must be registered in the operating system of the Storage Archive EE node that runs the REST API. The following command creates user restadmin in group users:

# useradd -u 1002 -g users -s /sbin/nologin -m restadmin

Any user registered in the operating system of the REST API node can view information. However, to manage resources, the user needs sudo permissions to run the eeadm-command. To allow user restadmin to manage resources, add the following statement to the sudoer file:

#### REST API allow restadmin to run eeadm
restadmin ALL=(ALL) NOPASSWD: /opt/ibm/ltfsee/bin/eeadm

To authenticate with username and password, use the following http-endpoint:

# curl -X POST http://localhost:7100/ibmsa/v2/login -c cookies.txt
-H 'Content-Type: application/json'
-d '{"username":"restadmin", "password":"*****"}'

Response:
{"Logged in":true,"status":"success"}

When SSL is used, then then the certificate must be provided. In the example below a self-created CA stored in /ca-path/ca-cert.pem is used:

# curl -X POST https://localhost:7143/ibmsa/v2/login -c cookies.txt
-H 'Content-Type: application/json'
-d '{"username":"restadmin", "password":"Passw0rd"}'
--cacert /ca-path/ca-cert.pem

Response:
{"Logged in":true,"status":"success"}

After successful login there is a file named cookies.txt stored in the local directory of the curl-client. This cookies-file contains the API token and is used for subsequent API request.

Authentication with private key and passphrase

Authentication with private key and passphrase allow viewing information as well as managing resources with the Storage Archive EE REST API. To authenticate with a private key and passphrase, the key with passphrase must be created on the Storage Archive EE node running the REST API. The following example creates a key with passphrase:

# ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): apikey <-- enter file name
Enter passphrase (empty for no passphrase): ******* <-- enter passphrase
Enter same passphrase again: ******* <-- enter passphrase again
Your identification has been saved in apikey.
Your public key has been saved in apikey.pub.

…

The key pair is stored in the local directory of the Storage Archive EE node:

# ls -l

-rw------- 1 root root 2655 Feb 4 09:47 apikey
-rw-r--r-- 1 root root 568 Feb 4 09:47 apikey.pub

For authentication with key and passphrase the full path and filename of the key is required (in this example it is /path-to-key/apikey. The key file must exist in the Storage Archive EE node, not in the client where the curl-command is executed. Here is an example for authentication with key and passphrase:

# curl -X POST 'http://localhost:7100/ibmsa/v2/login' -c cookies.txt
-H 'Content-Type: application/json'
-d '{"key":"/path-to-key/apikey", "passphrase":"******"}'

When SSL is used, then then the certificate must be provided. In the example below a self-created CA stored in /ca-path/ca-cert.pem is used:

# curl -X POST 'https://localhost:7143/ibmsa/v2/login' -c cookies.txt
-H 'Content-Type: application/json'
-d '{"key":"/root/silo/projects/restApi/key/apikey", "passphrase":"Passw0rd"}' --cacert /ca-path/ca-cert.pem

After successful login there is a file named cookies.txt stored in the local directory of the curl-client. This cookies-file contains the API token and is used for subsequent API request.

Viewing information

The Storage Archive EE REST API provides various endpoints to view information [5]. To view information about nodes, use the following command:

# curl -X GET 'http://localhost:7100/ibmsa/v2/drives' -b cookies.txt

Using https, the request looks like this:

# curl -X GET 'https://localhost:7143/ibmsa/v2/drives' -b cookies.txt
--cacert /ca-path/ca-cert.pem

The information is returned in JSON format. To better read the JSON-output the jq-tool can be used. This tool can be installed from a RHEL repository. Here is an example using jq:

# curl -X GET 'https://localhost:7143/ibmsa/v2/drives' -b cookies.txt
--cacert /ca-path/ca-cert.pem

Response:
[

{

    "address": 259,
    "host_device_name": "/dev/sg12",
    "host_scsi_address": "5.0.2.0",
    "id": "000782B61C",
    "library_id": "92398dfa-84b8-42c5-bfa7-80e25c747cdf",
    "library_name": "lib1",
    "node_hostname": "archnode1",
    "node_id": 1,
    "nodegroup_name": "G0",
    "role": "mrg",
    "scsi_firmware_revision": "R3G0",
    "scsi_product_id": "ULT3580-TD9",
    "scsi_vendor_id": "IBM",
    "state": "not_mounted",
    "status": "ok",
    "tape_barcode": "",
    "task_id": "",
    "type": "LTO9"

…

]

Filtering information

Information can be filtered using the URL modifier pretty, fields and sort [6]. For example, to show only fields id, status, state and tape_barcode in a pretty format, the following modifier can be used:

# curl -X GET 'https://localhost:7143/ibmsa/v2/drives?pretty&fields=id,status,state,tape_barcode' -b cookies.txt
--cacert /ca-cert/ca-cert.pem

Response:
[

{

    "id": "000782B61C",
    "status": "ok",
    "state": "not_mounted",
    "tape_barcode": ""

….

]

The jq-tool allows more advanced filtering and output options. For example, the output of the tape information can be converted in csv-format:

Response:
L90000L9,ok,appendable,pool1
L90001L9,ok,appendable,pool2
L90002L9,ok,appendable,pool3
L90004L9,ok,unassigned,
L90005L9,ok, unassigned

View file state information

One important endpoint is the file state endpoint. This endpoint allows to inquire the migration state of a single file:

# curl -X GET 'https://localhost:7143/ibmsa/v2/files/state?pretty&file=/ibm/archfs/test2/file_0.pdf' -b cookies.txt
--cacert/root/silo/projects/restApi/tls/ca-cert.pem

Response:

{
"Name": "/ibm/archfs/test2/file_0.pdf",
"State": "resident"
}

Note, in the current version of the REST API the file state endpoint requires all directories in the space managed file system to have execute permission for other (permission bit xx1). This is a known issue and may get improved in a future release.

In the example above, the directory /ibm/archfs/test2/ must have execute permission for others. In the example below the subdirectory test2 has permission 751 for username user in group users:

# ls -l /ibm/archfs/

drwxr-x--x 2 user users 4096 Feb 4 10:40 test2

The files itself do not need to have access permission for others, only the leading directory chain needs this permission.

Managing resources

The Storage Archive EE REST API provides various endpoints to manage resources [1]. This includes endpoints to manage the hardware such as library, nodes, and tapes, as well as endpoints to migrate and recall files.

Note: Managing resources requires the user that authenticated with the REST API to have permission to run the eeadm-command (see section Authentication with username). When authenticating with private key and passphrase, then managing resources is allowed.

The endpoints for managing resources require the CSRF token in header X-CSRF-TOKEN. After successful authentication with the REST API, the CSRF token is stored in the cookies-file:

# cat cookies.txt

# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
localhost FALSE / FALSE 0 csrf_access_token cc8a29b9-6014-41c9-a412-2ea676f7878e
#HttpOnly_localhost FALSE / FALSE 0 access_token_cookie eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.....

Remember the CRSF token from the cookies-files when migrating and recalling files.

Migration and recall

Let’s look at a simple scenario for migrating and recalling files. We have some files stored in the space managed file system of the Storage Archive EE node:

$ ls -l /ibm/archfs/test2/

-rw-r--r-- 1 user users 991232 Feb 4 10:40 file_0.pdf
-rw-r--r-- 1 user users 784384 Feb 4 10:40 file_1.pdf
-rw-r--r-- 1 user users 131072 Feb 4 10:40 file_2.pdf
-rw-r--r-- 1 user users 280576 Feb 4 10:40 file_3.pdf
-rw-r--r-- 1 user users 644096 Feb 4 10:40 file_4.pdf

Logging into the REST API via https as user restadmin:

# curl -X POST https://localhost:7143/ibmsa/v2/login -c cookies.txt
-H 'Content-Type: application/json'
-d '{"username":"restadmin", "password":"Passw0rd"}'
--cacert /ca-cert/ca-cert.pem

Response:
{"Logged in":true,"status":"success"}

The CSRF token in the file cookies.txt is: cc8a29b9-6014-41c9-a412-2ea676f7878e

First, we check the state of the files in the folder /ibm/archfs/test2 using a for-loop:

# for i in $(seq 0 4); do curl -X GET "https://localhost:7143/ibmsa/v2/files/state?pretty&file=/ibm/archfs/test2/file_$i.pdf"
-b cookies.txt --cacert /ca-cert/ca-cert.pem; done

{
"Name": "/ibm/archfs/test2/file_0.pdf",
"State": "resident"
}{
"Name": "/ibm/archfs/test2/file_1.pdf",
"State": "resident"
}{
"Name": "/ibm/archfs/test2/file_2.pdf",
"State": "resident"
}{
"Name": "/ibm/archfs/test2/file_3.pdf",
"State": "resident"
}{
"Name": "/ibm/archfs/test2/file_4.pdf",
"State": "resident"
}

Next, we migrate the files. For this purpose, we must enter the fully qualified names of the file into a file list (file-demo1.lst):

# cat file-demo1.lst

/ibm/archfs/test2/file_3.pdf
/ibm/archfs/test2/file_0.pdf
/ibm/archfs/test2/file_2.pdf
/ibm/archfs/test2/file_1.pdf
/ibm/archfs/test2/file_4.pdf

To migrate the files in the file list to the existing tape pool pool1, use the following command:

# curl -X POST 'https://localhost:7143/ibmsa/v2/migrate'
-F "file=@file-demo1.lst" -F "pool=pool1" -b cookies.txt
-H "Content-Type: multipart/form-data"
-H "X-CSRF-TOKEN: cc8a29b9-6014-41c9-a412-2ea676f7878e"
--cacert /ca-path/ca-cert.pem

Response:
{"status":"accepted","task_id":4934}

The migrate command started task 4934. To inquire the task status, use the following command:

# curl -X GET 'https://localhost:7143/ibmsa/v2/task/show/4934' -b cookies.txt
--cacert /ca-cert/ca-cert.pem | jq .

Response:
[
{
    "cmd_param": "/opt/ibm/ltfsee/bin/eeadm migrate /opt/ibm/ltfsee/rest/uploads/file-demo1.lst -p pool1",
    "completed_time": "2025-02-04T09:59:07.826Z",
    "created_time": "2025-02-04T09:58:30.554Z",
    "file_amount": 2831360,
    "file_count": 5,
    "id": "4934@2025-02-04T09:58:30.554Z",
    "inuse_drives": [],
    "inuse_libs": [],
    "inuse_node_groups": [],
    "inuse_pools": [],
    "inuse_tapes": [],
    "result": "succeeded",
    "started_time": "2025-02-04T09:58:30.556Z",
    "status": "completed",
    "task_id": 4934,
    "type": "migrate"
}

]

As shown above, the task completed. Let’s check the file states:

# for i in $(seq 0 4); do curl -X GET "https://localhost:7143/ibmsa/v2/files/state?pretty&file=/ibm/archfs/test2/file_$i.pdf"
-b cookies.txt --cacert /ca-cert/ca-cert.pem; done

{
"ID": "8642704550425025024-3560302634690375993-1353024093-44810-0",
"Name": "/ibm/archfs/test2/file_0.pdf",
"Replicas": "1",
"State": "migrated",
"Tape 1": "L90000L9@pool1@lib1 (tape state=appendable)"
}{

…

}{
"ID": "8642704550425025024-3560302634690375993-1349924204-44814-0",
"Name": "/ibm/archfs/test2/file_4.pdf",
"Replicas": "1",
"State": "migrated",
"Tape 1": "L90000L9@pool1@lib1 (tape state=appendable)"
}

All files were migrated. We can now recall the files:

# curl -X POST 'https://localhost:7143/ibmsa/v2/recall'
-F "file=@file-demo1.lst" -b cookies.txt
-H "Content-Type: multipart/form-data"
-H "X-CSRF-TOKEN: cc8a29b9-6014-41c9-a412-2ea676f7878e"
--cacert /ca-cert/ca-cert.pem

Response:
{"status":"accepted","task_id":4936}

The recall command started task 4936. To inquire the task status, use the following command:

# curl -X GET 'https://localhost:7143/ibmsa/v2/task/show/4936' -b cookies.txt
--cacert /ca-cert/ca-cert.pem | jq .

Response:
[

{
"cmd_param": "/opt/ibm/ltfsee/bin/eeadm recall /opt/ibm/ltfsee/rest/uploads/file-demo1.lst",

    "completed_time": "2025-02-04T10:02:20.778Z",
    "created_time": "2025-02-04T10:02:08.468Z",
    "file_amount": 2831360,
    "file_count": 5,
    …
    "result": "succeeded",
    "started_time": "2025-02-04T10:02:08.470Z",
    "status": "completed",
    "task_id": 4936,
    "type": "selective_recall"
}

]

The recall task completed, let’s check the file state again:

# for i in $(seq 0 4); do curl -X GET "https://localhost:7143/ibmsa/v2/files/state?pretty&file=/ibm/archfs/test2/file_$i.pdf"
-b cookies.txt --cacert /ca-cert/ca-cert.pem; done

{
"ID": "8642704550425025024-3560302634690375993-1353024093-44810-0",
"Name": "/ibm/archfs/test2/file_0.pdf",
"Replicas": "1",
"State": "premigrated",
"Tape 1": "L90000L9@pool1@lib1 (tape state=appendable)"
}{

…

}{
"ID": "8642704550425025024-3560302634690375993-1349924204-44814-0",
"Name": "/ibm/archfs/test2/file_4.pdf",
"Replicas": "1",
"State": "premigrated",
"Tape 1": "L90000L9@pool1@lib1 (tape state=appendable)"
}

As shown above all files are in status premigrated.

Conclusion

From my perspective the IBM Storage Archive EE REST API became very useful and functional. The endpoints to view information can easily be integrated with infrastructure monitoring tools to show the status of the tape resource, including tapes, libraries, tape drives and nodes. The view information endpoints also allow to view the migration state of files which is important for end users and applications when accessing data.

The endpoints for managing resources allow management of the tape resources and operations. Most importantly, are the endpoints to control migration and recalls. With these endpoints, space management operations (migration and recalls) can be automated. Results of these operations can easily be examined by showing the result of the associated operational tasks.
Furthermore, tapes can be assigned to pools, tapes can be reconciled and reclaimed, and tapes can be validated.

The Storage Archive EE REST API also addresses security aspects. It allows SSL protected communication (via https) between the client and REST API server. The user concept allows users to be configured in the operating system of the REST API server that can only view information and users that can leverage all functions of the API. Furthermore, CSRF tokens (Cross Site Request Forgery) helps to prevent CSRF attacks.

During my tests, I was not able to configure both http and https to work. Once I figure out how to do this, I will update this article.

Resources

[1] Managing Storage Archive resources:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=edition-how-use-rest-api-manage-resources

[2] Installing and configuring REST API:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=edition-installing-setting-up-rest-api

[3] List of required software:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=requirements-required-software-linux-systems#ltfs_ee_sw_req

[4] REST API authentication:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=edition-rest-api-authentication

[5] API endpoints for viewing information:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=edition-how-use-rest-api-request-information

[6] Filtering Information:

https://www.ibm.com/docs/en/storage-archive-ee/1.3.5?topic=information-common-get-parameters

#ibmtechxchange-ai

0 comments

10 views

IBM Storage

The online community where IBM Storage users meet, share, discuss, and learn.

Storage

The new IBM Storage Archive Enterprise Edition REST API

By Nils Haustein posted 21 days ago

Introduction

Installation and configuration

Configure the REST API

Enable SSL (https)

Start the REST API service

Using the REST API

User Authentication

Authentication with username

Authentication with private key and passphrase

Viewing information

Filtering information

View file state information

Managing resources

Migration and recall

Conclusion

Resources

Permalink

Additional
Resources

Office

Quick Links

IBM Storage

The online community where IBM Storage users meet, share, discuss, and learn.

Storage

The new IBM Storage Archive Enterprise Edition REST API

By Nils Haustein posted 21 days ago

Introduction

Installation and configuration

Configure the REST API

Enable SSL (https)

Start the REST API service

Using the REST API

User Authentication

Authentication with username

Authentication with private key and passphrase

Viewing information

Filtering information

View file state information

Managing resources

Migration and recall

Conclusion

Resources

Permalink

Additional Resources

Office

Quick Links

Additional
Resources