Introduction
gsutil is a python based command-line tool to access google cloud storage. One can perform wide range of bucket and object management tasks using gsutil, including:
- Creating and deleting buckets.
- Uploading, downloading, and deleting objects.
- Listing buckets and objects.
- Moving, copying, and renaming objects.
- Editing object and bucket ACLs.
This document describes installation and configuration of gsutil on AIX in detail. The document describes different dependencies needed to install gsutil. Also, some of the examples of accessing google cloud storage from AIX using gsutil are added in the document. gsutil is a python based hence you need to install latest python on AIX. You can use yum to install python on AIX or you can install it directly from AIX toolbox .
In the following sections step by step yum and gsutil installation is described.
At the end of the document, there are examples to use gsutil for accessing google cloud storage.
Python Installation
The best way to install any packages from AIX toolbox is YUM. Install yum on AIX by running yum.sh script. As part of yum installation python will get installed. Update all the packages to the latest level using ‘yum update’ and it will update to latest python. Make sure you have python version 2.17.15 or later from AIX toolbox.
To install YUM on AIX using yum.sh, download yum.sh to AIX system and run it as root user.
|
# ./yum.sh
Trying to download rpm.rte & yum_bundle.tar ...
Installing rpm.rte to latest version ...
Might take minutes depending on the number of rpms installed...
+-----------------------------------------------------------------------------+ Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done Verifying requisites...done Results...
….
…. … yum-metadata-parser-1.1.4-2 will be installed ....
Installing the packages.
5:yum-metadata-parser-1.1.4-2 ################################# [ 29%]
6:gdbm-1.8.3-5 ################################# [ 35%]
7:db-4.8.24-3 ################################# [ 41%]
8:python-2.7.10-1 ################################# [ 47%]
9:pysqlite-1.1.7-2 ################################# [ 53%]
10:python-iniparse-0.4-1 ################################# [ 59%]
11:ca-certificates-2016.10.7-2 ################################# [ 65%]
12:curl-7.52.1-1 ################################# [ 71%]
13:python-pycurl-7.19.3-1 ################################# [ 76%]
14:python-urlgrabber-3.10.1-1 ################################# [ 82%]
15:yum-3.4.3-5 ################################# [ 88%]
16:python-devel-2.7.10-1 ################################# [ 94%] 17:python-tools-2.7.10-1 ################################# [100%] Yum installed Successfully....
#
|
Once yum is installed successfully then update the installed packages to latest level. Run 'yum update' to update packages to the latest level.
|
# yum update
AIX_Toolbox | 2.9 kB
00:00:00
AIX_Toolbox/primary_db | 1.0 MB
00:00:00
AIX_Toolbox_72 | 2.9 kB
00:00:00
AIX_Toolbox_72/primary_db | 20 kB
00:00:00
AIX_Toolbox_noarch | 2.9 kB
00:00:00
AIX_Toolbox_noarch/primary_db | 51 kB
00:00:00
Setting up Update Process …..
……
……
……
Dependency Installed:
bash.ppc 0:4.4-3 bzip2.ppc 0:1.0.6-2 cyrus-sasl.ppc 0:2.1.26-2 expat.ppc
0:2.2.4-1 fontconfig.ppc 0:2.11.95-3 freetype2.ppc 0:2.8-1 gcc.ppc 0:8.1.0-1 gcc-cpp.ppc 0:8.1.0-1 gmp.ppc 0:6.1.2-1 info.ppc 0:6.3-1 libXft.ppc 0:2.3.2-4waixX11 libXrender.ppc 0:0.9.8-2waixX11 libffi.ppc 0:3.2.1-2 libgcc.ppc 0:8.1.0-1 libmpc.ppc 0:1.0.3-1 libssh2.ppc 0:1.8.0-2
libstdc++.ppc 0:8.1.0-1 mpfr.ppc 0:3.1.2-3 ncurses.ppc 0:6.1-1 openldap.ppc 0:2.4.45-2 zlib.ppc 0:1.2.11-1
Updated:
ca-certificates.ppc 0:2017.07.17-1 curl.ppc 0:7.60.0-1 db.ppc 0:6.2.32-1 expect.ppc 0:5.45-3 gdbm.ppc 0:1.12-1 gettext.ppc 0:0.19.8.1-1 glib2.ppc
0:2.56.1-1 pysqlite.ppc 0:2.8.3-1 python.ppc 0:2.7.15-1 python-devel.ppc 0:2.7.15-1 python-pycurl.ppc 0:7.43.01 python-tools.ppc 0:2.7.15-1 readline.ppc 0:7.0-5 sqlite.ppc 0:3.23.0-1 tcl.ppc
0:8.6.6-2 tk.ppc 0:8.6.6-1
yum.noarch 0:3.4.3-6
Complete! #
|
gsutil Installation
There are two methods to install gsutil on AIX
1. Installing from tar or zip archive
2. Installing from the Python package index (PyPI)
1. Installing from tar or zip archive
i. Download gsutil as gsutil.tar.gz or as gsutil.zip .
ii. Extract the archive files tar xfz gsutil.tar.gz -C <DIR>
iii. Export the path export PATH=${PATH}:<DIR>/gsutil
After this please goto gsutil config and usage part of the document.
2. Installing from the Python package index (PyPI)
To install gsutil on AIX, one can use AIX toolbox provided python-pip.
Install latest python-pip from AIX toolbox on your AIX system. Once python-pip is installed, using ‘pip install gsutil’ one can install gsutil.
After system is updated to latest packages install python-pip using yum
|
# yum install python-pip
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-pip.noarch 0:10.0.1-1 will be installed …….
.....
.....
Total
Installing : python-setuptools-0.9.8-2.noarch
1/2
Installing : python-pip-10.0.1-1.noarch
2/2
Installed:
python-pip.noarch 0:10.0.1-1
Dependency Installed:
python-setuptools.noarch 0:0.9.8-2
Complete!
#
|
Using pip, one can install gsutil. pip also installs all the dependent python-modules that are needed for gsutil. Some of these python modules require setuptools, (a python module) and the libffi development package. Hence, first update setuptool and install libffi-devel package.
|
# pip install --upgrade setuptools
Collecting setuptools Downloading
https://files.pythonhosted.org/packages/ff/f4/385715ccc461885f3cedf57a41ae3c12b5fec
3f35cce4c8706b1a112a133/setuptools-40.0.0-py2.py3-none-any.whl (567kB)
100% |################################| 573kB 11.5MB/s
Installing collected packages: setuptools
Found existing installation: setuptools 0.9.8
Uninstalling setuptools-0.9.8:
Successfully uninstalled setuptools-0.9.8
Successfully installed setuptools-40.0.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
#
# yum install libffi-devel
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package libffi-devel.ppc 0:3.2.1-2 will be installed
--> Processing Dependency: pkg-config for package: libffi-devel-3.2.1-2.ppc
--> Running transaction check
---> Package pkg-config.ppc 0:0.29.2-1 will be installed
--> Finished Dependency Resolution
……
……
……
……
Installed: libffi-devel.ppc 0:3.2.1-2
Dependency Installed:
pkg-config.ppc 0:0.29.2-1
Complete!
#
|
After above AIX system is ready for installation of gsutil using pip.
Run ‘pip install gsutil’ to install gsutil on AIX system.
|
# pip install gsutil
Collecting gsutil
Using cached
https://files.pythonhosted.org/packages/48/93/d298fb2d4c5969f08e369989f7c2e898590
45076652fa8aac797be732aba/gsutil-4.33.tar.gz
Requirement already satisfied: argcomplete>=1.9.4 in /opt/freeware/lib/python2.7/sitepackages (from gsutil) (1.9.4)
Requirement already satisfied: boto==2.48.0 in /opt/freeware/lib/python2.7/site-packages (from gsutil) (2.48.0)
Requirement already satisfied: crcmod>=1.7 in /opt/freeware/lib/python2.7/site-packages (from gsutil) (1.7)
Requirement already satisfied: fasteners>=0.14.1 in /opt/freeware/lib/python2.7/sitepackages (from gsutil) (0.14.1)
Collecting gcs-oauth2-boto-plugin>=2.1 (from gsutil)
Using cached
https://files.pythonhosted.org/packages/25/05/bc27be141e2c6bd5299558dc813fd6d3e0c
0674efb7a8870d6b8363cc2ca/gcs-oauth2-boto-plugin-2.1.tar.gz
Collecting google-apitools>=0.5.22 (from gsutil)
Using cached
https://files.pythonhosted.org/packages/eb/ec/3bc65e3d5f1df755d99485ba39a1aeb15f5
68f1e37b35405f4d5ff3977e4/google_apitools-0.5.23-py2-none-any.whl
Requirement already satisfied: httplib2>=0.11.3 in /opt/freeware/lib/python2.7/sitepackages (from gsutil) (0.11.3)
Requirement already satisfied: google-reauth>=0.1.0 in /opt/freeware/lib/python2.7/sitepackages (from gsutil) (0.1.0) Collecting mock==2.0.0 (from gsutil)
Using cached
https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076ce e4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl
Requirement already satisfied: monotonic>=1.4 in /opt/freeware/lib/python2.7/sitepackages (from gsutil) (1.5)
……
……
……
Successfully installed SocksiPy-branch-1.1 cffi-1.11.5 cryptography-2.3 enum34-1.1.6 funcsigs-1.0.2 gcs-oauth2-boto-plugin-2.1 google-apitools-0.5.23 gsutil-4.33 ipaddress1.0.22 mock-2.0.0 pbr-4.2.0 pyOpenSSL-18.0.0 python-gflags-3.1.2 retry-decorator-1.1.0 You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
#
|
gsutil config and usage
Now gsutil is install on AIX system. gsutil will get installed in <DIR>/gsutil or in
/opt/freeware/bin if using pip, make sure PATH set accordingly before using the gsutil.
|
# gsutil --version
gsutil version: 4.33
#
# gsutil --help
Usage: gsutil [-D] [-DD] [-h header]... [-m] [-o] [-q] [command [opts...] args...] Available commands:
acl Get, set, or change bucket and/or object ACLs
cat Concatenate object content to stdout
compose Concatenate a sequence of objects into a new composite object.
config Obtain credentials and create configuration file
cors Get or set a CORS JSON document for one or more buckets
cp Copy files and objects
defacl Get, set, or change default ACL on buckets
defstorageclass Get or set the default storage class on buckets
du Display object size usage
hash Calculate file hashes
help Get help about commands and topics
iam Get, set, or change bucket and/or object IAM permissions.
kms Configure Cloud KMS encryption
label Get, set, or change the label configuration of a bucket.
lifecycle Get or set lifecycle configuration for a bucket
logging Configure or retrieve logging on buckets
….
….
….
support Google Cloud Storage Support
throttling Throttling gsutil
versions Object Versioning and Concurrency Control wildcards Wildcard Names
Use gsutil help <command or topic> for detailed help.
#
|
Now we are all set to use gsutil to access google cloud storage. To access google cloud storage using gsutil first configure project and user for google cloud storage. Once project and user are configured then one can start working on the buckets and objects. In the below example, we will perform following operations 1. Configure the project and user
2. List project to see all buckets.
3. List bucket to see all files.
4. Copied a file into a bucket.
5. List bucket to see copied file.
6. Cat the file to see content
7. Removed file from bucket.
8. List bucket to see file is removed
To configure project and user run ‘gsutil config’ command. This command will provide a URL as shown in screen shot. Copy this URL and open in a browser on a PC. Sign in to your google account and you will get the authorization key as shown in the browser screen shot. Copy the authorization key and paste after “Enter the authorization code: “
Once authorization key is accepted, browse your google cloud project in browser and get the project id. Once enter project id in gsutil configuration, gsutil is configured to work on the project.
|
Enter the authorization code: <My authorization code>
Please navigate your browser to https://cloud.google.com/console#/project, then find the project you will use, and copy the Project ID string from the second column. Older projects do not have Project ID strings. For such projects, click the project and then copy the Project Number listed under that project.
What is your project-id? <My project id >
Boto config file "/.boto" created. If you need to use a proxy to access the Internet please see the instructions in that file. #
# gsutil ls
gs://artifacts.<My project id>.appspot.com/ gs://master-test-bucket/
#
|
Once gsutil is configured with user and project id, one can start working on the buckets and objects.
|
#
# gsutil ls
gs://artifacts.<My project id>.appspot.com/
gs://master-test-bucket/
#
# gsutil ls gs://master-test-bucket/
gs://master-test-bucket/hello.txt
#
# gsutil cp gsutil_test.txt gs://master-test-bucket/
Copying file://gsutil_test.txt [Content-Type=text/plain]...
/ [1 files][ 22.0 B/ 22.0 B]
Operation completed over 1 objects/22.0 B. #
# gsutil ls gs://master-test-bucket/
gs://master-test-bucket/gsutil_test.txt
gs://master-test-bucket/hello.txt
#
# gsutil cat gs://master-test-bucket/gsutil_test.txt
gsutil works from AIX
#
# gsutil rm gs://master-test-bucket/gsutil_test.txt
Removing gs://master-test-bucket/gsutil_test.txt...
/ [1 objects]
Operation completed over 1 objects
#
# gsutil ls gs://master-test-bucket/
gs://master-test-bucket/hello.txt
#
|
Here is the pdf version of the document Installing_and_Running_gsutil_on_AIX.pdf