Cognos Analytics

 View Only

Cognos Analytics with Jupyter Notebooks – An explicit walkthrough with common pitfalls and work arounds.

By MATT MORGAN posted Thu April 16, 2020 10:27 AM

  

Prerequisites and Supported Software.

The Build: The Cognos Analytics Server (the application tier, content manager, gateway) will need to be on the same build as the Jupyter Server. If there is a discrepancy in the builds, although the Jupyter Server may start up, it is likely one will run into issues when attempting to configure or integrate it with Cognos. Example: Both Cognos and Jupyter need to be on 11.1.5.

The Security: If the Cognos Analytics Server, or gateway (including external load balancers / DNS entries) are SSL enabled, then it is required that the Jupyter Server be SSL enabled. For Jupyter to function as expected, either all components are to be SSL enabled or none of the components are to be SSL enabled.  In an environment where the Gateway and Jupyter server are SSL enabled but the Cognos Analytics (dispatcher) server is not SSL enabled, will not function as expected.

Should you need assistance enabling SSL on the Jupyter server, please see the following how to article in conjunction with this blog:

How to secure Jupyter for Cognos Analytics 11.1.2 + with 3rd party certificates

The OS:  The Jupyter server itself can be installed on either Linux or Windows 10. Please note. Windows Servers are not applicable in this scenario. Confirmation of this can be found in the official documentation here:

Installing IBM Cognos Analytics for Jupyter Notebook Server

Note: Windows 10 is only supported as of 11.1.5 and above

The Docker Version: The versions of docker that are supported are Docker CE for CentOS, Docker CE for Ubuntu, Docker EE for RHEL, Docker for Windows. We will discuss how to download and install these for our purposes on Windows and CentOS.

It is important to note, the higher versions of docker seem to mitigate the most issues (based on recent support cases). Many errors can be over come by being on a more recent version of docker.

Internet Access: Internet access is not required for offline installs. However, if one has access to the box prior to it being taken off the network, one might consider installing troubleshooting tools from the default repos.

Optional tools: The following tools can be installed for ease of use or troubleshooting purposes. I will install them in this case for the purposes of demonstration. These are not IBM tools, not required and should be installed at your own risk and of your own volition.

  • telnet, iputils-ping, nano, x11 tools, an SFTP tool, an SSL tool (if required), any other software you might find helpful

Installation:

Step 1: Download the correct build

Downloading the right build is critical for installation. It is important that the build is identical to that of the Cognos Analytics server (the application tier, content manager, gateway). A Cognos Analytics 11.1.5 server will need a 11.1.5 Jupyter Server. The install will also need to correlate with the OS on which one is installing Jupyter.

The Cognos Jupyter server and the Cognos Analytics server do not need to be on the same operating system.

In our case today, the demonstration will include a REHL 7.7 (x86) - 11.1.5 Jupyter install connected to a Windows Server 11.1.5 Cognos server install. Please note, the majority of the steps are identical regardless of OS. Where there are major differences, I will include caveats for other Operating Systems.

One may verify the part numbers on the download page of your version by expanding the (+):

Downloading IBM Cognos Analytics 11.1.5.0

We will be using the Cognos Analytics installer Microsoft (For Cognos Analytics Server):

IBM Cognos Analytics Installer 2.0.191205 Microsoft Windows Multilingual

CC4Q2ML

Required

 

The Cognos Analytics Server for Windows:

IBM Cognos Analytics Server 11.1.5 Microsoft Windows Multilingual

CC54TML

Required

 

The Cognos Jupyter Notebook for Linux:

IBM Cognos Analytics for Jupyter Notebook 11.1.5 Linux x86-64 Multilingual

CC4QEML

Recommended

 

The Cognos Installer for Linux:

IBM Cognos Analytics Installer 2.0.191205 Linux x86 Multilingual

CC4Q4ML

Required

I installed all things I need for a basic Cognos environment (xclock libraries, text editors, cognos libraries, apache, telnets, etc. (Please install of your own volition)

yum groupinstall x11 fonts -y; \
yum install xclock xorg-x11* \
telnet uputils-ping  \
nano -y

 

I also made changes to the bash_profile so that I could easily ssh and sftp to the servers as I needed.  

Be sure to exit the session for these to take effect.

nano ~/.bash_profile

Then add something to the effect of:

DISPLAY=server1ip:magiccookienumber.0

This can be determined by running an xauth list.


Make the changes for x11 to the sshd file

nano /etc/ssh/sshd_config

Change the following lines:

X11Forwarding yes

X11DisplayOffset 10

X11UseLocalhost no

 

Then restart the service

service sshd restart 

One may have to exit the session and restart the session for this to take effect.

On your local machine download an xwindows client.

You can test this is set up correctly at your own discretion with a non-IBM tool such as:

xclock

If this is not correctly set up. Please work with your system administrator to make sure your xwindows is working.

Step 2: Install Docker (CENT OS – CE)

More information can be found on the docker website listed below. I have included the relevant Cognos steps below.

(**Please note, mixing CENT OS Docker CE and RHEL is unsupported. This is given as an example of steps. You can try similar steps with your licensed Docker for RHEL EE) 

 Get Docker Engine - Community for CentOS

Remove old versions of docker:

yum remove docker \

                  docker-client \

                  docker-client-latest \

                  docker-common \

                  docker-latest \

                  docker-latest-logrotate \

                  docker-logrotate \

                  docker-engine

Install required packages. yum-utils provides the yum-config-manager utility, and device-mapper-persistent-data and lvm2 are required by the devicemapper storage driver. (these should be installed by default so you might already have them)

yum install -y yum-utils   device-mapper-persistent-data   lvm2

Remove any old docker repos:

rm /etc/yum.repos.d/docker*.repo

Set up the stable repository

yum-config-manager \

    --add-repo \

    https://download.docker.com/linux/centos/docker-ce.repo

Install the docker engine

yum install docker-ce docker-ce-cli containerd.io

List and sort the versions available on the repo

yum list docker-ce --showduplicates | sort -r

Run the following command to install a specific version:

yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io

I used the most recent at the time

sudo yum install docker-ce-18.06.3.ce-3.el7 docker-ce-cli-18.06.3.ce-3.el7 containerd.io

 start docker

systemctl start docker

Check that you have the most recent/supported version of docker:

docker version

Now you can start docker:

systemctl start docker

Potential error:

Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Go to the docker.sock file and change the permissions (according to your company’s standards). In house this was resolved by:

cd /var/run/

chmod 666 docker.sock

Step 2(Windows 10) : Install Docker 

Download and install your favorite supported version of docker. I got mine from the Docker website.

Click on “Download From Docker Hub”

Click on “Get Docker”

Run the executable.

Let it download.

Be sure to leave “USE WINDOWS CONTAINERS INSTEAD OF LINUX CONTAINERS” unchecked.



The Cognos Jupyter containers are ubuntu.

You can verify this yourself once this is installed by bashing into the container.  

To bash in: docker exec -it -u 0 ca_jupyter_hub bash
To check OS: cat /etc/os-release

NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="
https://www.ubuntu.com/"
SUPPORT_URL="
https://help.ubuntu.com/"
BUG_REPORT_URL="
https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="
https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

 Let it finish installing.

Launch Docker and make sure it is installed correctly. You should see a little white Docker whale on the bottom corner. If it is red, engage your system admin to help deduce what is wrong.


Note: Some pitfalls here include the fact that nested virtualization may be needed.

You need to make sure that under Task Manager > Performance tab that it indicates Virtualization is Enabled.

In order to assure this, before I started and connected to the VM on the machine hosting the VM, I ran the following commands in powershell:

  1. Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All -Verbose
  2. Enable-WindowsOptionalFeature -Online -FeatureName Containers -All -Verbose
  3. bcdedit /set hypervisorlaunchtype Auto
  4. Set-VMProcessor -VMName <Enter-VM-Name> -ExposeVirtualizationExtensions $true -Verbose
  5. net stop vmms
  6. net start vmms

Step 3: Install Jupyter

Navigate to the installers

cd /tmp/installers

Run the installer

./ ca_instl_lnxi38664_2.0.191205.bin

Select the /zip repository in the installers directory recently created:





Please note, if you receive the following error, you likely do not have enough disk space on the server:

Error processing tar file(exit status 1): write /opt/*****: no space left on device

Around 35 GB of available space is needed for the images once completed, this does not include the intermediate/temporary space that docker requires for various layers during the install. Verify and allocate a minimum of 60 GB of space to the /var/lib/docker/ directory to run the install with no issues.

Step 4(RHEL): Configure Jupyter 

CD to the scripts directory of the install. Because we are on Linux, we will go to the UNIX directory.

cd /opt/ibm/cognos/jupyter/dist/scripts/unix

If you execute an LS here you will see a few scrips and configuration files



build.sh

A script that builds/rebuilds the Jupyter Hub, Jupyter Viewer, and Server Instance docker images

config.conf

A configuration file that is used when building the images. Here is where you may need to define the hostname, port, and certificate locations.

install.sh

A script that installs the Jupyter Hub and Jupyter Viewer containers. This calls the build and startup scripts.

prune.sh

 A script that removes unnecessary/older images

startup.sh

A script that starts the containers.

stop.sh

A script that stops the containers.

uninstall.sh

A script that uninstalls the Jupyter Hub and Jupyter Viewer


Before anything, open the config.conf file and confirm the accuracy of the information presented. For our purposes we can imagine this having 2 important parts. The first part related to the SSL component and the second part related to the Configuration component. While other sections are important, they are note covered in this demonstration.


The SSL component will be discussed in greater detail in the following technote:

How to secure Jupyter for Cognos Analytics 11.1.2 + with 3rd party certificates

The configuration section on RHEL will be comprised of the hostname and port.

1. The hostname should refer to the Jupyter server and the port should be open and accessible.

By default RHEL instances will define the hostname as $(hostname) and this will cause Jupyter to start up using the hostname specified by the machine hosting the install. You can define this yourself or on RHEL you can check what this will resolve to by running:

echo $HOSTNAME



2. The port number is the port with which the Jupyter server will start on and be reachable externally. By default this is 8000.

You will need to be sure that the Cognos Analytics Server and the browser can reach the hostname:port. A simple test would be to go to the Cognos Analytics server (where the dispatcher/application tier is installed) and telnet to the Jupyter server $HOSTNAME and port number you are defining in the config.conf file.

Example: Go to the cognos server and type in the following


Step 4 (Windows) : Configure Jupyter 

Navigate to the install location. You can do this in file explorer or by cd’ing there in a command prompt.  

 

By Default: C:\Program Files\ibm\cognos\jupyter\dist\scripts\windows

Edit the config.conf file making the HOST_NAME variable set as either the fully qualified hostname or the IP address of the Jupyter server.  This should be reachable by the cognos server.

 

Step 5(RHEL) : Run the installer script 

Navigate to the unix directory and run the ./installer.

cd /opt/ibm/cognos/jupyter/dist/scripts/unix

./install.sh

When this has started up correctly you should see the following entry within the install.sh start up:

Check the containers are running

docker ps

 

If you paste this in the browser, you should be able to see the following message, which indicates your browser can successfully communicate with the Jupyter server. This is not an error code, this is expected.

http://servername:8000

Common installer issues (RHEL)

RHEL (11.1.4 and lower)

ERROR:

 

groupadd: invalid group ID 'docker'

The command '/bin/sh -c groupadd --gid $DOCKER_GROUP_UID docker &&   usermod -aG docker $NB_USER &&   gpasswd -a $NB_USER docker' returned a non-zero code: 3 


cd to the build script

cd /opt/ibm/cognos/jupyter/dist/scripts/unix/

edit the build file

nano build.sh

Resolve this by changing the docker group to dockerroot in the group id section like this:


Old versions of Docker

ERROR:

 

chown=ca_user:users pip.conf $HOME/.config/pip/.

Unknown flag: chown"

 

The chown flag used by Jupyter, is only supported as of Docker version 17.09 and up. Upgrade your version of Docker using the steps outlined above. One can consult the Docker documentation .

Offline Installs

ERROR:

 

Collecting pixiedust (from -r additional_packages.txt (line 4))

 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fd55b28bd90>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pixiedust/


Change directory to the tmp folder:

cd /tmp

Now Download Pixiedust using a wget

wget https://files.pythonhosted.org/packages/16/ba/7488f06b48238205562f9d63aaae2303c060c5dfd63b1ddd3bd9d4656eb1/pixiedust-1.1.18.tar.gz 

Then extract it

tar -xvf pixiedust-1.1.18.tar.gz

Navigate to the following directory

/ibm/cognos/jupyter/dist/scripts/

Edit the file

nano additional_packages.txt

Comment out all the lines with # 

Add the following line:

--no-index --find-links=/tmp/pixiedust-1.1.18  pixiedust.

 

Step 5 (Windows) : Run the installer script 

Execute the install script as an administrator:



Alternatively (if you would like to see error codes that pop up if it does not start the first time), open a command prompt as an administrator and cd to the location. Following that run the install.bat file 

cd C:\Program Files\ibm\cognos\jupyter\dist\scripts\windows

install.bat

If the script finished indicating it started successfully, you can run a docker ps to see if the containers are up.

docker ps



Step 6: Verify the Cognos Configuration

Verify the dispatcher URI for external application readys with the same syntax as follows:

http://servername.domain.com:9300/bi/v1/disp


Step 7: Configure the Cognos administration portal

Please access the dispatcher URI in your browser:

http://servername.domain.com:9300/bi/v1/disp

Click on the Manage Button

Navigate to Configuration > System > Environment (tab)

For the Jupyter service location place the URL that was specified on start-up during step 7.

http://jupyter1.fyre.ibm.com:8000

Be sure not to include a trailing slash here.

Click apply and refresh the browser (you may need to clear cache and start a new session)

Click on New > Notebook

 

And test your notebook!







#CognosAnalyticswithWatson
#LearnCognosAnalytics
#News-BA
#News-BA-home
0 comments
1878 views

Permalink