Apache Hadoop 3.1.1 is supported with HDFS Transparency 3.1.0-2. When you use Apache Hadoop, the configuration files of HDFS Transparency are located under /var/mmfs/hadoop/etc/hadoop. By default, the logs of HDFS Transparency are located under /var/log/transparency/.
If you want to configure Apache Hadoop 3.1.1 with HDFS Transparency 3.1.0-2, execute the following steps:
1 Set ulimit nofile to 64K on all the nodes.
2 Set up the ntp server to synchronize the time on all nodes.
3 Root password-less access from NameNodes to all other DataNodes.
4 Install HDFS Transparency 3.1.0-2 on all HDFS Transparency nodes
ssh to TransparencyNode1.
5 Update the /var/mmfs/hadoop/etc/hadoop/core-site.xml with your NameNode hostname.
6 Update the /var/mmfs/hadoop/etc/hadoop/hdfs-site.xml according to your configuration:
7 Update the /var/mmfs/hadoop/etc/hadoop/gpfs-site.xml for the gpfs.mnt.dir, gpfs.data.dir and gpfs.storage.type configurations.
8 Update the /var/mmfs/hadoop/etc/hadoop/hadoop-env.sh and change export JAVA_HOME= into export JAVA_HOME=#BigDataandAnalytics#IBMSpectrumScale#opensource