File and Object Storage

IBM Spectrum Scale HDFS Transparency Apache Hadoop 3.1.x Support

By Archive User posted Mon June 24, 2019 09:32 AM

  
Apache Hadoop 3.1.1 is supported with HDFS Transparency 3.1.0-2. When you use Apache Hadoop, the configuration files of HDFS Transparency are located under /var/mmfs/hadoop/etc/hadoop. By default, the logs of HDFS Transparency are located under /var/log/transparency/.
If you want to configure Apache Hadoop 3.1.1 with HDFS Transparency 3.1.0-2, execute the following steps:

1 Set ulimit nofile to 64K on all the nodes.

2 Set up the ntp server to synchronize the time on all nodes.

3 Root password-less access from NameNodes to all other DataNodes.

4 Install HDFS Transparency 3.1.0-2 on all HDFS Transparency nodes

ssh to TransparencyNode1.

5 Update the /var/mmfs/hadoop/etc/hadoop/core-site.xml with your NameNode hostname.



6 Update the /var/mmfs/hadoop/etc/hadoop/hdfs-site.xml according to your configuration:



7 Update the /var/mmfs/hadoop/etc/hadoop/gpfs-site.xml for the gpfs.mnt.dir, gpfs.data.dir and gpfs.storage.type configurations.

8 Update the /var/mmfs/hadoop/etc/hadoop/hadoop-env.sh and change export JAVA_HOME= into export JAVA_HOME=
#BigDataandAnalytics
#IBMSpectrumScale
#opensource
2 comments
0 views

Permalink

Comments

Tue July 02, 2019 09:41 AM

Has Update with the latest info

Tue June 25, 2019 11:35 AM

Hallo fycheng,
the blog should be updated because there are new Version of transparency available like gpfs.hdfs-protocol-3.1.0-2.x86_64 and the paths and konfigs are different.
Regards Renar