Originally posted by: 7S62_hui_liu
HDFS federation provides MapReduce with the ability to start multiple HDFS namespaces in the cluster, monitor their health, and fail over in case of daemon or host failure. With HDFS High Availabilty (HA), when a NameNode or Secondary NameNode fails, the service controller in IBM Spectrum Symphony (EGOSC) restarts it. When starting a NameNode or Secondary NameNode instance, its corresponding virtual IP (VIP) is assigned to the current host; when a NameNode or Secondary NameNode service is stopped, the assigned VIP is removed.
This blog details the steps that are required to configure high availability for a federated HDFS cluster.
You can configure HA for your federated HDFS cluster in two ways:
- Method 1: By adding the VIP addresses to the hdfs-site.xml file.
- Method 2: By adding the VIP addresses to the vipConf.sh script.
Method 1: Enable HA by adding VIP addresses to hdfs-site.xml
-
Copy the Hadoop package and extract its contents. For example, decompress hadoop-2.7.2.tar.gz to /opt; Hadoop home is now /opt/hadoop-2.7.2/.
- From the Hadoop conf path, complete the following steps:
2.1. Set the JAVA_HOME path in hadoop-env.sh. For example, in /opt/hadoop-2.7.2/etc/Hadoop/hadoop-env.sh, set JAVA_HOME as follows:
export JAVA_HOME=/pcc/app/Linux_jdk1.7.0_x86_64
2.2. Configure the master and master-candidate information in core-site.xml. Set the master IP, multiple master-candidate host IP, and the HDFS port (8020 is the HDFS port by default; you can modify this value).
For example, when 192.0.2.116 is the master host IP, 192.0.2.117 is the master-candidate host IP, and 8020 is the HDFS port, configure /opt/hadoop-2.7.2/etc/Hadoop/core-site.xml as follows:
<configuration>
<property>
<name>dfs.client.retry.policy.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>viewfs:///</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./ns1</name>
<value>hdfs://192.0.2.116:8020</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./ns2</name>
<value>hdfs://192.0.2.117:8020</value>
</property>
</configuration>
2.3. Configure the master and master-candidate information in hdfs-site.xml. Set the namespace number, master IP, multiple master-candidate host IP, and the HDFS port.
For example, when 192.0.2.116 is the master host IP and 192.0.2.117 is the master-candidate host IP, configure /opt/hadoop-2.7.2/etc/Hadoop/hdfs-site.xml as follows; ns1,ns2 means there are two namespaces, so there is one master and one master-candidate host:
<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2</value>
</property>
<property>
<name>dfs.namenode.name.dir.ns1</name>
<value>/tmp/share/ns1</value>
</property>
<property>
<name>dfs.namenode.name.dir.ns2</name>
<value>/tmp/share/ns2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>192.0.2.116:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1</name>
<value>192.0.2.116:50070</value>
</property>
<property>
<name>dfs.namenode.https-address.ns1</name>
<value>192.0.2.116:50470</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.ns1</name>
<value>192.0.2.116:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address.ns1</name>
<value>192.0.2.116:50091</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns2</name>
<value>192.0.2.117:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.ns2</name>
<value>192.0.2.117:50070</value>
</property>
<property>
<name>dfs.namenode.https-address.ns2</name>
<value>192.0.2.117:50470</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.ns2</name>
<value>192.0.2.117:50090</value>
</property>
<property>
<name>dfs.namenode.secondary.https-address.ns2</name>
<value>192.0.2.117:50091</value>
</property>
<property>
<name>dfs.namenode.edits.toleration.length</name>
<value>8192</value>
</property>
</configuration>
2.4. Format HDFS. For example:
/opt/hadoop-2.7.2/bin/hdfs namenode –format
- Repeat step 1 and 2 on all hosts.
- Install the IBM Spectrum Symphony package on all hosts.
On the master host, it’s strongly recommended to set the JAVA_HOME, HADOOP_HOME, HADOOP_VERSION environment to avoid issues with modifying them after installation.
For example:
setenv CLUSTERNAME host1_hdfs_ha
setenv CLUSTERADMIN user1
setenv BASEPORT 23110
setenv SIMPLIFIEDWEM N
setenv DERBY_DB_HOST host1
setenv JAVA_HOME /pcc/app/Linux_jdk1.7.0_x86_64
setenv HADOOP_YARN_HOME /opt/hadoop-2.7.2/
setenv HADOOP_HOME /opt/hadoop-2.7.2
setenv HADOOP_VERSION 2_7_x
setenv DFS_GUI_HOSTNAME host1
setenv DFS_GUI_PORT 50070
- Go to the master host and modify the following files under $EGO_CONFDIR/../../eservice/esc/conf/services/:
5.1. Modify namenode.xml and secondarynode.xml to set the MinInstances and MaxInstances value, which should be the same as the namespaces’ number.
For example, hdfs-site.xml has ns1,ns2 -- two namespaces, so set MinInstances and MaxInstances to 2.
<sc:MinInstances>2</sc:MinInstances>
<sc:MaxInstances>2</sc:MaxInstances>
Set NAMESPACE_BASE_NAME and FEDERATION_VIP_ENABLE. For example:
<ego:EnvironmentVariable name="NAMESPACE_BASE_NAME">ns</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
5.2 Modify datanode.xml to enable FEDERATION_VIP_ENABLE:
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
5.3 Configure vipConf.sh under $SOAM_HOME/mapreduce/$SOAM_VERSION/${EGO_MACHINE_TYPE}/etc/ to set the master and master-candidate host network card. Default value is eth0.
For example, when enp22s0f4 is the master network card and eth2 is the master-candidate host network card, configure vipConf.sh as follows:
#!/bin/sh
function getVIPConfig4NameNodes {
... ...
case $1 in
1 )
SYM_HA_HDFS_VIRTUAL_IP=
SYM_HA_HDFS_ETH=enp22s0f4
SYM_HA_HDFS_BROADCAST=
SYM_HA_HDFS_NETMASK=
;;
2 )
SYM_HA_HDFS_VIRTUAL_IP=
SYM_HA_HDFS_ETH=eth2
SYM_HA_HDFS_BROADCAST=
SYM_HA_HDFS_NETMASK=
... ...
}
function getVIPConfig4SecondaryNameNodes {
... ...
case $1 in
1 )
SYM_HA_HDFS_VIRTUAL_IP=
SYM_HA_HDFS_ETH=enp22s0f4
SYM_HA_HDFS_BROADCAST=
SYM_HA_HDFS_NETMASK=
;;
2 )
SYM_HA_HDFS_VIRTUAL_IP=
SYM_HA_HDFS_ETH=eth2
SYM_HA_HDFS_BROADCAST=
SYM_HA_HDFS_NETMASK=
... ...
}
- Start the cluster:
egosh ego start
Check the NameNode/SecondaryNode/DataNode services. For example:
[root@host1 logs]# egosh service list
SERVICE STATE ALLOC CONSUMER RGROUP RESOURCE SLOTS SEQ_NO INST_STATE ACTI
… …
Seconda* STARTED 15 /HDFS/Se* host1 1 1 RUN 67
Host2 1 2 RUN 68
MRSS STARTED 8 /Compute*MapRed*host3 1 1 RUN 76
NameNode STARTED 14 /HDFS/Na* host1 1 1 RUN 62
Host2 1 2 RUN 64
DataNode STARTED 16 /HDFS/Da* host3 1 1 RUN 73
… …
Check the HDFS logs under $SOAM_HOME/mapreduce/logs. For example:
[root@host1 logs]# cat Platform-HDFS-namenode-host1.log
[Thu Feb 16 01:57:48 EST 2017] NameNode serviceMaxInstance <2>, serviceMinInstance <2>, number of NameServices defined as <2>.
[Thu Feb 16 01:57:48 EST 2017] Namenode Virtual IP was not configured in vipConf.sh.
… …
[Thu Feb 16 01:57:56 EST 2017] All HDFS services (NameNode, DataNode, SecondaryNode) for cluster host1_hdfs_ha started successfully!
[root@host1 logs]#
[root@host1 logs]# cat Platform-HDFS-secondarynode-host1.log
[Thu Feb 16 01:57:54 EST 2017] Setting EGOCC_FILE as /tmp/tmp.hRE4bOu7aM...
… …
[Thu Feb 16 01:57:59 EST 2017] SecondaryNode for cluster host1_hdfs_ha started successfully!
Method 2: Enable HA by adding VIP addresses to vipConf.sh
- Copy the Hadoop package and extract its contents. For example, decompress hadoop-2.7.2.tar.gz to /opt; Hadoop home is now /opt/hadoop-2.7.2/.
- From the Hadoop conf path, complete the following steps:
2.1. Set the JAVA_HOME path in hadoop-env.sh. For example, in /opt/hadoop-2.7.2/etc/Hadoop/hadoop-env.sh, set JAVA_HOME as follows:
export JAVA_HOME=/pcc/app/Linux_jdk1.7.0_x86_64
2.2. Configure the master and master-candidate information in core-site.xml. Set the master IP, multiple master-candidate host IP, and the HDFS port (8020 is the HDFS port by default; you can modify this value).
For example, when 192.0.2.116 is the master host IP, 192.0.2.117 is the master-candidate host IP, and 8020 is the HDFS port, configure /opt/hadoop-2.7.2/etc/Hadoop/core-site.xml as follows:
<configuration>
<property>
<name>dfs.client.retry.policy.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>viewfs:///</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./ns1</name>
<value>hdfs://192.0.2.116:8020</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./ns2</name>
<value>hdfs://192.0.2.117:8020</value>
</property>
</configuration>
2.3. Configure the master and master-candidate information in hdfs-site.xml to set the namespace number, path, etc. For example:
<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2</value>
</property>
<property>
<name>dfs.namenode.name.dir.ns1</name>
<value>/home/huiliu/0216_mr_test/ns1</value>
</property>
<property>
<name>dfs.namenode.name.dir.ns2</name>
<value>/home/huiliu/0216_mr_test/ns2</value>
</property>
<property>
<name>dfs.namenode.edits.toleration.length</name>
<value>8192</value>
</property>
</configuration>
2.4. Format HDFS. For example:
/opt/hadoop-2.7.2/bin/hdfs namenode –format
- Repeat step 1 and 2 on all hosts.
- Install the IBM Spectrum Symphony package on all hosts.
On the master host, it’s strongly recommended to set the JAVA_HOME, HADOOP_HOME, HADOOP_VERSION environment to avoid issues with modifying them after installation.
For example:
setenv CLUSTERNAME host1_hdfs_ha
setenv CLUSTERADMIN user1
setenv BASEPORT 23110
setenv SIMPLIFIEDWEM N
setenv DERBY_DB_HOST host1
setenv JAVA_HOME /pcc/app/Linux_jdk1.7.0_x86_64
setenv HADOOP_YARN_HOME /opt/hadoop-2.7.2/
setenv HADOOP_HOME /opt/hadoop-2.7.2
setenv HADOOP_VERSION 2_7_x
setenv DFS_GUI_HOSTNAME host1
setenv DFS_GUI_PORT 50070
- Go to the master host and modify the following files under $EGO_CONFDIR/../../eservice/esc/conf/services/:
5.1. Modify namenode.xml and secondarynode.xml to set the MinInstances and MaxInstances value, which should be the same as the namespaces’ number.
For example, hdfs-site.xml has ns1,ns2 -- two namespaces, so set MinInstances and MaxInstances to 2.
<sc:MinInstances>2</sc:MinInstances>
<sc:MaxInstances>2</sc:MaxInstances>
Set NAMESPACE_BASE_NAME and FEDERATION_VIP_ENABLE. For example:
<ego:EnvironmentVariable name="NAMESPACE_BASE_NAME">ns</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
Set the PMR_HDFS_PORT value to be consistent with core-site.xml. For example:
<ego:EnvironmentVariable name="PMR_HDFS_PORT">8020</ego:EnvironmentVariable>
5.2 Modify datanode.xml to enable FEDERATION_VIP_ENABLE:
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
5.3 Configure vipConf.sh under $SOAM_HOME/mapreduce/$SOAM_VERSION/${EGO_MACHINE_TYPE}/etc/ to set the master and master-candidate host IP, network card, netmask, and broadcast.
For example, when the master host and master-candidate host have the following configuration, configure vipConf.sh as follows:
Master host:
netword card: enp22s0f4
ip: 192.0.2.116
netmask: 255.255.255.0
broadcast: 192.0.2.255
Master-candidate host:
netword card: eth2
ip: 192.0.2.117
netmask: 255.255.255.0
broadcast: 192.0.2.255
#!/bin/sh
function getVIPConfig4NameNodes {
... ...
case $1 in
1 )
SYM_HA_HDFS_VIRTUAL_IP=192.0.2.116
SYM_HA_HDFS_ETH=enp22s0f4
SYM_HA_HDFS_BROADCAST=192.0.2.255
SYM_HA_HDFS_NETMASK=255.255.255.0
;;
2 )
SYM_HA_HDFS_VIRTUAL_IP=192.0.2.117
SYM_HA_HDFS_ETH=eth2
SYM_HA_HDFS_BROADCAST=192.0.2.255
SYM_HA_HDFS_NETMASK=255.255.255.0
... ...
}
function getVIPConfig4SecondaryNameNodes {
... ...
case $1 in
1 )
SYM_HA_HDFS_VIRTUAL_IP=192.0.2.116
SYM_HA_HDFS_ETH=enp22s0f4
SYM_HA_HDFS_BROADCAST=192.0.2.255
SYM_HA_HDFS_NETMASK=255.255.255.0
;;
2 )
SYM_HA_HDFS_VIRTUAL_IP=192.0.2.117
SYM_HA_HDFS_ETH=eth2
SYM_HA_HDFS_BROADCAST=192.0.2.255
SYM_HA_HDFS_NETMASK=255.255.255.0
... ...
}
- Start the cluster:
egosh ego start
Check the NameNode/SecondaryNode/DataNode services.
Check the HDFS logs under $SOAM_HOME/mapreduce/logs.
That's it! We hope these steps helped you set up high availability for your federated cluster more easily. Let us know how it went by responding in our forum! As always, follow this blog series for more articles.
#SpectrumComputingGroup